Docker Compose: Retry Database Connect With Docker and Go

When I implemented a Docker Compose file that starts our entire system, I stumbled upon a common problem. The database container is ready, but the database itself is not. So the services try to establish a connection, but fail, because the database is still starting.

Our system architecture looks like this. A mysql database contains the data, and a gateway service works as a reverse proxy to the internal network of microservices (written in Golang). To avoid “works on my machine” problems, a common docker-compose file should be used so that every developer uses the same setup.

Running multi-container Docker applications

The related docker-compose.yaml file (short version) looks like this:

version: '3.7'
services:
  database:
    container_name: db
    build:
        context: ./database/mysql
    ports:
      - "3306:3306"
    restart: always
  user-service:
    container_name: user-service
    build: ../user-service
    environment:
      - DB_SERVER=database
    ports: 
      - "8081:8081"
    depends_on:
      - "database"

Each Golang microservice and the mysql database have a separate Dockerfile referenced from the docker-compose.yaml file. The microservices are using multi-stage builds as explained in this post.

FROM mysql:5.7

ENV MYSQL_DATABASE=mydatabase
ENV MYSQL_ROOT_PASSWORD=topsecret
ENV MYSQL_USER=admin
ENV MYSQL_PASSWORD=secretpw

EXPOSE 3306

First, I added the depends_on option to all microservices. You can control the order of service startup and shutdown with it. That tells Docker Compose to only start the microservice containers once the database container is running.

The problem is that Docker does not know that a running container is not the same as a running database. Therefore, the microservice containers are started, try to connect and fail as the database is not ready yet. Another problem, of course, is that the services only try to connect once before shutting down.

Up to version 2 of Docker compose, depends_on had a sub-parameter “condition” which allowed to define a health check. However, this was removed in v3.

depends_on:
      db:
        condition: service_healthy

Resilient Application Startup

To handle this, design your application to attempt to re-establish a connection to the database after a failure. If the application retries the connection, it can eventually connect to the database.

https://docs.docker.com/compose/startup-order/

I followed the advice to check in my application code to see if the database is ready. If this is not the case, the service waits two seconds before attempting to connect again. It gives up after 30 failed attempts.

db, err := sql.Open("mysql", dbConnData)
if err != nil {
   log.Panicln(err.Error())
   return err
}

retryCount := 30
for {
   err := db.Ping()
   if err != nil {
	 if retryCount == 0 {
		log.Fatalf("Not able to establish connection to database %s", dbConfig.Database)
	}

	log.Printf(fmt.Sprintf("Could not connect to database. Wait 2 seconds. %d retries left...", retryCount))
	retryCount--
	time.Sleep(2 * time.Second)
   } else {
	break
   }
}

I found it to be confusing that db.Open() does not connect to the database but db.Ping() does. That is, we call db.Ping(). If it returns an error (driver.ErrBadConn), the service waits for two seconds and decreases the retry counter.

Open may just validate its arguments without creating a connection to the database. To verify that the data source name is valid, call Ping.

https://golang.org/pkg/database/sql/#Open

Final Solution in Action

The database and the service are started with

docker run -p 3306:3306 database
docker run user_service

The -p flag publishes the exposed port to the host. Otherwise, the port would only be accessible within the container network.

Exemplary console log where the the connecting microservice waits while the database is started

Further links