default-address-pools

ERROR: for container_a UnixHTTPConnectionPool(host=’localhost’, port=None): Read timed out. (read timeout=60)

 in Applications / Docker  tagged COMPOSE_HTTP_TIMEOUT / container / default-address-pools / docker / docker-compose / images / nginx / pull / reverse proxy / timed out / UnixHTTPConnectionPool / update by

There is this docker server that we have access to, which probably due to lousy planning, we put way too many containers on it. The server does not have SSD disks, and for that reason, whenever there are too many IO operations, it becomes unresponsive. When we mass update all containers by updating the images using the following command and then issuing a fresh docker-compose, we get a lot of time-out errors.

The commands we use to update the images and recreate our containers using the new images are the following:
(Please note that these commands need to execute from the folder where the file docker-compose.yml resides)

#Update all docker images that have the 'latest' tag
docker images --format "{{.Repository}}:{{.Tag}}" | grep ':latest' | xargs -L1 docker pull;
#Rebuild all containers using the new images.
docker-compose up -d;

After executing the second command, we often get many copies of the following error:

ERROR: for container_a  UnixHTTPConnectionPool(host='localhost', port=None): Read timed out. (read timeout=60)

This error indicates that the recreate command was waiting for too long for the docker daemon to respond with no success. At the end of the output, we can see that it was waiting for 60 seconds.

At the end of the output, we get the following information and advice:

ERROR: An HTTP request took too long to complete. Retry with --verbose to obtain debug information.
If you encounter this issue regularly because of slow network conditions, consider setting COMPOSE_HTTP_TIMEOUT to a higher value (current value: 60).

Following the advice, we used the following command to overwrite the value of the COMPOSE_HTTP_TIMEOUT variable to a more significant number.

#Increase timeout period to 120 seconds.
export COMPOSE_HTTP_TIMEOUT=120;
#Rebuild all containers using the new images.
docker-compose up -d;

Doing so, we were able to rebuild all containers without reissuing many times the up command.

Sidenote

This server really does have a lot of containers, we had to create the file /etc/docker/daemon.json so that we would have enough network addressing space to handle all the bridges and sub-networks.

The contents of /etc/docker/daemon.json are:

{
  "default-address-pools": [
    {
      "base": "172.80.0.0/16",
      "size": 24
    },
    {
      "base": "172.90.0.0/16",
      "size": 24
    }
  ]
}

The above configuration solved the following problem for us:

ERROR: could not find an available, non-overlapping IPv4 address pool among the defaults to assign to the network

Ubuntu – Overwrite dockerd default settings

 in Bash / GNU/Linux  tagged bash / bridge / default-address-pools / docker / docker-compose / dockerd / GNU/Linux / network / systemctl / ubuntu by

Trying to create a new bridge on docker, we got the following error

$ docker-compose up -d;
Creating network "docker-compose_new_bridge" with driver "bridge"
ERROR: could not find an available, non-overlapping IPv4 address pool among the defaults to assign to the network

After investigating, we realized that it was due to some default limitations of docker that did not allow more virtual networks to be created. To overcome the problem, we read that we had to give access to more address space using the /etc/docker/daemon.json.

On Ubuntu that file did not exist so we created it and copied the following content to it:

{
  "default-address-pools": [
    {
      "base": "172.80.0.0/16",
      "size": 24
    },
    {
      "base": "172.90.0.0/16",
      "size": 24
    }
  ]
}

Source: https://docs.docker.com/engine/reference/commandline/dockerd/

This configuration allowed Docker to reserve the network address space 172.80.[0-255].0/24 and 172.90.[0-255].0/24, that provided the daemon a total of 512 networks each owning 256 addresses.

To apply the changes to the daemon, we restarted it:

sudo systemctl restart docker.service;

and then we applied our changes to our docker ecosystem:

docker-compose up -d;