docker


Rough notes on setting up an Ubuntu server with docker

Static IP

First, we set up a static IP to our Ubuntu server using netplan. To do so, we created the following file:

/etc/netplan/01-netcfg.yaml

using the following command

sudo nano /etc/netplan/01-netcfg.yaml;

and added the following content to it:

# This file describes the network interfaces available on your system
# For more information, see netplan(5).
network:
  version: 2
  renderer: networkd
  ethernets:
    enp3s0f0:
      dhcp4: no
      addresses: [192.168.45.13/24]
      gateway4: 192.168.45.1
      nameservers:
          addresses: [1.1.1.1,8.8.8.8]

To apply the changes, we executed the following:

sudo netplan apply;

Update everything (the operating system and all packages)

Usually, it is a good idea to update your system before making significant changes to it:

sudo apt update -y; sudo apt upgrade -y; sudo apt autoremove -y;

Install docker

In this setup we did not use the docker version that is available on the Ubuntu repositories, we went for the official ones from docker.com. To install it, we used the following commands:

sudo apt-get install ca-certificates curl gnupg lsb-release;
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /usr/share/keyrings/docker-archive-keyring.gpg;
echo   "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/docker-archive-keyring.gpg] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null;
sudo apt-get update;
sudo apt-get install docker-ce docker-ce-cli containerd.io;

Install docker-compose

Again, we installed the official docker-compose from github.com instead of the one available in the Ubuntu repositories. At the time that this post was created, version 1.29.2 was the recommended one:

sudo curl -L "https://github.com/docker/compose/releases/download/1.29.2/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose;
sudo chmod +x /usr/local/bin/docker-compose;

Increase network pool for docker daemon

To handle the following problem:

ERROR: could not find an available, non-overlapping IPv4 address pool among the defaults to assign to the network

We created the following file,

/etc/docker/daemon.json

using the command:

sudo nano /etc/docker/daemon.json;

and added the following content to it:

{
  "default-address-pools": [
    {
      "base": "172.80.0.0/16",
      "size": 24
    },
    {
      "base": "172.90.0.0/16",
      "size": 24
    }
  ]
}

We executed the following command to restart the docker daemon and get the network changes applied:

sudo systemctl restart docker;

Gave access to our user to manage docker

We added our user to the docker group so that we could manage the docker daemon without sudo rights.

sudo usermod -aG docker $USER;

Deploying

After we copied everything in place, we executed the following command to create our containers and start them with the appropriate networks and volumes:

export COMPOSE_HTTP_TIMEOUT=120;
docker-compose up -d --remove-orphans;

We had to increase the timeout as we were getting the following error:

ERROR: for container_a  UnixHTTPConnectionPool(host='localhost', port=None): Read timed out. (read timeout=60)
ERROR: An HTTP request took too long to complete. Retry with --verbose to obtain debug information.
If you encounter this issue regularly because of slow network conditions, consider setting COMPOSE_HTTP_TIMEOUT to a higher value (current value: 60).

Stopping all containers using a filter on the name

docker container stop $(docker container ls -q --filter name=_web);

The above command will find all containers whose names contain _web and stop them. That command is actually two commands where one is nested inside the other.

#This command finds all containers that their name contains _web, using the -q parameter, we only get back the container ID and not all information about them.
docker container ls -q --filter name=_web;
#The second command takes as input the output of the nested command and stops all containers that are returned.
docker container stop $(docker container ls -q --filter name=_web);

ERROR: for container_a UnixHTTPConnectionPool(host=’localhost’, port=None): Read timed out. (read timeout=60)

There is this docker server that we have access to, which probably due to lousy planning, we put way too many containers on it. The server does not have SSD disks, and for that reason, whenever there are too many IO operations, it becomes unresponsive. When we mass update all containers by updating the images using the following command and then issuing a fresh docker-compose, we get a lot of time-out errors.

The commands we use to update the images and recreate our containers using the new images are the following:
(Please note that these commands need to execute from the folder where the file docker-compose.yml resides)

#Update all docker images that have the 'latest' tag
docker images --format "{{.Repository}}:{{.Tag}}" | grep ':latest' | xargs -L1 docker pull;
#Rebuild all containers using the new images.
docker-compose up -d;

After executing the second command, we often get many copies of the following error:

ERROR: for container_a  UnixHTTPConnectionPool(host='localhost', port=None): Read timed out. (read timeout=60)

This error indicates that the recreate command was waiting for too long for the docker daemon to respond with no success. At the end of the output, we can see that it was waiting for 60 seconds.

At the end of the output, we get the following information and advice:

ERROR: An HTTP request took too long to complete. Retry with --verbose to obtain debug information.
If you encounter this issue regularly because of slow network conditions, consider setting COMPOSE_HTTP_TIMEOUT to a higher value (current value: 60).

Following the advice, we used the following command to overwrite the value of the COMPOSE_HTTP_TIMEOUT variable to a more significant number.

#Increase timeout period to 120 seconds.
export COMPOSE_HTTP_TIMEOUT=120;
#Rebuild all containers using the new images.
docker-compose up -d;

Doing so, we were able to rebuild all containers without reissuing many times the up command.

Sidenote

This server really does have a lot of containers, we had to create the file /etc/docker/daemon.json so that we would have enough network addressing space to handle all the bridges and sub-networks.

The contents of /etc/docker/daemon.json are:

{
  "default-address-pools": [
    {
      "base": "172.80.0.0/16",
      "size": 24
    },
    {
      "base": "172.90.0.0/16",
      "size": 24
    }
  ]
}

The above configuration solved the following problem for us:

ERROR: could not find an available, non-overlapping IPv4 address pool among the defaults to assign to the network

Docker: WARNING: Host is already in use by another container

We use docker to manage multiple instances of various tools on a server that we control. We have an Nginx server working as a reverse proxy that forwards all requests to the appropriate containers in the configuration. Sometimes, after updating the container images and recreating the containers, we get the error that ports 80 and 443 are already in use by another container. This problem can happen even if no other container asks for them.

The following excerpt demonstrates the problem as mentioned above.

[email protected]:~/docker-compose$ docker-compose up -d --remove-orphans;
Recreating container_a ... 
Recreating container_a ... done
Recreating container_b   ... done
Recreating container_c          ... 
Recreating nginx_reverse_proxy        ... error
Recreating container_d          ... done
Recreating container_e       ... done
Recreating container_f  ... done
WARNING: Host is already in use by another container

ERROR: for nginx_reverse_proxy  Cannot start service nginx_reverse_proxy: driver failed programming external connectivity on endpoint nginx_reverse_proxy (5a790ed7e1b24aa36cb88cbd3f49d306efa8fe023bf5b3312655218319f23a35): Bind for 0.0.0.0:443 failed: port is already allocated

ERROR: for nginx_reverse_proxy  Cannot start service nginx_reverse_proxy: driver failed programming external connectivity on endpoint nginx_reverse_proxy (5a790ed7e1b24aa36cb88cbd3f49d306efa8fe023bf5b3312655218319f23a35): Bind for 0.0.0.0:443 failed: port is already allocated
ERROR: Encountered errors while bringing up the project.
[email protected]:~/docker-compose$ sudo systemctl restart docker.socket docker.service;

To solve this issue, we had to restart two services using the systemctl command:

  • docker.socket
  • docker.service

Specifically, on an Ubuntu server, we used the following command:

sudo systemctl restart docker.socket docker.service;

Create a project using Symfony website-skeleton version 4 and then create a docker image out of it

This guide will present the steps we followed on a GNU/Linux Ubuntu 20.04LTS to create a new project out of the Symfony website skeleton and then create a new docker application image of it.

Install core dependecies

Install php-cli instead of php as we do not want to install the additional dependencies of php like apache2.
p7zip-full is needed for the package manager of composer later on. If it is missing, we will be getting one of the following errors:

Failed to download symfony/requirements-checker from dist: The zip extension and unzip/7z commands are both missing, skipping.
As there is no 'unzip' nor '7z' command installed zip files are being unpacked using the PHP zip extension.

php-xml will be required later on while creating the skeleton project for Symfony. If it is missing, you will get the following error:

symfony/framework-bundle requires ext-xml * -> it is missing from your system. Install or enable PHP's xml extension
sudo apt install php-cli php-xml p7zip-full;

Composer is a PHP utility for managing dependencies. It allows you to indicate the libraries your project relies on, and it will take care of installing and updating them. To fast install it, open a terminal and type the following command:

curl -Ss getcomposer.org/installer | php;
# Moving the composer into the /usr/local/bin/ folder will allow us to access it from any folder later on as that folder is in the default PATH variable.
sudo mv composer.phar /usr/local/bin/composer;

Symfony provides a tool to check if your operating system meets the required requirements rapidly. In addition, if suitable, the tool makes installation recommendations. To install the tool, run the following command:

composer require symfony/requirements-checker;
$ composer require symfony/requirements-checker;
Using version ^2.0 for symfony/requirements-checker
./composer.json has been updated
Running composer update symfony/requirements-checker
Loading composer repositories with package information
Updating dependencies
Lock file operations: 1 install, 0 updates, 0 removals
  - Locking symfony/requirements-checker (v2.0.1)
Writing lock file
Installing dependencies from lock file (including require-dev)
Package operations: 1 install, 0 updates, 0 removals
  - Installing symfony/requirements-checker (v2.0.1): Extracting archive
Generating autoload files
1 package you are using is looking for funding.
Use the `composer fund` command to find out more!

Once done, you can safely delete the requirements-checker:

composer remove symfony/requirements-checker;

Create the Symfony project

Using the basic skeleton, you can create a minimal Symfony project with the following command. We install the latest version of version 4.4 of the website skeleton project in this example. We found the list of versions here https://packagist.org/packages/symfony/website-skeleton.

composer create-project symfony/website-skeleton=4.4.99 symfony-skeleton;

When we got the following warning, we typed y, not sure what changes, so we stayed with the default option:

  -  WARNING  symfony/mailer (>=4.3): From github.com/symfony/recipes:master
    The recipe for this package contains some Docker configuration.

    This may create/update docker-compose.yml or update Dockerfile (if it exists).

    Do you want to include Docker configuration from recipes?
    [y] Yes
    [n] No
    [p] Yes permanently, never ask again for this project
    [x] No permanently, never ask again for this project
    (defaults to y): y

Then you need to run the following commands to install all dependencies and execute the project:

cd symfony-skeleton;
composer install;
composer require --dev symfony/web-server-bundle;
php bin/console server:start *:8000;

By now, you should see in a browser the landing page of your skeleton project.

# Stop the php webserver and release the port, we will need it later on.
php bin/console server:stop;

Install docker on Ubuntu

First of all, make sure your system is clean and remove any old versions:

sudo apt-get remove docker docker-engine docker.io containerd runc;
# You might want to execute `sudo apt autoremove -y;` as well to cleanup everything. We cannot ask everyone to do so as we are not sure of what complications it might have on each computer+software configurations.

We will be installing docker by adding its repositories to our system:

sudo apt-get update;
sudo apt-get install ca-certificates curl gnupg lsb-release;
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /usr/share/keyrings/docker-archive-keyring.gpg;
echo "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/docker-archive-keyring.gpg] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null;
sudo apt-get update;
sudo apt-get install docker-ce docker-ce-cli containerd.io;
sudo docker run hello-world;

If the installation was OK, you should see the following message:

n$ sudo docker run hello-world
Unable to find image 'hello-world:latest' locally
latest: Pulling from library/hello-world
2db29710123e: Pull complete 
Digest: sha256:cc15c5b292d8525effc0f89cb299f1804f3a725c8d05e158653a563f15e4f685
Status: Downloaded newer image for hello-world:latest

Hello from Docker!
This message shows that your installation appears to be working correctly.

To generate this message, Docker took the following steps:
 1. The Docker client contacted the Docker daemon.
 2. The Docker daemon pulled the "hello-world" image from the Docker Hub.
    (amd64)
 3. The Docker daemon created a new container from that image which runs the
    executable that produces the output you are currently reading.
 4. The Docker daemon streamed that output to the Docker client, which sent it
    to your terminal.

To try something more ambitious, you can run an Ubuntu container with:
 $ docker run -it ubuntu bash

Share images, automate workflows, and more with a free Docker ID:
 https://hub.docker.com/

For more examples and ideas, visit:
 https://docs.docker.com/get-started/

Make the docker application image

Execute the following command on a terminal to get your php version:

php --version;

In case you get something different than version 7.4, please note it and update the contents of the DockerFile below accordingly. In our case, the results for the version were the ones right below and that is why we used the line FROM php:7.4-cli in our DockerFile.

$ php --version
PHP 7.4.3 (cli) (built: Oct 25 2021 18:20:54) ( NTS )
Copyright (c) The PHP Group
Zend Engine v3.4.0, Copyright (c) Zend Technologies
    with Zend OPcache v7.4.3, Copyright (c), by Zend Technologies

If you are not already at the root of your project (e.g., the symfony-skeleton folder), go to that folder and create a new text file with the name Dockerfile in there. The contents of the file should be the following:

# Dockerfile
FROM php:7.4-cli

RUN apt-get update -y && apt-get install -y libmcrypt-dev

RUN curl -sS https://getcomposer.org/installer | php -- --install-dir=/usr/local/bin --filename=composer
RUN apt-get update && apt-get install -y libonig-dev
RUN docker-php-ext-install pdo

WORKDIR /app
COPY . /app

RUN composer install

EXPOSE 8000
CMD php bin/console server:run 0.0.0.0:8000

Once you have Docker and Docker Machine installed on your machine, creating the container is a breeze. The command below will seek your Dockerfile and download all of the layers required to execute your container image. It will then complete the commands in the Dockerfile, leaving you with a container that is ready to use.

You’ll use the docker build command to create your php Symfony docker container, and you’ll give it a tag or a name so you can refer to it later when you want to execute it. The command’s final component instructs Docker to build from a specific directory.

sudo docker build -t symfony-project .;

To execute the new application image:

sudo docker run -it -p 8000:8000 symfony-project;

To export the Docker image as a tar file:

sudo docker save -o ~/symfony-skeleton.tar symfony-project;

To import the Docker image from the tar file:

sudo docker load -i symfony-skeleton.tar;


Quick notes on optimizing a mysql/mariadb database in a docker container

First, to avoid opening the network of the container to another network, you need to open a shell to the docker container itself. To do so, use the following command:

docker exec -it website_db /bin/bash;

After executing the exec command, you will get a shell to the docker container. Use the following command to optimize all databases and their tables:

mysqlcheck -p -o --all-databases;

The -p parameter instructs the command to ask for a password (which probably and hopefully have set).

To exit the console and close the connection just type exit; after you are done with the above command.

Note: Where we used the word website_db you need to use the name of your container. If you are not sure of the name of the container, you can list all of the containers with their names using the following command:

docker container ls;