systemctl


Rough notes on setting up an Ubuntu server with docker

Static IP

First, we set up a static IP to our Ubuntu server using netplan. To do so, we created the following file:

/etc/netplan/01-netcfg.yaml

using the following command

sudo nano /etc/netplan/01-netcfg.yaml;

and added the following content to it:

# This file describes the network interfaces available on your system
# For more information, see netplan(5).
network:
  version: 2
  renderer: networkd
  ethernets:
    enp3s0f0:
      dhcp4: no
      addresses: [192.168.45.13/24]
      gateway4: 192.168.45.1
      nameservers:
          addresses: [1.1.1.1,8.8.8.8]

To apply the changes, we executed the following:

sudo netplan apply;

Update everything (the operating system and all packages)

Usually, it is a good idea to update your system before making significant changes to it:

sudo apt update -y; sudo apt upgrade -y; sudo apt autoremove -y;

Install docker

In this setup we did not use the docker version that is available on the Ubuntu repositories, we went for the official ones from docker.com. To install it, we used the following commands:

sudo apt-get install ca-certificates curl gnupg lsb-release;
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /usr/share/keyrings/docker-archive-keyring.gpg;
echo   "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/docker-archive-keyring.gpg] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null;
sudo apt-get update;
sudo apt-get install docker-ce docker-ce-cli containerd.io;

Install docker-compose

Again, we installed the official docker-compose from github.com instead of the one available in the Ubuntu repositories. At the time that this post was created, version 1.29.2 was the recommended one:

sudo curl -L "https://github.com/docker/compose/releases/download/1.29.2/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose;
sudo chmod +x /usr/local/bin/docker-compose;

Increase network pool for docker daemon

To handle the following problem:

ERROR: could not find an available, non-overlapping IPv4 address pool among the defaults to assign to the network

We created the following file,

/etc/docker/daemon.json

using the command:

sudo nano /etc/docker/daemon.json;

and added the following content to it:

{
  "default-address-pools": [
    {
      "base": "172.80.0.0/16",
      "size": 24
    },
    {
      "base": "172.90.0.0/16",
      "size": 24
    }
  ]
}

We executed the following command to restart the docker daemon and get the network changes applied:

sudo systemctl restart docker;

Gave access to our user to manage docker

We added our user to the docker group so that we could manage the docker daemon without sudo rights.

sudo usermod -aG docker $USER;

Deploying

After we copied everything in place, we executed the following command to create our containers and start them with the appropriate networks and volumes:

export COMPOSE_HTTP_TIMEOUT=120;
docker-compose up -d --remove-orphans;

We had to increase the timeout as we were getting the following error:

ERROR: for container_a  UnixHTTPConnectionPool(host='localhost', port=None): Read timed out. (read timeout=60)
ERROR: An HTTP request took too long to complete. Retry with --verbose to obtain debug information.
If you encounter this issue regularly because of slow network conditions, consider setting COMPOSE_HTTP_TIMEOUT to a higher value (current value: 60).

Stopping all containers using a filter on the name

docker container stop $(docker container ls -q --filter name=_web);

The above command will find all containers whose names contain _web and stop them. That command is actually two commands where one is nested inside the other.

#This command finds all containers that their name contains _web, using the -q parameter, we only get back the container ID and not all information about them.
docker container ls -q --filter name=_web;
#The second command takes as input the output of the nested command and stops all containers that are returned.
docker container stop $(docker container ls -q --filter name=_web);

Docker: WARNING: Host is already in use by another container

We use docker to manage multiple instances of various tools on a server that we control. We have an Nginx server working as a reverse proxy that forwards all requests to the appropriate containers in the configuration. Sometimes, after updating the container images and recreating the containers, we get the error that ports 80 and 443 are already in use by another container. This problem can happen even if no other container asks for them.

The following excerpt demonstrates the problem as mentioned above.

[email protected]:~/docker-compose$ docker-compose up -d --remove-orphans;
Recreating container_a ... 
Recreating container_a ... done
Recreating container_b   ... done
Recreating container_c          ... 
Recreating nginx_reverse_proxy        ... error
Recreating container_d          ... done
Recreating container_e       ... done
Recreating container_f  ... done
WARNING: Host is already in use by another container

ERROR: for nginx_reverse_proxy  Cannot start service nginx_reverse_proxy: driver failed programming external connectivity on endpoint nginx_reverse_proxy (5a790ed7e1b24aa36cb88cbd3f49d306efa8fe023bf5b3312655218319f23a35): Bind for 0.0.0.0:443 failed: port is already allocated

ERROR: for nginx_reverse_proxy  Cannot start service nginx_reverse_proxy: driver failed programming external connectivity on endpoint nginx_reverse_proxy (5a790ed7e1b24aa36cb88cbd3f49d306efa8fe023bf5b3312655218319f23a35): Bind for 0.0.0.0:443 failed: port is already allocated
ERROR: Encountered errors while bringing up the project.
[email protected]:~/docker-compose$ sudo systemctl restart docker.socket docker.service;

To solve this issue, we had to restart two services using the systemctl command:

  • docker.socket
  • docker.service

Specifically, on an Ubuntu server, we used the following command:

sudo systemctl restart docker.socket docker.service;

Ubuntu – Overwrite dockerd default settings

Trying to create a new bridge on docker, we got the following error

$ docker-compose up -d;
Creating network "docker-compose_new_bridge" with driver "bridge"
ERROR: could not find an available, non-overlapping IPv4 address pool among the defaults to assign to the network

After investigating, we realized that it was due to some default limitations of docker that did not allow more virtual networks to be created. To overcome the problem, we read that we had to give access to more address space using the /etc/docker/daemon.json.

On Ubuntu that file did not exist so we created it and copied the following content to it:

{
  "default-address-pools": [
    {
      "base": "172.80.0.0/16",
      "size": 24
    },
    {
      "base": "172.90.0.0/16",
      "size": 24
    }
  ]
}

Source: https://docs.docker.com/engine/reference/commandline/dockerd/

This configuration allowed Docker to reserve the network address space 172.80.[0-255].0/24 and 172.90.[0-255].0/24, that provided the daemon a total of 512 networks each owning 256 addresses.

To apply the changes to the daemon, we restarted it:

sudo systemctl restart docker.service;

and then we applied our changes to our docker ecosystem:

docker-compose up -d;

GNU/Linux Fedora 27: Prevent Network Manager from restarting after reboot 1

Recently we were working on a Fedora 27 GNU/Linux box where we needed to completely disable the Network Manager.
Initially, we just stopped the NetworkManager service and then disabled it thinking that it would be enough.
To our surprise after we rebooted the box, we noticed that the Network Manager was active again!

After some research we found out that another service called NetworkManager-wait-online was starting the NetworkManager as some sort of recovery mechanism.
So, in order to permanently block NetworkManager from starting on boot, we disabled NetworkManager-wait-online as well.

In the end our solution to disable the NetworkManager service came down to executing the following commands as root (or using sudo):

systemctl stop NetworkManager;
systemctl stop NetworkManager-wait-online;

systemctl disable NetworkManager;
systemctl disable NetworkManager-wait-online;


Ubuntu: install / start/stop enable/disable ssh server 1

OpenSSH is a freely available version of the Secure Shell (SSH) protocol family of tools for remotely controlling, or transferring files between, computers.

Install SSH server

To install the openssh-server on an Ubuntu, you need execute the following command as root or using sudo:

apt-get install openssh-server -y;

Disable SSH server

To disable the ssh service, execute the following command as root or using sudo:

systemctl disable ssh;

Enable SSH server

To enable the ssh service, execute the following command as root or using sudo:

systemctl enable ssh;

Stop SSH server

To stop (or deactivate) the ssh service, execute the following command as root or using sudo:

systemctl stop ssh;

Start SSH server

To start (or activate) the ssh service, execute the following command as root or using sudo:

systemctl start ssh;

Status of SSH server

To check the status of the ssh service, execute the following command as root or using sudo:

systemctl status ssh;

CONCEPTS

In a nutshell:

  • enabled is a service that is configured to start when the system boots
  • disabled is a service that is configured to not start when the system boots
  • active is a service that is currently running
  • inactive is a service that is currently stopped and may be disabled, but it can be started and become active

In much more detail:

systemd provides a dependency system between various entities called “units” of 12 different types. Units encapsulate various objects that are relevant for system boot-up and maintenance. The majority of units are configured in unit configuration files, whose syntax and basic set of options is described in systemd.unit(5), however some are created automatically from other configuration, dynamically from system state or programmatically at runtime. Units may be “active” (meaning started, bound, plugged in, …, depending on the unit type, see below), or “inactive” (meaning stopped, unbound, unplugged, …), as well as in the process of being activated or deactivated, i.e. between the two states (these states are called “activating”, “deactivating”). A special “failed” state is available as well, which is very similar to “inactive” and is entered when the service failed in some way (process returned error code on exit, or crashed, or an operation timed out). If this state is entered, the cause will be logged, for later reference. Note that the various unit types may have a number of additional substates, which are mapped to the five generalized unit states described here.
— From man systemd