docker

Do not use the snap version of docker for Ray on Ubuntu 20.04LTS

 in GNU/Linux  tagged docker / ray / snap by

If you are trying to deploy a local Ray cluster on Ubuntu machines and you are getting the following error:

Shared connection to 192.168.1.74 closed.
     Running docker exec ray_container printenv HOME
       Full command is ssh -tt -i ~/.ssh/id_rsa -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o IdentitiesOnly=yes -o ExitOnForwardFailure=yes -o ServerAliveInterval=5 -o ServerAliveCountMax=3 -o ControlMaster=auto -o ControlPath=/tmp/ray_ssh_71415f9f14/c21f969b5f/%C -o ControlPersist=10s -o ConnectTimeout=120s tux@192.168.1.74 bash --login -c -i 'true && source ~/.bashrc && export OMP_NUM_THREADS=1 PYTHONWARNINGS=ignore && (docker exec ray_container printenv HOME)'
 Shared connection to 192.168.1.74 closed.
     Running docker cp /tmp/ray_tmp_mount/default/~/ray_bootstrap_config.yaml ray_container:/home/ray/ray_bootstrap_config.yaml
       Full command is ssh -tt -i ~/.ssh/id_rsa -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o IdentitiesOnly=yes -o ExitOnForwardFailure=yes -o ServerAliveInterval=5 -o ServerAliveCountMax=3 -o ControlMaster=auto -o ControlPath=/tmp/ray_ssh_71415f9f14/c21f969b5f/%C -o ControlPersist=10s -o ConnectTimeout=120s tux@192.168.1.74 bash --login -c -i 'true && source ~/.bashrc && export OMP_NUM_THREADS=1 PYTHONWARNINGS=ignore && (docker cp /tmp/ray_tmp_mount/default/~/ray_bootstrap_config.yaml ray_container:/home/ray/ray_bootstrap_config.yaml)'
 lstat /tmp/ray_tmp_mount/default/~: no such file or directory
 Shared connection to 192.168.1.74 closed.
 2021-06-09 11:41:09,299    INFO node_provider.py:93 -- ClusterState: Writing cluster state: ['192.168.1.70', '192.168.1.74']

You might need to consider removing the snap version of docker and follow the official instructions of docker.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
# From https://docs.docker.com/engine/install/ubuntu/
sudo apt-get remove docker docker-engine docker.io containerd runc;
sudo apt-get update;
sudo apt-get install apt-transport-https ca-certificates curl gnupg lsb-release;
 
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /usr/share/keyrings/docker-archive-keyring.gpg
echo "deb [arch=amd64 signed-by=/usr/share/keyrings/docker-archive-keyring.gpg] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
 
sudo apt-get update
sudo apt-get install docker-ce docker-ce-cli containerd.io
 
sudo addgroup --system docker
sudo adduser $USER docker
newgrp docker
sudo systemctl restart docker

Horrible Solution: How to delete all docker logs 2

 in Docker  tagged delete logs / docker / truncate by

Recently, we needed to delete the logs of a running docker setup. To do so, we used the following horrible solution:

First, we executed docker system info | grep "Docker Root Dir";to get the installation path of docker. It resulted to the following:

Docker Root Dir: /var/lib/docker

Then, we truncated all log files using the following command while executing as root:

truncate -s 0 /var/lib/docker/containers/*/*-json.log;

Bonus

To make the command a onliner, use the following:

1
sudo sh -c 'truncate -s 0 $(docker system info | grep "Docker Root Dir" | cut -d ":" -f2 | cut -d " " -f2-)/containers/*/*-json.log';

How to update all pulled Docker images that are tagged as latest

 in Docker  tagged all images / docker / latest / pull by

Recently, we moved a client to Docker and we needed to give them a way to automagically update all “latest” Docker images.
Since Docker does not have a single command to update all pulled images we used this one-liner to update all images at once:

1
docker images --format "{{.Repository}}:{{.Tag}}" | grep ':latest' | xargs -L1 docker pull;

The above command will:

  1. Print all images in the format RepositoryName:Tag
  2. Then it will filter all lines that end with the suffix :latest (which is the tag we are interested in)
  3. Finally, for each result (which is one per line) it will be fed  via the command xargs -L1 to the command docker pull

Please note that you cannot really update an existing container using docker commands, what you need to do is actually:

  1. Stop the container whose image was updated
  2. Delete it
  3. Recreate it using the parameters of the previous container

As you understand, it is a good practice to save all of your data in volumes outside the container to make the update process easy.

For example, below you will find the commands using which we updated the jwilder/nginx-proxy and the jrcs/letsencrypt-nginx-proxy-companion images along with the two containers that were using them:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
docker container stop nginx-proxy nginx-letsencrypt;
docker container rm nginx-proxy nginx-letsencrypt;
docker run -d -p 443:443 \
     --name nginx-proxy \
     --net reverse-proxy \
     -v $HOME/certs:/etc/nginx/certs:ro \
     -v /etc/nginx/vhost.d \
     -v /usr/share/nginx/html \
     -v /var/run/docker.sock:/tmp/docker.sock:ro \
     -v $HOME/my_proxy.conf:/etc/nginx/conf.d/my_proxy.conf:ro \
     --label com.github.jrcs.letsencrypt_nginx_proxy_companion.nginx_proxy=true \
     jwilder/nginx-proxy:latest;
 
docker run -d \
     --name nginx-letsencrypt \
     --net reverse-proxy \
     --volumes-from nginx-proxy \
     -v $HOME/certs:/etc/nginx/certs:rw \
     -v /var/run/docker.sock:/var/run/docker.sock:ro \
     jrcs/letsencrypt-nginx-proxy-companion:latest;