timeout


Python 3 program that accepts a path to a binary file executes it, captures all output, and kills it once a part of the output matches a pattern or a timeout occurs

import argparse
import subprocess
import shlex
import signal


# Function to execute a binary and kill it when a needle is found in its output or after a timeout
# Parameters:
#   binary_path: Path to the binary to execute
#   binary_parameters: Parameters to pass to the binary
#   needle: Needle to match in the output
#   timeout: Timeout after which to kill the process
# Returns:
#   The return code of the binary
# Notes:
#   This function uses subprocess.Popen to execute the binary in a sandbox and capture its output.
#   It then loops through the output lines and checks if the needle is found in the output.
#   If the needle is found, the process is killed and the function returns. If the needle is not found,
#   the function waits for the process to finish and returns its return code.
#   Since popen is used, it does not accept a timeout parameter.
#   Instead, the timeout is implemented by using the timeout command to run the binary.
#   We could not use the timeout parameter of subprocess.run because it blocks the execution of the script
#   until the process finishes.
#   This means that we would not be able to capture the output of the process.
def using_popen(binary_path, binary_parameters, needle, timeout):
    # Define the command to run the binary file
    command = f"timeout {timeout} {binary_path} {binary_parameters}"
    # Use subprocess to execute the command in a sandbox and capture its output
    process = subprocess.Popen(shlex.split(command), stdout=subprocess.PIPE, stderr=subprocess.PIPE)

    # Loop through the output lines and process them
    while True:
        output = process.stderr.readline().decode().strip()
        if output == "" and process.poll() is not None:
            break
        if output != "":
            # Process the output here
            print(output)
            # Check if the output contains the keyword
            if needle in output:
                # Kill the process if the keyword is found
                process.send_signal(signal.SIGINT)
                break
    return process.returncode


if __name__ == "__main__":
    # Parse command line arguments
    parser = argparse.ArgumentParser(
        description='Execute a binary and kill it either when a needle is found in its output or after a timeout')
    parser.add_argument('-e', '--executable', type=str, help='Path to the executable to run')
    parser.add_argument('-p', '--parameters', type=str, help='Parameters to pass to the executable')
    parser.add_argument('-n', '--needle', type=str, help='Needle to match in the output')
    parser.add_argument('-t', '--timeout', type=str, help='Timeout after which to kill the process')
    # Example usage
    # python main.py -e "python" -p "-u -m http.server" -n "404" -t "15s"

    args = parser.parse_args()
    # Execute the binary and capture its output
    return_code = using_popen(args.executable, args.parameters, args.needle, args.timeout)

    # Print the http server's return code
    print("Http server returned with code:", return_code)

The given code is a Python script that executes a binary and kills it when a specified string is found in its output or after a specified timeout. Let’s break down the code into its constituent parts.

First, the script imports several Python modules – argparse, subprocess, shlex, and signal.

argparse is a module that makes it easy to write user-friendly command-line interfaces. It parses the command-line arguments and options specified by the user and returns them in a convenient format.

subprocess is a module that allows you to spawn new processes, connect to their input/output/error pipes, and obtain their return codes.

shlex is a module that provides a simple way to parse command-line strings into a list of tokens, which can then be passed to subprocess.Popen().

signal is a module that provides mechanisms to handle asynchronous events such as interrupts and signals.

Next, the script defines a function called using_popen() that takes four parameters: binary_path, binary_parameters, needle, and timeout. The function first constructs a command string that combines the binary_path and binary_parameters arguments and adds a timeout command to limit the execution time of the process.

The subprocess.Popen() function is then called to create a new process with the constructed command. The stdout and stderr arguments are set to subprocess.PIPE so that the output of the process can be captured.

A while loop is then entered to read the process’s stderr output line by line. The decode() method is used to convert the byte string output to a regular string, and the strip() method is used to remove any whitespace characters from the beginning and end of the string.

If the output string is empty and the process has finished running (process.poll() is not None), the loop is terminated. Otherwise, if the output string is not empty, it is printed to the console.

If the specified needle string is found in the output, the process is terminated by sending a signal.SIGINT signal to it.

After the loop has completed, the return code of the process is retrieved using process.returncode and returned by the function.

Finally, the script checks if it is being run as the main program using the __name__ attribute. If it is, it uses the argparse module to parse the command-line arguments specified by the user. The using_popen() function is then called with the parsed arguments, and its return code is printed to the console along with a message indicating the completion of the script.

In summary, this script provides a convenient way to execute a binary with specified parameters, limit its execution time, and terminate it early if a specified string is found in its output. It also provides a user-friendly command-line interface for specifying these parameters.


Rough notes on setting up an Ubuntu 22.04LTS server with docker and snap 1

IP allocations

First, we set up a static IP on the network device that would handle all external traffic and a DHCP on the network device that would access the management network, which is connected for maintenance.

To do so, we created the following file:

/etc/netplan/01-netcfg.yaml

using the following command:

sudo nano /etc/netplan/01-netcfg.yaml;

and added the following content to it:

# This file describes the network interfaces available on your system
# For more information, see netplan(5).
network:
  version: 2
  renderer: networkd
  ethernets:
    eth0:
      dhcp4: no
      addresses: [192.168.45.13/24]
      gateway4: 192.168.45.1
      nameservers:
          addresses: [1.1.1.1,8.8.8.8]
    eth1:
      dhcp4: yes

To apply the changes, we executed the following:

sudo netplan apply;

Update everything (the operating system and all packages)

Usually, it is a good idea to update your system before making significant changes to it:

sudo apt update -y; sudo apt upgrade -y; sudo apt autoremove -y;

Install docker via snap

In this setup, we did not use the docker version available on the Ubuntu repositories, we went for the ones from the snap. To install it, we used the following commands:

sudo apt install snapd;
sudo snap install docker;

Increase network pool for docker daemon

To handle the following problem:

ERROR: could not find an available, non-overlapping IPv4 address pool among the defaults to assign to the network

We modified the following file

/var/snap/docker/current/config/daemon.json

using the command:

sudo nano /var/snap/docker/current/config/daemon.json;

and set the content to be as follows:

{
    "log-level":        "error",
    "storage-driver":   "overlay2",
    "default-address-pools": [
        {
            "base": "172.80.0.0/16",
            "size": 24
        },
        {
            "base": "172.90.0.0/16",
            "size": 24
        }
    ]
}

We executed the following command to restart the docker daemon and get the network changes applied:

sudo snap disable docker;
sudo snap enable docker;

Gave access to our user to manage the docker

We added our user to the docker group so that we could manage the docker daemon without sudo rights.

sudo addgroup --system docker;
sudo adduser $USER docker;
newgrp docker;
sudo snap disable docker;
sudo snap enable docker;

After that, we made sure that the access rights to the volumes were correct:

sudo chown -R www-data:www-data /volumes/*
sudo chown -R tux:tux /volumes/letsencrypt/ /volumes/reverse/private/

Deploying

After we copied everything in place, we executed the following command to create our containers and start them with the appropriate networks and volumes:

export COMPOSE_HTTP_TIMEOUT=600;
docker-compose up -d --remove-orphans;

We had to increase the timeout as we were getting the following error:

ERROR: for container_a  UnixHTTPConnectionPool(host='localhost', port=None): Read timed out. (read timeout=60)
ERROR: An HTTP request took too long to complete. Retry with --verbose to obtain debug information.
If you encounter this issue regularly because of slow network conditions, consider setting COMPOSE_HTTP_TIMEOUT to a higher value (current value: 60).

Updating the databases and performing any repairs

First, we connected to a terminal of the database container using the following command:

docker exec -it mariadb_c1 /bin/bash;

From there, we executed the following commands:

mysql_upgrade --user=root --password;
mysqlcheck -p -o --all-databases;

Bulk / Batch stopping docker containers

The following commands will help you stop many docker containers simultaneously. Of course, you can change the command stop to another, for example rm or whatever suits your needs.

You need to keep in mind that if you have dependencies between containers, you might need to execute the commands below more than once.

Stop all docker containers.

docker container stop $(docker container ls -q);
#This command creates a list of all containers.
#Using the -q parameter, we only get back the container ID and not all information about them.
#Then it will stop each container one by one.

Stop specific docker containers using a filter on their name.

docker container stop $(docker container ls -q --filter name=_web);
#This command finds all containers that their name contains _web.
#Using the -q parameter, we only get back the container ID and not all information about them.
#Then it will stop each container one by one.

A personal note

Check the system for things you might need to configure, like a crontab or other services.

A script that handles privileges on the docker volumes

To avoid access problems with the various external volumes we created the mysql user and group on the host machine as follows:

sudo groupadd -g 999 mysql;
sudo useradd -u 999 mysql -g mysql;

Then we execute the following to repair ownership issues with our containers. Please note that this script is custom to a particular installation and might not meet your needs.

#!/bin/bash

sudo chown -R www-data:www-data ~/volumes/*;
sudo chown -R bob:bob ~/volumes/letsencrypt/ ~/volumes/reverse/private/;
find ~/volumes/ -maxdepth 2 -type d -name mysql -exec sudo chown -R mysql:mysql '{}' \;;

Bind for 0.0.0.0:443 failed: port is already allocated

On a Docker installation that we have, we updated the image files for our containers using the following command:

docker images --format "{{.Repository}}:{{.Tag}}" | grep ':latest' | xargs -L1 docker pull;

Then we tried to update our container, as usual, using the docker-compose command.

export COMPOSE_HTTP_TIMEOUT=180; # We extend the timeout to ensure there is enough time for all containers to start
docker-compose up -d --remove-orphans;

Unfortunately, we got the following error:

export COMPOSE_HTTP_TIMEOUT=180;
docker-compose up -d --remove-orphans;

Starting entry ... 
Starting entry ... error

ERROR: for entry  Cannot start service entry: driver failed programming external connectivity on endpoint entry (d3a5d95f55c4e872801e92b1f32d9693553bd553c414a371b8ba903cb48c2bd5): Bind for 0.0.0.0:443 failed: port is already allocated

ERROR: for entry  Cannot start service entry: driver failed programming external connectivity on endpoint entry (d3a5d95f55c4e872801e92b1f32d9693553bd553c414a371b8ba903cb48c2bd5): Bind for 0.0.0.0:443 failed: port is already allocated
ERROR: Encountered errors while bringing up the project.

We used the docker container ls command to check which container was hoarding port 443, but none was doing so. Because of this, we assumed that docker ran into a bug. The first step we took (and the last) which solved the problem was to restart the docker service as follows:

sudo service docker restart;

This command was enough to fix our problem without messing with docker further.