Applications


Bind for 0.0.0.0:443 failed: port is already allocated

On a Docker installation that we have, we updated the image files for our containers using the following command:

docker images --format "{{.Repository}}:{{.Tag}}" | grep ':latest' | xargs -L1 docker pull;

Then we tried to update our container, as usual, using the docker-compose command.

export COMPOSE_HTTP_TIMEOUT=180; # We extend the timeout to ensure there is enough time for all containers to start
docker-compose up -d --remove-orphans;

Unfortunately, we got the following error:

export COMPOSE_HTTP_TIMEOUT=180;
docker-compose up -d --remove-orphans;

Starting entry ... 
Starting entry ... error

ERROR: for entry  Cannot start service entry: driver failed programming external connectivity on endpoint entry (d3a5d95f55c4e872801e92b1f32d9693553bd553c414a371b8ba903cb48c2bd5): Bind for 0.0.0.0:443 failed: port is already allocated

ERROR: for entry  Cannot start service entry: driver failed programming external connectivity on endpoint entry (d3a5d95f55c4e872801e92b1f32d9693553bd553c414a371b8ba903cb48c2bd5): Bind for 0.0.0.0:443 failed: port is already allocated
ERROR: Encountered errors while bringing up the project.

We used the docker container ls command to check which container was hoarding port 443, but none was doing so. Because of this, we assumed that docker ran into a bug. The first step we took (and the last) which solved the problem was to restart the docker service as follows:

sudo service docker restart;

This command was enough to fix our problem without messing with docker further.


ewf-tools and Ubuntu 1

Recently, we installed the ewf-tools package from the Ubuntu repositories:

sudo apt-get install ewf-tools;

When we tried to use it, we got the following errors:

ewfmount ./DISK.E01 /tmp/disk/
ewfmount 20140807

Unable to open source image(s)
libcdata_internal_array_resize: invalid entries size value exceeds maximum.
libcdata_array_resize: unable to resize array.
libmfdata_list_resize: unable to resize elements array.
libewf_segment_file_read_volume_section: unable to resize chunk table list.
libewf_handle_open_read_segment_files: unable to read section: volume.
libewf_handle_open_file_io_pool: unable to read segment files.
libewf_handle_open: unable to open handle using a file IO pool.
mount_handle_open: unable to open file(s).

To fix the issue, we uninstalled ewf-tools then installed the following packages:

sudo apt remove ewf-tools;
sudo apt-get install libfuse-dev libfuse2 uuid-dev lbzip2 python3-wchartype;
sudo apt-get install ewf-tools;

Finally, we reinstalled ewf-tools , and this time they worked!

Note

We also downloaded the latest version from the repository, built the code, and tried to use that package with the same result. The code from the repository had the same problem, which worked after we installed the packages mentioned above. For this reason, we believe the problem is not a matter of the version but rather a matter of configuration and dependencies.


youtube-dl and embedded Vimeo

Youtube-DL being an amazing tool as it is, has a way to download Vimeo videos that give the following error:

Cannot download embed-only video without embedding URL. Please call youtube-dl with the URL of the page that embeds this video.

To do so, we needed to use the --referer parameter with the value of the website that we found the embedded video on. For example:

youtube-dl -v "https://player.vimeo.com/video/622fa342f" --referer "https://example.com/courses/intro/hello/";

The --referer parameter specifies a custom referer and can be used if the video access is restricted to one domain.

(youtube-dl) [email protected]:~$ youtube-dl -v "https://player.vimeo.com/video/622fa342f"
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['-v', 'https://player.vimeo.com/video/622fa342f']
[debug] Encodings: locale UTF-8, fs utf-8, out utf-8, pref UTF-8
[debug] youtube-dl version 2021.12.17
[debug] Python version 3.9.7 (CPython) - Linux-5.13.0-44-generic-x86_64-with-glibc2.31
[debug] exe versions: ffmpeg present, ffprobe present
[debug] Proxy map: {}
 622fa342f: Downloading webpage
ERROR: Cannot download embed-only video without embedding URL. Please call youtube-dl with the URL of the page that embeds this video.
Traceback (most recent call last):
  File "/home/tux/anaconda3/envs/youtube-dl/lib/python3.9/site-packages/youtube_dl/extractor/common.py", line 634, in _request_webpage
    return self._downloader.urlopen(url_or_request)
  File "/home/tux/anaconda3/envs/youtube-dl/lib/python3.9/site-packages/youtube_dl/YoutubeDL.py", line 2288, in urlopen
    return self._opener.open(req, timeout=self._socket_timeout)
  File "/home/tux/anaconda3/envs/youtube-dl/lib/python3.9/urllib/request.py", line 523, in open
    response = meth(req, response)
  File "/home/tux/anaconda3/envs/youtube-dl/lib/python3.9/urllib/request.py", line 632, in http_response
    response = self.parent.error(
  File "/home/tux/anaconda3/envs/youtube-dl/lib/python3.9/urllib/request.py", line 561, in error
    return self._call_chain(*args)
  File "/home/tux/anaconda3/envs/youtube-dl/lib/python3.9/urllib/request.py", line 494, in _call_chain
    result = func(*args)
  File "/home/tux/anaconda3/envs/youtube-dl/lib/python3.9/urllib/request.py", line 641, in http_error_default
    raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 403: Forbidden

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/tux/anaconda3/envs/youtube-dl/lib/python3.9/site-packages/youtube_dl/extractor/vimeo.py", line 636, in _real_extract
    webpage, urlh = self._download_webpage_handle(
  File "/home/tux/anaconda3/envs/youtube-dl/lib/python3.9/site-packages/youtube_dl/extractor/common.py", line 667, in _download_webpage_handle
    urlh = self._request_webpage(url_or_request, video_id, note, errnote, fatal, data=data, headers=headers, query=query, expected_status=expected_status)
  File "/home/tux/anaconda3/envs/youtube-dl/lib/python3.9/site-packages/youtube_dl/extractor/common.py", line 652, in _request_webpage
    raise ExtractorError(errmsg, sys.exc_info()[2], cause=err)
youtube_dl.utils.ExtractorError: Unable to download webpage: HTTP Error 403: Forbidden (caused by <HTTPError 403: 'Forbidden'>); please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; see  https://yt-dl.org/update  on how to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
Traceback (most recent call last):
  File "/home/tux/anaconda3/envs/youtube-dl/lib/python3.9/site-packages/youtube_dl/extractor/common.py", line 634, in _request_webpage
    return self._downloader.urlopen(url_or_request)
  File "/home/tux/anaconda3/envs/youtube-dl/lib/python3.9/site-packages/youtube_dl/YoutubeDL.py", line 2288, in urlopen
    return self._opener.open(req, timeout=self._socket_timeout)
  File "/home/tux/anaconda3/envs/youtube-dl/lib/python3.9/urllib/request.py", line 523, in open
    response = meth(req, response)
  File "/home/tux/anaconda3/envs/youtube-dl/lib/python3.9/urllib/request.py", line 632, in http_response
    response = self.parent.error(
  File "/home/tux/anaconda3/envs/youtube-dl/lib/python3.9/urllib/request.py", line 561, in error
    return self._call_chain(*args)
  File "/home/tux/anaconda3/envs/youtube-dl/lib/python3.9/urllib/request.py", line 494, in _call_chain
    result = func(*args)
  File "/home/tux/anaconda3/envs/youtube-dl/lib/python3.9/urllib/request.py", line 641, in http_error_default
    raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 403: Forbidden

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/tux/anaconda3/envs/youtube-dl/lib/python3.9/site-packages/youtube_dl/extractor/vimeo.py", line 636, in _real_extract
    webpage, urlh = self._download_webpage_handle(
  File "/home/tux/anaconda3/envs/youtube-dl/lib/python3.9/site-packages/youtube_dl/extractor/common.py", line 667, in _download_webpage_handle
    urlh = self._request_webpage(url_or_request, video_id, note, errnote, fatal, data=data, headers=headers, query=query, expected_status=expected_status)
  File "/home/tux/anaconda3/envs/youtube-dl/lib/python3.9/site-packages/youtube_dl/extractor/common.py", line 652, in _request_webpage
    raise ExtractorError(errmsg, sys.exc_info()[2], cause=err)
youtube_dl.utils.ExtractorError: Unable to download webpage: HTTP Error 403: Forbidden (caused by <HTTPError 403: 'Forbidden'>); please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; see  https://yt-dl.org/update  on how to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/tux/anaconda3/envs/youtube-dl/lib/python3.9/site-packages/youtube_dl/YoutubeDL.py", line 815, in wrapper
    return func(self, *args, **kwargs)
  File "/home/tux/anaconda3/envs/youtube-dl/lib/python3.9/site-packages/youtube_dl/YoutubeDL.py", line 836, in __extract_info
    ie_result = ie.extract(url)
  File "/home/tux/anaconda3/envs/youtube-dl/lib/python3.9/site-packages/youtube_dl/extractor/common.py", line 534, in extract
    ie_result = self._real_extract(url)
  File "/home/tux/anaconda3/envs/youtube-dl/lib/python3.9/site-packages/youtube_dl/extractor/vimeo.py", line 643, in _real_extract
    raise ExtractorError(
youtube_dl.utils.ExtractorError: Cannot download embed-only video without embedding URL. Please call youtube-dl with the URL of the page that embeds this video.


Incorrect definition of table mysql.column_stats: expected column

2022-06-07  7:17:02 3051 [ERROR] Incorrect definition of table mysql.column_stats: expected column 'hist_type' at position 9 to have type enum('SINGLE_PREC_HB','DOUBLE_PREC_HB','JSON_HB'), found type enum('SINGLE_PREC_HB','DOUBLE_PREC_HB').
2022-06-07  7:17:02 3051 [ERROR] Incorrect definition of table mysql.column_stats: expected column 'histogram' at position 10 to have type longblob, found type varbinary(255).

While checking the logs of a MariaDB docker container, we found the above error lines repeating thousands of times. It appears that there was an issue during the migration of the database to a newer version. The solution was to manually execute the command mysql_upgrade. To execute it, we first had to gain access to a shell inside the container, we did that using docker exec -it CONTAINER_NAME /bin/bash as below:

# Gain shell access to the database container
docker exec -it mariadb_alpha /bin/bash;
# In the shell of the container, we executed the following to automatically fix a variety of problems/errors
mysql_upgrade --user=root --password;

Unknown/unsupported storage engine: InnoDB

2022-05-30 06:09:48+00:00 [Note] [Entrypoint]: Entrypoint script for MariaDB Server 1:10.8.3+maria~jammy started.
2022-05-30 06:09:48+00:00 [Note] [Entrypoint]: Switching to dedicated user 'mysql'
2022-05-30 06:09:48+00:00 [Note] [Entrypoint]: Entrypoint script for MariaDB Server 1:10.8.3+maria~jammy started.
2022-05-30 06:09:49+00:00 [Note] [Entrypoint]: MariaDB upgrade information missing, assuming required
2022-05-30 06:09:49+00:00 [Note] [Entrypoint]: MariaDB upgrade (mariadb-upgrade) required, but skipped due to $MARIADB_AUTO_UPGRADE setting
2022-05-30  6:09:49 0 [Note] mariadbd (server 10.8.3-MariaDB-1:10.8.3+maria~jammy) starting as process 1 ...
2022-05-30  6:09:49 0 [Note] InnoDB: Compressed tables use zlib 1.2.11
2022-05-30  6:09:49 0 [Note] InnoDB: Number of transaction pools: 1
2022-05-30  6:09:49 0 [Note] InnoDB: Using crc32 + pclmulqdq instructions
2022-05-30  6:09:49 0 [Note] mariadbd: O_TMPFILE is not supported on /tmp (disabling future attempts)
2022-05-30  6:09:49 0 [Warning] mariadbd: io_uring_queue_init() failed with ENOSYS: check seccomp filters, and the kernel version (newer than 5.1 required)
2022-05-30  6:09:49 0 [Warning] InnoDB: liburing disabled: falling back to innodb_use_native_aio=OFF
2022-05-30  6:09:49 0 [Note] InnoDB: Initializing buffer pool, total size = 128.000MiB, chunk size = 2.000MiB
2022-05-30  6:09:49 0 [Note] InnoDB: Completed initialization of buffer pool
2022-05-30  6:09:49 0 [Note] InnoDB: File system buffers for log disabled (block size=512 bytes)
2022-05-30  6:09:49 0 [ERROR] InnoDB: Upgrade after a crash is not supported. The redo log was created with MariaDB 10.5.4.
2022-05-30  6:09:49 0 [ERROR] InnoDB: Plugin initialization aborted with error Generic error
2022-05-30  6:09:49 0 [Note] InnoDB: Starting shutdown...
2022-05-30  6:09:49 0 [ERROR] Plugin 'InnoDB' init function returned error.
2022-05-30  6:09:49 0 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
2022-05-30  6:09:49 0 [Note] Plugin 'FEEDBACK' is disabled.
2022-05-30  6:09:49 0 [ERROR] Unknown/unsupported storage engine: InnoDB
2022-05-30  6:09:49 0 [ERROR] Aborting

Recently, we were working on a MariaDB installation in Docker which was using the latest version of the container. The definition in our configuration file was as follows:

# An excerpt from our docker-compose.yml
mariadb_alpha:
    depends_on:
      - another_container
    image: mariadb
    container_name: mariadb_alpha
    networks:
      - mariadb_alpha
    volumes:
      - /alpha/mysql:/var/lib/mysql
    restart: unless-stopped
    environment:
      MYSQL_ROOT_PASSWORD: qwerty
      MYSQL_DATABASE: aplha
      MYSQL_USER: user
      MYSQL_PASSWORD: password

After an update, the database stopped working and the logs were giving the above errors. Specifically, we got the error that InnoDB was an unknown or unsupported storage engine which is really bad! The command we used to view the logs is the following:

# We used the following docker command to view the logs of the container
docker container logs mariadb_alpha;

We noticed the following line from the records, which was extremely useful:

InnoDB: Upgrade after a crash is not supported. The redo log was created with MariaDB 10.5.4.

From this information, we were able to understand that the last time the database functioned properly, it was using MariaDB version 10.5.4. By visiting the official docker image website for MariaDB, we were able to see that there was a version tagged 10.5. We modified our YML file and changed the image of the container to the one below which uses the 10.5 tagged image:

# An excerpt from our docker-compose.yml
mariadb_alpha:
    ...
    image: mariadb:10.5
    ...

Then, we rebuilt our container using the docker-compose command:

#We increase the timeout to avoid issues
export COMPOSE_HTTP_TIMEOUT=180;
docker-compose up -d --remove-orphans;

After the setup was complete, we were able to see that the container was working as expected! We reverted the change in the YML file back to image: mariadb and executed docker-compose once more. The MariaDB container was updated to the latest version and was working as expected again!