Monthly Archives: June 2022

ewf-tools and Ubuntu 1

 in Applications / GNU/Linux  tagged bash / e01 / ewf / ewf-tools / ewfmount / forensic / GNU/Linux / ubuntu by

Recently, we installed the ewf-tools package from the Ubuntu repositories:

sudo apt-get install ewf-tools;

When we tried to use it, we got the following errors:

ewfmount ./DISK.E01 /tmp/disk/
ewfmount 20140807

Unable to open source image(s)
libcdata_internal_array_resize: invalid entries size value exceeds maximum.
libcdata_array_resize: unable to resize array.
libmfdata_list_resize: unable to resize elements array.
libewf_segment_file_read_volume_section: unable to resize chunk table list.
libewf_handle_open_read_segment_files: unable to read section: volume.
libewf_handle_open_file_io_pool: unable to read segment files.
libewf_handle_open: unable to open handle using a file IO pool.
mount_handle_open: unable to open file(s).

To fix the issue, we uninstalled ewf-tools then installed the following packages:

sudo apt remove ewf-tools;
sudo apt-get install libfuse-dev libfuse2 uuid-dev lbzip2 python3-wchartype;
sudo apt-get install ewf-tools;

Finally, we reinstalled ewf-tools , and this time they worked!

Note

We also downloaded the latest version from the repository, built the code, and tried to use that package with the same result. The code from the repository had the same problem, which worked after we installed the packages mentioned above. For this reason, we believe the problem is not a matter of the version but rather a matter of configuration and dependencies.

An example of MySQL code that executes TRIM() to remove a prefix and/or a suffix from all entries that match a WHERE clause

 in Programming / SQL  tagged MariaDB / mysql / prefix / suffix / trim by

The following code will remove the prefix http://wow.example.com from all rows that match the where clause:

Update `my_table`
set
`my_column` = TRIM(
  LEADING 'http://wow.example.com'
  FROM `my_column`)
WHERE (`my_column` LIKE '%http://wow.example.com/%');

The next block will remove the suffix index.php from all entries that match the where clause:

Update `my_table`
set
`my_column` = TRIM(
  TRAILING 'index.php'
  FROM `my_column`)
WHERE (`my_column` LIKE '%/index.php%');

In case we need to remove the string needle both from the prefix and the suffix while using a where clause, we can use the following code:

Update `my_table`
set
`my_column` = TRIM(
  BOTH 'needle'
  FROM `my_column`)
WHERE (`my_column` LIKE '%needle%');

youtube-dl and embedded Vimeo

 in Applications  tagged domain / referer / vimeo / youtube-dl by

Youtube-DL being an amazing tool as it is, has a way to download Vimeo videos that give the following error:

Cannot download embed-only video without embedding URL. Please call youtube-dl with the URL of the page that embeds this video.

To do so, we needed to use the --referer parameter with the value of the website that we found the embedded video on. For example:

youtube-dl -v "https://player.vimeo.com/video/622fa342f" --referer "https://example.com/courses/intro/hello/";

The --referer parameter specifies a custom referer and can be used if the video access is restricted to one domain.

(youtube-dl) tux@bob:~$ youtube-dl -v "https://player.vimeo.com/video/622fa342f"
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['-v', 'https://player.vimeo.com/video/622fa342f']
[debug] Encodings: locale UTF-8, fs utf-8, out utf-8, pref UTF-8
[debug] youtube-dl version 2021.12.17
[debug] Python version 3.9.7 (CPython) - Linux-5.13.0-44-generic-x86_64-with-glibc2.31
[debug] exe versions: ffmpeg present, ffprobe present
[debug] Proxy map: {}
 622fa342f: Downloading webpage
ERROR: Cannot download embed-only video without embedding URL. Please call youtube-dl with the URL of the page that embeds this video.
Traceback (most recent call last):
  File "/home/tux/anaconda3/envs/youtube-dl/lib/python3.9/site-packages/youtube_dl/extractor/common.py", line 634, in _request_webpage
    return self._downloader.urlopen(url_or_request)
  File "/home/tux/anaconda3/envs/youtube-dl/lib/python3.9/site-packages/youtube_dl/YoutubeDL.py", line 2288, in urlopen
    return self._opener.open(req, timeout=self._socket_timeout)
  File "/home/tux/anaconda3/envs/youtube-dl/lib/python3.9/urllib/request.py", line 523, in open
    response = meth(req, response)
  File "/home/tux/anaconda3/envs/youtube-dl/lib/python3.9/urllib/request.py", line 632, in http_response
    response = self.parent.error(
  File "/home/tux/anaconda3/envs/youtube-dl/lib/python3.9/urllib/request.py", line 561, in error
    return self._call_chain(*args)
  File "/home/tux/anaconda3/envs/youtube-dl/lib/python3.9/urllib/request.py", line 494, in _call_chain
    result = func(*args)
  File "/home/tux/anaconda3/envs/youtube-dl/lib/python3.9/urllib/request.py", line 641, in http_error_default
    raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 403: Forbidden

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/tux/anaconda3/envs/youtube-dl/lib/python3.9/site-packages/youtube_dl/extractor/vimeo.py", line 636, in _real_extract
    webpage, urlh = self._download_webpage_handle(
  File "/home/tux/anaconda3/envs/youtube-dl/lib/python3.9/site-packages/youtube_dl/extractor/common.py", line 667, in _download_webpage_handle
    urlh = self._request_webpage(url_or_request, video_id, note, errnote, fatal, data=data, headers=headers, query=query, expected_status=expected_status)
  File "/home/tux/anaconda3/envs/youtube-dl/lib/python3.9/site-packages/youtube_dl/extractor/common.py", line 652, in _request_webpage
    raise ExtractorError(errmsg, sys.exc_info()[2], cause=err)
youtube_dl.utils.ExtractorError: Unable to download webpage: HTTP Error 403: Forbidden (caused by <HTTPError 403: 'Forbidden'>); please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; see  https://yt-dl.org/update  on how to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
Traceback (most recent call last):
  File "/home/tux/anaconda3/envs/youtube-dl/lib/python3.9/site-packages/youtube_dl/extractor/common.py", line 634, in _request_webpage
    return self._downloader.urlopen(url_or_request)
  File "/home/tux/anaconda3/envs/youtube-dl/lib/python3.9/site-packages/youtube_dl/YoutubeDL.py", line 2288, in urlopen
    return self._opener.open(req, timeout=self._socket_timeout)
  File "/home/tux/anaconda3/envs/youtube-dl/lib/python3.9/urllib/request.py", line 523, in open
    response = meth(req, response)
  File "/home/tux/anaconda3/envs/youtube-dl/lib/python3.9/urllib/request.py", line 632, in http_response
    response = self.parent.error(
  File "/home/tux/anaconda3/envs/youtube-dl/lib/python3.9/urllib/request.py", line 561, in error
    return self._call_chain(*args)
  File "/home/tux/anaconda3/envs/youtube-dl/lib/python3.9/urllib/request.py", line 494, in _call_chain
    result = func(*args)
  File "/home/tux/anaconda3/envs/youtube-dl/lib/python3.9/urllib/request.py", line 641, in http_error_default
    raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 403: Forbidden

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/tux/anaconda3/envs/youtube-dl/lib/python3.9/site-packages/youtube_dl/extractor/vimeo.py", line 636, in _real_extract
    webpage, urlh = self._download_webpage_handle(
  File "/home/tux/anaconda3/envs/youtube-dl/lib/python3.9/site-packages/youtube_dl/extractor/common.py", line 667, in _download_webpage_handle
    urlh = self._request_webpage(url_or_request, video_id, note, errnote, fatal, data=data, headers=headers, query=query, expected_status=expected_status)
  File "/home/tux/anaconda3/envs/youtube-dl/lib/python3.9/site-packages/youtube_dl/extractor/common.py", line 652, in _request_webpage
    raise ExtractorError(errmsg, sys.exc_info()[2], cause=err)
youtube_dl.utils.ExtractorError: Unable to download webpage: HTTP Error 403: Forbidden (caused by <HTTPError 403: 'Forbidden'>); please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; see  https://yt-dl.org/update  on how to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/tux/anaconda3/envs/youtube-dl/lib/python3.9/site-packages/youtube_dl/YoutubeDL.py", line 815, in wrapper
    return func(self, *args, **kwargs)
  File "/home/tux/anaconda3/envs/youtube-dl/lib/python3.9/site-packages/youtube_dl/YoutubeDL.py", line 836, in __extract_info
    ie_result = ie.extract(url)
  File "/home/tux/anaconda3/envs/youtube-dl/lib/python3.9/site-packages/youtube_dl/extractor/common.py", line 534, in extract
    ie_result = self._real_extract(url)
  File "/home/tux/anaconda3/envs/youtube-dl/lib/python3.9/site-packages/youtube_dl/extractor/vimeo.py", line 643, in _real_extract
    raise ExtractorError(
youtube_dl.utils.ExtractorError: Cannot download embed-only video without embedding URL. Please call youtube-dl with the URL of the page that embeds this video.

Incorrect definition of table mysql.column_stats: expected column 1

 in Docker  tagged docker / MariaDB / mysql / mysql_upgrade by 
2022-06-07  7:17:02 3051 [ERROR] Incorrect definition of table mysql.column_stats: expected column 'hist_type' at position 9 to have type enum('SINGLE_PREC_HB','DOUBLE_PREC_HB','JSON_HB'), found type enum('SINGLE_PREC_HB','DOUBLE_PREC_HB').
2022-06-07  7:17:02 3051 [ERROR] Incorrect definition of table mysql.column_stats: expected column 'histogram' at position 10 to have type longblob, found type varbinary(255).

While checking the logs of a MariaDB docker container, we found the above error lines repeating thousands of times. It appears that there was an issue during the migration of the database to a newer version. The solution was to manually execute the command mysql_upgrade. To execute it, we first had to gain access to a shell inside the container, we did that using docker exec -it CONTAINER_NAME /bin/bash as below:

# Gain shell access to the database container
docker exec -it mariadb_alpha /bin/bash;
# In the shell of the container, we executed the following to automatically fix a variety of problems/errors
mysql_upgrade --user=root --password;