GNU/Linux

Bash: Problem with reading files with spaces in the name using a for loop

 in Bash  tagged bash / find / for / for loop by

Recently we were working on a bash script that was supposed to find and process some files that matched certain criteria. The script would process the files one by one and the criteria would be matched using the find command. To implement our solution, we returned the results of the find back to the for loop in an attempt to keep it simple and human readable.

Our original code was the following:
(do not use it, see explanation below)

for file in `find $search_path -type f -name '*.kml'`; do
  # Formatting KML file to be human friendly.
  xmllint --format "$file" > "$output_path/$file";
done

Soon we realized that we had a very nasty bug, the way we formatted the command it would break filenames that had spaces in them into multiple for loop entries and thus we would get incorrect filenames back to process.

To solve this issue we needed a way to force our loop to read the results of find one line at a time instead of one word at a time. The solution we used in the end was fairly different than the original code as it had the following significant changes:

  • the results of the find command were piped into the loop
  • the loop was not longer a for loop and a while loop was used instead
  • it used the read command that reads one line at a time to fill in the filename variable
    (the -r parameter does not allow backslashes to escape any characters)

Solution

find $search_path -type f -name '*.kml' | 
while read -r file; do
  # Formatting KML file to be human friendly.
  xmllint --format "$file" > "$output_path/$file";
done

Create an encrypted 7zip archive with encrypted header as well (no filenames are visible)

 in Bash  tagged 7z / 7zip / encryption / filenames / headers / password / protection by

In case you come to a scenario where you need to encrypt, password protect the contents of a 7zip archive and make sure that not even the filenames of the contents are visible, 7zip has your back! As you can see in the following example you can implement the above requirements very easily.

7z a -p"pbVfdPs27Dc" -mhe hello.7z file1.bin file2.doc files.*

The structure of the above 7z command is the following:

#Based on: 7z <command> [<switches>...] <archive_name> [<file_names>...]
7z a -p"Some Password!.32@" -mhe <archive_name> [<file_names>...]

To break it down, it goes like this:

  • We used the <command> a, which instructs the tool to add the listed files to the listed archive (if the archive does not exist, it will create it).
  • The <switch> -p, allows you to set the password for the archive.
  • The second <switch> -mhe (or -mhe=on) it enables data and header archive encryption.
    In case you cannot find this switch at the manual, check the examples in the man page (This command works on GNU/Linux, it was tested on Fedora).

A simple way to find which DHCP server gave you an IP

 in GNU/Linux  tagged DHCP / DHCPACK / grep / IP / journalctl by

Recently, we were trying to find which DHCP server was responding to the messages on the network. Using a DHCP-enabled client on a Fedora 26 GNU/Linux we grepped the contents of journalctl to find the DHCP acknowledgment messages (DHCPACK) and figure out the IP of the DHCP server.

The command we used was the following:

sudo journalctl | grep DHCPACK;

And it gave us results such as the ones below:

[user@sys-net ~]$ sudo journalctl | grep DHCPACK
Nov 12 13:08:28 sys-net dhclient[578]: DHCPACK from 10.1.101.252 (xid=0x80ec760c)
Nov 12 13:08:34 sys-net dhclient[720]: DHCPACK from 10.1.101.252 (xid=0x2ed6486f)
Nov 12 11:51:19 sys-net dhclient[1248]: DHCPACK from 10.1.101.252 (xid=0xe3dd491c)
Nov 12 12:02:09 sys-net dhclient[1407]: DHCPACK from 10.1.101.252 (xid=0x1fa42c2d)
Nov 12 12:11:03 sys-net dhclient[1508]: DHCPACK from 10.1.101.252 (xid=0x91c3990a)
Nov 12 12:14:06 sys-net dhclient[1607]: DHCPACK from 10.1.101.252 (xid=0x57ebb515)
Nov 12 12:19:27 sys-net dhclient[1710]: DHCPACK from 10.1.101.252 (xid=0x5450c250)
Nov 12 12:19:39 sys-net dhclient[1776]: DHCPACK from 10.1.101.252 (xid=0x2c38d517)
Nov 12 12:39:53 sys-net dhclient[1837]: DHCPACK from 192.168.1.1 (xid=0xe7a1182d)
Nov 12 12:40:51 sys-net dhclient[1837]: DHCPACK from 192.168.1.1 (xid=0xe7a1182d)
Nov 12 12:41:51 sys-net dhclient[1837]: DHCPACK from 192.168.1.1 (xid=0xe7a1182d)
Nov 12 12:42:44 sys-net dhclient[1837]: DHCPACK from 192.168.1.1 (xid=0xe7a1182d)
Nov 12 12:43:33 sys-net dhclient[1837]: DHCPACK from 192.168.1.1 (xid=0xe7a1182d)
Nov 12 12:44:31 sys-net dhclient[1837]: DHCPACK from 192.168.1.1 (xid=0xe7a1182d)
Nov 12 12:46:20 sys-net dhclient[2053]: DHCPACK from 192.168.1.1 (xid=0xbb006001)

It is important to use sudo or else you will not be seeing messages from other users and the system. As, only users in groups ‘adm’, ‘systemd-journal’, ‘wheel’ can see all messages.

youtube-dl does not work properly on Qubes 4.0 – Fedora 26

 in GNU/Linux  tagged dnf / fedora 26 / qubes / qubes 4.0 / qubes-os / youtube-dl by

While working on a GNU/Linux Fedora 26 virtual machine running under the Qubes 4.0 OS, we installed youtube-dl through dnf to download some media off the net.


sudo dnf install youtube-dl;

When we tried to use it, we got the error youtube_dl.utils.RegexNotFoundError: Unable to extract Initial JS player signature function name. This issue was resolved years ago, so it led us to the conclusion that our version of youtube-dl was out of date. To test this hypothesis, we updated youtube-dl with pip.


sudo pip install --upgrade youtube_dl;

After the update was complete, we tried to use youtube-dl, this time with success!!

Full Logs:

[george@local Music]$ youtube-dl
bash: youtube-dl: command not found...
[george@local Music]$ sudo dnf install youtube-dl
Last metadata expiration check: 0:52:47 ago on Sat Nov 10 10:43:46 2018.
Dependencies resolved.
================================================================================
Package Arch Version Repository Size
================================================================================
Installing:
youtube-dl noarch 2018.04.16-1.fc26 updates 2.6 M

Transaction Summary
================================================================================
Install 1 Package

Total download size: 2.6 M
Installed size: 11 M
Is this ok [y/N]: y
Downloading Packages:
youtube-dl-2018.04.16-1.fc26.noarch.rpm 270 kB/s | 2.6 MB 00:09 
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Total 258 kB/s | 2.6 MB 00:10 
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
Preparing : 1/1 
Installing : youtube-dl-2018.04.16-1.fc26.noarch 1/1 
Running scriptlet: youtube-dl-2018.04.16-1.fc26.noarch 1/1 
Running as unit: run-rfddd15fff2d14d109826a90f59325e97.service
Verifying : youtube-dl-2018.04.16-1.fc26.noarch 1/1 
Notifying dom0 about installed applications

Installed:
youtube-dl.noarch 2018.04.16-1.fc26

Complete!
[george@local Music]$ youtube-dl https://www.youtube.com/watch?list=r6akoO34yUvK8ddtjnzL
 r6akoO34yUvK8ddtjnzL: Downloading webpage
[download] Downloading playlist: How to cook a banana
 playlist How to cook a banana: Downloading 18 videos
[download] Downloading video 1 of 18
 ir86d1hTSv1r: Downloading webpage
 ir86d1hTSv1r: Downloading video info webpage
 ir86d1hTSv1r: Extracting video information
 ir86d1hTSv1r: Downloading js player vfls4aurX
ERROR: Signature extraction failed: Traceback (most recent call last):
File "/usr/lib/python3.6/site-packages/youtube_dl/extractor/youtube.py", line 1191, in _decrypt_signature
video_id, player_url, s
File "/usr/lib/python3.6/site-packages/youtube_dl/extractor/youtube.py", line 1102, in _extract_signature_function
res = self._parse_sig_js(code)
File "/usr/lib/python3.6/site-packages/youtube_dl/extractor/youtube.py", line 1163, in _parse_sig_js
jscode, 'Initial JS player signature function name', group='sig')
File "/usr/lib/python3.6/site-packages/youtube_dl/extractor/common.py", line 808, in _search_regex
raise RegexNotFoundError('Unable to extract %s' % _name)
youtube_dl.utils.RegexNotFoundError: Unable to extract Initial JS player signature function name; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; see https://yt-dl.org/update on how to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
(caused by RegexNotFoundError('Unable to extract \x1b[0;34mInitial JS player signature function name\x1b[0m; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; see https://yt-dl.org/update on how to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.',)); please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; see https://yt-dl.org/update on how to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
[george@local Music]$ sudo pip install --upgrade youtube_dl
WARNING: Running pip install with root privileges is generally not a good idea. Try `pip install --user` instead.
Collecting youtube_dl
Downloading https://files.pythonhosted.org/packages/6c/a4/c2e3fbd8b9c7ccbca3f220c4fb0914926669bf9080fb8f2f4db4811706be/youtube_dl-2018.11.7-py2.py3-none-any.whl (1.8MB)
100% |████████████████████████████████| 1.8MB 355kB/s 
Installing collected packages: youtube-dl
Successfully installed youtube-dl-2018.11.7
[george@local Music]$ youtube-dl https://www.youtube.com/watch?list=r6akoO34yUvK8ddtjnzL
 r6akoO34yUvK8ddtjnzL: Downloading webpage
[download] Downloading playlist: How to cook a banana
 playlist How to cook a banana: Downloading 18 videos
[download] Downloading video 1 of 18
 ir86d1hTSv1r: Downloading webpage
 ir86d1hTSv1r: Downloading video info webpage
 ir86d1hTSv1r: Downloading js player vfls4aurX
WARNING: Requested formats are incompatible for merge and will be merged into mkv.