Bash


Assigning auto-increment IDs to empty fields in a KML/XML file

Recently we were processing some KML files using OpenLayers and at some point we realised that some place-marks were not appearing on the map. After inspecting the debug console and the files more carefully we understood that OpenLayers did not like empty placemark IDs.

To mitigate the problem we wrote the following AWK script that will go over all lines in the KML/XML file, find the empty id fields (id="") and assign them with an auto-increment value. A note here, initially we just replaced all empty IDs with the same value but it seems that OpenLayers does not treat kindly conflicts on IDs and thus we had to go with an auto-increment solution.

# Assigning auto-increment IDs to the placemarkers as openlayers does not show conflicting-ID elements.
awk -i inplace '{
  for(x=1;x<=NF;x++) {
    if($x~/id=""/) {
      sub(/id=""/,"id=\"" (++i) "\"")
    }
  }
}1' "$output_path/$file_name";

Side notes

In case you already have some IDs defined, you would have to make your code a bit more complex… You would first need to find all filled IDs and then you would have two options:

  • empty them and execute the above script
  • or register them and make sure the script does not create conflicting IDs either by starting the variable i from a number greater than the biggest registered ID or making it even harder by filling in the gaps between the already registered IDs..
Advertisements

Add a new line whenever the first column changes

Recently we were processing some results from an SQL query on the command line, we were grouping the results based on the first column of the query results and we wanted to add an empty line between each group of data.

Instead of messing with SQL specifics, we decided to use awk to finish the task which resulted in the following code:

awk -F '|' -v column=1 'NR>1 && $column != previous { print "" } { previous = $column } { print $0 }'

Explanation:

  • -F fs or --field-separator fs: This option allowed us to choose the input field separator, in other words the character that should be used to split our text into columns. We used the value | because that character is the default column delimiter for sqlite.
  • -v var=val or --assign var=val: We used the -v option to define a variable to be used later on in the script. The value 1 was assigned to the variable column before execution of the program began and it was available event to the BEGIN rule of the AWK program. We did this to make the code a bit more modular, we could have just hardcoded the number in.
  • NR>1 && $column != previous { print "" } : Here we defined an if statement that checks two options: First we make sure that we are not on the first line of the input by using the NR>1 (and thus avoid creating an empty line which will be the first line of the output).
    Second, we check that the last value we had for the column of interest did change since last time. (We still did not define the value of previous, it is on the next step). When both statements are true (we are not on the first row and the value of the column in the current row is different than the value of the column in the previous row) it will print out an empty line.
  • { previous = $column }: This part is executed on ALL lines (even the first one) no matter what the values are. What this line does is to translate the value of the column variable from being a number (the index of the column that we are interested in) into the actual value that the column has at that specific line. That value is then copied to the previous variable to allow us to perform the check in the previous point once we move to the next line.
  • { print $0 }: Finally, this part is also executed on all lines and it instructs awk to print the input row whole and as is. This whole part could be replaced by a true value like the value 1. In awk as you see in this example, you define a series of operations. Each operation is constructed by a pattern to be matched and an action. Each pattern is evaluated for each input line, and in the cases where the pattern matches, the action is executed. The user can choose to omit either the pattern or the action for any operation. When a pattern is omitted, the action is executed on every line. When the action is omitted, then awk will execute { print $0 }. So, by adding a true value on its own it will be translated as on each line execute { print $0 } which prints the whole row as is.

Example

1|1|0.0564904019731175
1|2|0.103176086258974
1|3|0.12910406904073
1|4|0.188592489201024
1|5|0.169676224898487
1|6|0.164690820027741
1|7|0.128458728519047
1|8|0.18549773544014
1|9|0.155677575617836
1|10|0.153941343314285
2|1|0.217221158956016
2|2|0.23390973064067
2|3|0.180231657220626
2|4|0.257673927303071
2|5|0.261393785194329
2|6|0.273441488895552
2|7|0.242815632929545
2|8|0.262269697286057
2|9|0.256054399760891
2|10|0.262613705138411
3|1|0.378589461360716
3|2|0.33008177312116
3|3|0.380973166776554
3|4|0.340431190160728
3|5|0.38189416214207
3|6|0.364842933594872
3|7|0.372958396398964
3|8|0.350010176652464
3|9|0.355815612501188
3|10|0.380553180349294

Will become

1|1|0.0564904019731175
1|2|0.103176086258974
1|3|0.12910406904073
1|4|0.188592489201024
1|5|0.169676224898487
1|6|0.164690820027741
1|7|0.128458728519047
1|8|0.18549773544014
1|9|0.155677575617836
1|10|0.153941343314285

2|1|0.217221158956016
2|2|0.23390973064067
2|3|0.180231657220626
2|4|0.257673927303071
2|5|0.261393785194329
2|6|0.273441488895552
2|7|0.242815632929545
2|8|0.262269697286057
2|9|0.256054399760891
2|10|0.262613705138411

3|1|0.378589461360716
3|2|0.33008177312116
3|3|0.380973166776554
3|4|0.340431190160728
3|5|0.38189416214207
3|6|0.364842933594872
3|7|0.372958396398964
3|8|0.350010176652464
3|9|0.355815612501188
3|10|0.380553180349294

Bash: Problem with reading files with spaces in the name using a for loop

Recently we were working on a bash script that was supposed to find and process some files that matched certain criteria. The script would process the files one by one and the criteria would be matched using the find command. To implement our solution, we returned the results of the find back to the for loop in an attempt to keep it simple and human readable.

Our original code was the following:
(do not use it, see explanation below)

for file in `find $search_path -type f -name '*.kml'`; do
  # Formatting KML file to be human friendly.
  xmllint --format "$file" > "$output_path/$file";
done

Soon we realized that we had a very nasty bug, the way we formatted the command it would break filenames that had spaces in them into multiple for loop entries and thus we would get incorrect filenames back to process.

To solve this issue we needed a way to force our loop to read the results of find one line at a time instead of one word at a time. The solution we used in the end was fairly different than the original code as it had the following significant changes:

  • the results of the find command were piped into the loop
  • the loop was not longer a for loop and a while loop was used instead
  • it used the read command that reads one line at a time to fill in the filename variable
    (the -r parameter does not allow backslashes to escape any characters)

Solution

find $search_path -type f -name '*.kml' | 
while read -r file; do
  # Formatting KML file to be human friendly.
  xmllint --format "$file" > "$output_path/$file";
done


Create an encrypted 7zip archive with encrypted header as well (no filenames are visible)

In case you come to a scenario where you need to encrypt, password protect the contents of a 7zip archive and make sure that not even the filenames of the contents are visible, 7zip has your back! As you can see in the following example you can implement the above requirements very easily.

7z a -p"pbVfdPs27Dc" -mhe hello.7z file1.bin file2.doc files.*

The structure of the above 7z command is the following:

#Based on: 7z <command> [<switches>...] <archive_name> [<file_names>...]
7z a -p"Some [email protected]" -mhe <archive_name> [<file_names>...]

To break it down, it goes like this:

  • We used the <command> a, which instructs the tool to add the listed files to the listed archive (if the archive does not exist, it will create it).
  • The <switch> -p, allows you to set the password for the archive.
  • The second <switch> -mhe (or -mhe=on) it enables data and header archive encryption.
    In case you cannot find this switch at the manual, check the examples in the man page (This command works on GNU/Linux, it was tested on Fedora).

NetCat (nc) as a webserver

Recently, we needed to perform some tests in a network. Specifically, we wanted to check the configuration of a firewall and see what IP are blocked and/or which ports are allowed to go through. To do so, we used NetCat to setup a small web-server to perform our tests.

Netcat (often abbreviated to nc) is a computer networking utility for reading from and writing to network connections using TCP or UDP. Netcat is designed to be a dependable back-end that can be used directly or easily driven by other programs and scripts. At the same time, it is a feature-rich network debugging and investigation tool, since it can produce almost any kind of connection its user could need and has a number of built-in capabilities.

From: https://en.wikipedia.org/wiki/Netcat

while true;
do
  echo -e "HTTP/1.1 200 OK\r\n\r\n<h1>$(hostname) is live</h1>$(date)" | nc -vl -p 5555;
done

or in one line

while true; do echo -e "HTTP/1.1 200 OK\r\n\r\n<h1>$(hostname) is live</h1>$(date)" | nc -vl -p 5555; done

Explanation of code:

  • The above code creates an infinite loop that calls nc in listening mode, we had to do this as nc will terminate as soon as it serves one client.
  • Using echo we create an html 200 response along with a small “webpage”.
  • While generating the webpage, echo -e will execute the commands hostname and date to get the current system values adding them to the resulted text.
  • The resulted text is then piped to nc to be served as a response to any incoming clients.
  • The date and time that nc will show to the client is not the current date and time when visiting the webpage but the one that was when echo was executed.

nc parameters:

  • -v, --verbose Sets the verbosity level and it can be used several times to increase it even further
  • -l, --listen Instructs nc to bind and listen for incoming connections (just like a web-server)
  • -p, --source-portwith port parameter specifies the source port to be used by nc

Custom terminator layout with multiple tabs and terminals

The following terminator layout ( Terminator Layout (235 downloads) ) opens 3 different tabs, the first two tabs contain only one terminal each and the third one has 4 terminals in a 2×2 matrix.
Each of these tabs have their own custom name set and following each terminal has its name set to make it easier for the user to recognize the purpose of each one.

Terminator Layout (235 downloads)

After opening these terminals, the configuration file, contains specific commands to be executed by each terminal, allowing you to automate a some trivial part of your day to day operations.
In this example, each terminal will navigate to a specific project or connect via ssh to some server, then it will perform some operation like performing a git pull and finally it will preserve the connection for you by starting a new bash instance to continue using that terminal.

Feel free to edit the layout and create a custom configuration for your tabs / the terminals and the commands.

Installation / Usage

  1. Replace the config ( Terminator Layout (235 downloads) ) file in your home user folder ~/.config/terminator/  with the one we provide
    (In nautilus press Ctrl+H to view hidden files and folders if you cannot find the .config folder)
  2. Open terminator and execute the following:
    terminator -l init;

If you want to create an alias for this command:
Open .bashrc file at your home user folder and add the following

alias my-init="terminator -l init"

For any new terminal in terminator, executing my-init will spawn a new window of terminator that has all the configuration from the file loaded into it.

Contents of Terminator Layout (235 downloads)

[global_config]
[keybindings]
[layouts]
  [[default]]
    [[[child1]]]
      parent = window0
      type = Terminal
    [[[window0]]]
      parent = ""
      type = Window
  [[init]]
    [[[child0]]]
      fullscreen = False
      last_active_window = True
      maximised = True
      order = 0
      parent = ""
      position = 0:26
      size = 1918, 1002
      title = /bin/bash
      type = Window
    [[[child1]]]
      active_page = 0
      labels = www, MA, all other, dev logs, staging logs, live logs
      last_active_term = d3c317d7-964a-4625-96d0-39deb5166072, 93ce7874-059e-4794-b337-7b640654a3d6, db090e6f-07e4-431e-ad86-a8b6cb965b5e, 906a5f4d-a3af-4da8-8385-673b132e7edd, 1b48b3b9-216c-470b-be53-ec1e8c6fdc0b, cb7d737c-a064-4e0e-ad5e-59c47d7bdd3b
      order = 0
      parent = child0
      type = Notebook
    [[[child11]]]
      order = 3
      parent = child1
      position = 956
      ratio = 0.500261643119
      type = HPaned
    [[[child14]]]
      order = 4
      parent = child1
      position = 956
      ratio = 0.500261643119
      type = HPaned
    [[[child17]]]
      order = 5
      parent = child1
      position = 956
      ratio = 0.500261643119
      type = HPaned
    [[[child4]]]
      order = 2
      parent = child1
      position = 956
      ratio = 0.500261643119
      type = HPaned
    [[[child5]]]
      order = 0
      parent = child4
      position = 481
      ratio = 0.500520291363
      type = VPaned
    [[[child8]]]
      order = 1
      parent = child4
      position = 481
      ratio = 0.500520291363
      type = VPaned
    [[[terminal10]]]
      command = cd /vhosts/www.example.com/; git pull; bash
      order = 1
      parent = child8
      profile = default
      title = www.example.com
      type = Terminal
      uuid = db090e6f-07e4-431e-ad86-a8b6cb965b5e
    [[[terminal12]]]
      command = "ssh -t git 'cd vhosts/www.bytefreaks.net/ci_applications/registration_forms/logs/; ll; bash'"
      directory = ""
      order = 0
      parent = child11
      profile = default
      title = WWW dev logs
      type = Terminal
      uuid = 4c08356b-b516-4286-8b6d-ba071f1394f3
    [[[terminal13]]]
      command = "ssh -t git 'cd vhosts/my.bytefreaks.net/symfony/var/logs/; ll; bash'"
      directory = ""
      order = 1
      parent = child11
      profile = default
      title = MA dev logs
      type = Terminal
      uuid = 906a5f4d-a3af-4da8-8385-673b132e7edd
    [[[terminal15]]]
      command = "ssh -t ptl-web3 'cd /data/var/www/vhosts/staging-www.bytefreaks.net/htdocs/ci_applications/registration_forms/logs/; ls -la; bash'"
      order = 0
      parent = child14
      profile = default
      title = WWW staging logs
      type = Terminal
      uuid = 1b48b3b9-216c-470b-be53-ec1e8c6fdc0b
    [[[terminal16]]]
      command = "ssh -t ptl-web3 'cd /data/var/www/vhosts/staging-my.bytefreaks.net/htdocs/symfony/var/logs/; ls -la; bash'"
      order = 1
      parent = child14
      profile = default
      title = MA staging logs
      type = Terminal
      uuid = e26e94cd-855a-44ff-9c67-66b1c03bac56
    [[[terminal18]]]
      command = "ssh -t ptl-web3 'cd /data/var/www/vhosts/www.bytefreaks.net/htdocs/ci_applications/registration_forms/logs/; ls -la; bash'"
      directory = ""
      order = 0
      parent = child17
      profile = default
      title = WWW Live logs
      type = Terminal
      uuid = 70466609-2d01-45d2-84b4-e377b111e540
    [[[terminal19]]]
      command = "ssh -t ptl-web3 'cd /data/var/www/vhosts/my.bytefreaks.net/htdocs/symfony/var/logs/; ls -la; bash'"
      directory = ""
      order = 1
      parent = child17
      profile = default
      title = MA Live logs
      type = Terminal
      uuid = cb7d737c-a064-4e0e-ad5e-59c47d7bdd3b
    [[[terminal2]]]
      command = cd /vhosts/www.bytefreaks.net/; git pull; bash
      order = 0
      parent = child1
      profile = default
      title = www.bytefreaks.net
      type = Terminal
      uuid = d3c317d7-964a-4625-96d0-39deb5166072
    [[[terminal3]]]
      command = cd /vhosts/my.bytefreaks.net/; git pull; bash
      order = 1
      parent = child1
      profile = default
      title = my.bytefreaks.net
      type = Terminal
      uuid = 93ce7874-059e-4794-b337-7b640654a3d6
    [[[terminal6]]]
      command = cd /vhosts/www.michanicos.com/; git pull; bash
      order = 0
      parent = child5
      profile = default
      title = www.michanicos.com
      type = Terminal
      uuid = 2f204209-0c0b-4fab-b883-95f95f5d38e9
    [[[terminal7]]]
      command = cd /vhosts/www.etea.com.cy/; git pull; bash
      order = 1
      parent = child5
      profile = default
      title = www.etea.com.cy
      type = Terminal
      uuid = 6f801914-5225-4e1f-b54c-f48540274614
    [[[terminal9]]]
      command = cd /vhosts/www.ieee.org/; git pull; bash
      order = 0
      parent = child8
      profile = default
      title = www.ieee.org
      type = Terminal
      uuid = 3dbfe3a7-2e25-4e7d-bb02-dc4aeeeda47f
[plugins]
[profiles]
  [[default]]
    background_darkness = 0.8
    cursor_color = "#ffffff"
    foreground_color = "#ffffff"

How we create bootable GNU/Linux USB flash drives from terminal

A very important tool in our everyday life are the LiveUSB GNU/Linux flash drives.
We keep an updated collection of several GNU/Linux flavors/distributions (Fedora, CentOS, (L/X)Ubuntu, Kali etc.) that are used depending on the scenario.

The command we use is the following:

sudo dd bs=4M if=path/to/OS.iso of=/dev/sdX conv=fdatasync;

dd allows you to convert and copy a file and we use it to copy the ISO file of the operating system onto the USB flash drive.

Notes:

  1. You need to unmount the USB flash drive before formatting it, e.g.:
    sudo umount /dev/sdXY;
  2. You need to use the device filename and not a partition filename:
    e.g. You need to use /dev/sdX and NOT /dev/sdX1
  3. You need to use either the root account or execute the command with sudo
  4. If you do not know the filename associated with your flash drive, use an application like the following ones to determine which /dev file is mapped to the USB flash drive:
    gnome-disks; or
    lsblk; or
    sudo fdisk -l;

The parameters we use are the following:

  • bs=SIZE_IN_BYTES defines up to how many bytes should be read and written at a time.
    In our case we used 4 Megabytes (4M).
  • if=INPUT_FILE defines the file to be read, we use this parameter to point to the OS ISO file that we want to write on the USB drive.
  • of=OUTPUT_FILE defines the filename where the data is to be written in.
    In GNU/Linux, devices are accessible like files as well so we used /dev/sdX here that happened to be the device file assigned to our USB device.
  • conv=CONVS converts the file as per the comma separated symbol list
    fdatasync physically writes output file data before finishing, we use this parameter to be sure that all I/O operations are done well before dd terminates, this way we are certain that our USB device will be ready to use as soon as the application is done.

ffmpeg: Create a video countdown

The code below was used to generate the video countdown timers that are available in the following playlist using ffmpeg:

#This example will create a 3 second video, with 100 frames per second and it will print the elapsed and remaining times using a two second accuracy.
fps=100;
seconds=3;
mantissaDigits=2;
upperFont=600;
lowerFont=100;
ffmpeg -loop 1 -i ~/Pictures/Black-Background.png -c:v libx264 -r $fps -t $seconds -pix_fmt yuv420p -vf "fps=$fps,drawtext=fontfile='/usr/share/fonts/urw-base35/C059-Bold.otf':fontcolor=yellow:fontsize=$upperFont:x=(w-text_w)/2:y=(h-text_h)/2:text='%{eif\:($seconds-t)\:d}.%{eif\:(mod($seconds-t, 1)*pow(10,$mantissaDigits))\:d\:$mantissaDigits}',drawtext=fontfile='/usr/share/fonts/urw-base35/C059-Bold.otf':fontcolor=yellow:fontsize=$lowerFont:x=(w-text_w)/2:y=((h-text_h)/2)+$upperFont:text='Elapsed\: %{eif\:(t)\:d}.%{eif\:(mod(t, 1)*pow(10,$mantissaDigits))\:d\:$mantissaDigits}'" "$seconds seconds countdown timer.mp4";

Notes:

  • We used a single black frame for the background that defined the size of the video frame as well.
  • Using the fps variable we defined the number of Frames per Second for the video.
  • The seconds variable defined the number of seconds the duration of the video should be.
  • The mantissaDigits variable defines how many decimal digits should be shown after the dot.
  • upperFont and lowerFont define the size of the fonts in the upper row and the lower one respectively.
  • We used the drawtext directive twice to write to the frames.

Notes on the first drawtext:

  • fontfile='/usr/share/fonts/urw-base35/C059-Bold.otf' defines the font to be used for the text.
  • fontcolor=yellow defines the color of the font of the text.
  • fontsize=$upperFont defines the size of the font of the text.
  • x=(w-text_w)/2 defines the X-coordinate of the location for the text on the frame, here we center the text horizontally on the frame.
  • y=(h-text_h)/2 defines the Y-coordinate of the location for the text on the frame, here we center the text vertically on the frame.
  • text='%{eif\:($seconds-t)\:d}.%{eif\:(mod($seconds-t, 1)*pow(10,$mantissaDigits))\:d\:$mantissaDigits}' We print the remaining seconds for the video to finish with specific decimal digit accuracy.

Notes on the second drawtext:

  • drawtext=fontfile='/usr/share/fonts/urw-base35/C059-Bold.otf' defines the font to be used for the text.
  • fontcolor=yellow defines the color of the font of the text.
  • fontsize=$lowerFont defines the size of the font of the text.
  • x=(w-text_w)/2 defines the X-coordinate of the location for the text on the frame, here we center the text horizontally on the frame.
  • y=((h-text_h)/2)+$upperFont defines the Y-coordinate of the location for the text on the frame, here shift the text from the vertical center  of the frame.
  • text='Elapsed\: %{eif\:(t)\:d}.%{eif\:(mod(t, 1)*pow(10,$mantissaDigits))\:d\:$mantissaDigits}' We print the elapsed seconds since the video started with specific decimal digit accuracy.

Perform diff on two folders

To perform a recursive diff on all the files of two folders we just need to add the -r (or --recursive) parameter that recursively compares any subdirectories found.

To avoid needless messages from the tool, we can also use the -q (or --brief) parameter that reports only when files differ.

Example of performing diff on two folders recursively while preventing needless messages.

diff -rq aFolder someOtherFolder;