bash

Add a new line whenever the first column changes 3

 in Bash / GNU/Linux  tagged awk / bash / new line by

Recently we were processing some results from an SQL query on the command line, we were grouping the results based on the first column of the query results and we wanted to add an empty line between each group of data.

Instead of messing with SQL specifics, we decided to use awk to finish the task which resulted in the following code:

awk -F '|' -v column=1 'NR>1 && $column != previous { print "" } { previous = $column } { print $0 }'

Explanation:

  • -F fs or --field-separator fs: This option allowed us to choose the input field separator, in other words the character that should be used to split our text into columns. We used the value | because that character is the default column delimiter for sqlite.
  • -v var=val or --assign var=val: We used the -v option to define a variable to be used later on in the script. The value 1 was assigned to the variable column before execution of the program began and it was available event to the BEGIN rule of the AWK program. We did this to make the code a bit more modular, we could have just hardcoded the number in.
  • NR>1 && $column != previous { print "" } : Here we defined an if statement that checks two options: First we make sure that we are not on the first line of the input by using the NR>1 (and thus avoid creating an empty line which will be the first line of the output).
    Second, we check that the last value we had for the column of interest did change since last time. (We still did not define the value of previous, it is on the next step). When both statements are true (we are not on the first row and the value of the column in the current row is different than the value of the column in the previous row) it will print out an empty line.
  • { previous = $column }: This part is executed on ALL lines (even the first one) no matter what the values are. What this line does is to translate the value of the column variable from being a number (the index of the column that we are interested in) into the actual value that the column has at that specific line. That value is then copied to the previous variable to allow us to perform the check in the previous point once we move to the next line.
  • { print $0 }: Finally, this part is also executed on all lines and it instructs awk to print the input row whole and as is. This whole part could be replaced by a true value like the value 1. In awk as you see in this example, you define a series of operations. Each operation is constructed by a pattern to be matched and an action. Each pattern is evaluated for each input line, and in the cases where the pattern matches, the action is executed. The user can choose to omit either the pattern or the action for any operation. When a pattern is omitted, the action is executed on every line. When the action is omitted, then awk will execute { print $0 }. So, by adding a true value on its own it will be translated as on each line execute { print $0 } which prints the whole row as is.

Example

1|1|0.0564904019731175
1|2|0.103176086258974
1|3|0.12910406904073
1|4|0.188592489201024
1|5|0.169676224898487
1|6|0.164690820027741
1|7|0.128458728519047
1|8|0.18549773544014
1|9|0.155677575617836
1|10|0.153941343314285
2|1|0.217221158956016
2|2|0.23390973064067
2|3|0.180231657220626
2|4|0.257673927303071
2|5|0.261393785194329
2|6|0.273441488895552
2|7|0.242815632929545
2|8|0.262269697286057
2|9|0.256054399760891
2|10|0.262613705138411
3|1|0.378589461360716
3|2|0.33008177312116
3|3|0.380973166776554
3|4|0.340431190160728
3|5|0.38189416214207
3|6|0.364842933594872
3|7|0.372958396398964
3|8|0.350010176652464
3|9|0.355815612501188
3|10|0.380553180349294

Will become

1|1|0.0564904019731175
1|2|0.103176086258974
1|3|0.12910406904073
1|4|0.188592489201024
1|5|0.169676224898487
1|6|0.164690820027741
1|7|0.128458728519047
1|8|0.18549773544014
1|9|0.155677575617836
1|10|0.153941343314285

2|1|0.217221158956016
2|2|0.23390973064067
2|3|0.180231657220626
2|4|0.257673927303071
2|5|0.261393785194329
2|6|0.273441488895552
2|7|0.242815632929545
2|8|0.262269697286057
2|9|0.256054399760891
2|10|0.262613705138411

3|1|0.378589461360716
3|2|0.33008177312116
3|3|0.380973166776554
3|4|0.340431190160728
3|5|0.38189416214207
3|6|0.364842933594872
3|7|0.372958396398964
3|8|0.350010176652464
3|9|0.355815612501188
3|10|0.380553180349294

Bash: Problem with reading files with spaces in the name using a for loop

 in Bash  tagged bash / find / for / for loop by

Recently we were working on a bash script that was supposed to find and process some files that matched certain criteria. The script would process the files one by one and the criteria would be matched using the find command. To implement our solution, we returned the results of the find back to the for loop in an attempt to keep it simple and human readable.

Our original code was the following:
(do not use it, see explanation below)

for file in `find $search_path -type f -name '*.kml'`; do
  # Formatting KML file to be human friendly.
  xmllint --format "$file" > "$output_path/$file";
done

Soon we realized that we had a very nasty bug, the way we formatted the command it would break filenames that had spaces in them into multiple for loop entries and thus we would get incorrect filenames back to process.

To solve this issue we needed a way to force our loop to read the results of find one line at a time instead of one word at a time. The solution we used in the end was fairly different than the original code as it had the following significant changes:

  • the results of the find command were piped into the loop
  • the loop was not longer a for loop and a while loop was used instead
  • it used the read command that reads one line at a time to fill in the filename variable
    (the -r parameter does not allow backslashes to escape any characters)

Solution

find $search_path -type f -name '*.kml' | 
while read -r file; do
  # Formatting KML file to be human friendly.
  xmllint --format "$file" > "$output_path/$file";
done

Custom terminator layout with multiple tabs and terminals

 in Bash / GNU/Linux  tagged alias / bash / bashrc / config / ssh / template / terminator by

The following terminator layout ([download id=”4658″]) opens 3 different tabs, the first two tabs contain only one terminal each and the third one has 4 terminals in a 2×2 matrix.
Each of these tabs have their own custom name set and following each terminal has its name set to make it easier for the user to recognize the purpose of each one.

[download id=”4658″]

After opening these terminals, the configuration file, contains specific commands to be executed by each terminal, allowing you to automate a some trivial part of your day to day operations.
In this example, each terminal will navigate to a specific project or connect via ssh to some server, then it will perform some operation like performing a git pull and finally it will preserve the connection for you by starting a new bash instance to continue using that terminal.

Feel free to edit the layout and create a custom configuration for your tabs / the terminals and the commands.

Installation / Usage

  1. Replace the config ([download id=”4658″]) file in your home user folder ~/.config/terminator/  with the one we provide
    (In nautilus press Ctrl+H to view hidden files and folders if you cannot find the .config folder)
  2. Open terminator and execute the following:
    terminator -l init;

If you want to create an alias for this command:
Open .bashrc file at your home user folder and add the following

alias my-init="terminator -l init"

For any new terminal in terminator, executing my-init will spawn a new window of terminator that has all the configuration from the file loaded into it.

Contents of [download id=”4658″]

[global_config]
[keybindings]
[layouts]
  [[default]]
    [[[child1]]]
      parent = window0
      type = Terminal
    [[[window0]]]
      parent = ""
      type = Window
  [[init]]
    [[[child0]]]
      fullscreen = False
      last_active_window = True
      maximised = True
      order = 0
      parent = ""
      position = 0:26
      size = 1918, 1002
      title = /bin/bash
      type = Window
    [[[child1]]]
      active_page = 0
      labels = www, MA, all other, dev logs, staging logs, live logs
      last_active_term = d3c317d7-964a-4625-96d0-39deb5166072, 93ce7874-059e-4794-b337-7b640654a3d6, db090e6f-07e4-431e-ad86-a8b6cb965b5e, 906a5f4d-a3af-4da8-8385-673b132e7edd, 1b48b3b9-216c-470b-be53-ec1e8c6fdc0b, cb7d737c-a064-4e0e-ad5e-59c47d7bdd3b
      order = 0
      parent = child0
      type = Notebook
    [[[child11]]]
      order = 3
      parent = child1
      position = 956
      ratio = 0.500261643119
      type = HPaned
    [[[child14]]]
      order = 4
      parent = child1
      position = 956
      ratio = 0.500261643119
      type = HPaned
    [[[child17]]]
      order = 5
      parent = child1
      position = 956
      ratio = 0.500261643119
      type = HPaned
    [[[child4]]]
      order = 2
      parent = child1
      position = 956
      ratio = 0.500261643119
      type = HPaned
    [[[child5]]]
      order = 0
      parent = child4
      position = 481
      ratio = 0.500520291363
      type = VPaned
    [[[child8]]]
      order = 1
      parent = child4
      position = 481
      ratio = 0.500520291363
      type = VPaned
    [[[terminal10]]]
      command = cd /vhosts/www.example.com/; git pull; bash
      order = 1
      parent = child8
      profile = default
      title = www.example.com
      type = Terminal
      uuid = db090e6f-07e4-431e-ad86-a8b6cb965b5e
    [[[terminal12]]]
      command = "ssh -t git 'cd vhosts/www.bytefreaks.net/ci_applications/registration_forms/logs/; ll; bash'"
      directory = ""
      order = 0
      parent = child11
      profile = default
      title = WWW dev logs
      type = Terminal
      uuid = 4c08356b-b516-4286-8b6d-ba071f1394f3
    [[[terminal13]]]
      command = "ssh -t git 'cd vhosts/my.bytefreaks.net/symfony/var/logs/; ll; bash'"
      directory = ""
      order = 1
      parent = child11
      profile = default
      title = MA dev logs
      type = Terminal
      uuid = 906a5f4d-a3af-4da8-8385-673b132e7edd
    [[[terminal15]]]
      command = "ssh -t web13 'cd /data/var/www/vhosts/staging-www.bytefreaks.net/htdocs/ci_applications/registration_forms/logs/; ls -la; bash'"
      order = 0
      parent = child14
      profile = default
      title = WWW staging logs
      type = Terminal
      uuid = 1b48b3b9-216c-470b-be53-ec1e8c6fdc0b
    [[[terminal16]]]
      command = "ssh -t web13 'cd /data/var/www/vhosts/staging-my.bytefreaks.net/htdocs/symfony/var/logs/; ls -la; bash'"
      order = 1
      parent = child14
      profile = default
      title = MA staging logs
      type = Terminal
      uuid = e26e94cd-855a-44ff-9c67-66b1c03bac56
    [[[terminal18]]]
      command = "ssh -t web13 'cd /data/var/www/vhosts/www.bytefreaks.net/htdocs/ci_applications/registration_forms/logs/; ls -la; bash'"
      directory = ""
      order = 0
      parent = child17
      profile = default
      title = WWW Live logs
      type = Terminal
      uuid = 70466609-2d01-45d2-84b4-e377b111e540
    [[[terminal19]]]
      command = "ssh -t web13 'cd /data/var/www/vhosts/my.bytefreaks.net/htdocs/symfony/var/logs/; ls -la; bash'"
      directory = ""
      order = 1
      parent = child17
      profile = default
      title = MA Live logs
      type = Terminal
      uuid = cb7d737c-a064-4e0e-ad5e-59c47d7bdd3b
    [[[terminal2]]]
      command = cd /vhosts/www.bytefreaks.net/; git pull; bash
      order = 0
      parent = child1
      profile = default
      title = www.bytefreaks.net
      type = Terminal
      uuid = d3c317d7-964a-4625-96d0-39deb5166072
    [[[terminal3]]]
      command = cd /vhosts/my.bytefreaks.net/; git pull; bash
      order = 1
      parent = child1
      profile = default
      title = my.bytefreaks.net
      type = Terminal
      uuid = 93ce7874-059e-4794-b337-7b640654a3d6
    [[[terminal6]]]
      command = cd /vhosts/www.michanicos.com/; git pull; bash
      order = 0
      parent = child5
      profile = default
      title = www.michanicos.com
      type = Terminal
      uuid = 2f204209-0c0b-4fab-b883-95f95f5d38e9
    [[[terminal7]]]
      command = cd /vhosts/www.etea.com.cy/; git pull; bash
      order = 1
      parent = child5
      profile = default
      title = www.etea.com.cy
      type = Terminal
      uuid = 6f801914-5225-4e1f-b54c-f48540274614
    [[[terminal9]]]
      command = cd /vhosts/www.ieee.org/; git pull; bash
      order = 0
      parent = child8
      profile = default
      title = www.ieee.org
      type = Terminal
      uuid = 3dbfe3a7-2e25-4e7d-bb02-dc4aeeeda47f
[plugins]
[profiles]
  [[default]]
    background_darkness = 0.8
    cursor_color = "#ffffff"
    foreground_color = "#ffffff"

Get execution time in seconds

 in Bash  tagged bash / GNU/Linux / howtos / modulo / random / time by

The following methods demonstrate different methods on how to compute the time a potion of code or script take to complete their execution.

[download id=”2158″]

 

Method 1 – Using date

The following example will calculate the execution time in seconds by subtracting the system date and time in seconds since 1970-01-01 00:00:00 UTC once right before the script goes to the computation part and once right after.

In order to get the system date and time in seconds since 1970-01-01 00:00:00 UTC we use the command date +%s.

[download id=”2158″]


#!/bin/bash

#Print the system date and time in seconds since 1970-01-01 00:00:00 UTC
startTime=$(date +%s);

#We pick a random number between 1 and 10.
#Then we delay the execution for that amount of seconds.
sleep $(( (RANDOM % 10) + 1 ));

endTime=$(date +%s);

#Subtract endTime from startTime to get the total execution time
totalTime=$(($endTime-$startTime));

echo "Process finished after $totalTime seconds";

exit 0;

Method 2 – Using bash internal SECONDS variable

The following example will calculate the execution time in seconds by reseting the bash internal variable SECONDS to 0, forcing the shell to continue counting from there.

[download id=”2158″]


#!/bin/bash

#This variable expands to the number of seconds since the shell was started.
#We set it to 0, forcing the shell to continue counting from there.
SECONDS=0;

#We pick a random number between 1 and 10.
#Then we delay the execution for that amount of seconds.
sleep $(( (RANDOM % 10) + 1 ));

echo "Process finished after $SECONDS seconds";

exit 0;

Method 3 – Using bash time

The following example uses the bash time command, which reports the time consumed by a pipeline’s execution.
When time command is executed without its complete path, then the bash built-in time command is executed, instead of the GNU time command. We will use the bash time command in this example and we will use it to run a whole block of commands.
Please note that time command will return the time in seconds as a float (i.e. there will be decimal places. e.g. 1 will be printed as 1.00).

[download id=”2158″]


#!/bin/bash

#The bash time command reports the time consumed by pipeline's execution
#When time command is executed without its complete path, then the bash built-in time command is executed, instead of the GNU time command.
#We will use the bash time command in this example and we will use it to run a whole block of commands.

#We change the output format of time to print elapsed real time in seconds.
TIMEFORMAT="%E";
#We pick a random number between 1 and 10.
#Then we delay the execution for that amount of seconds.
totalTime=`time ( sleep $(( (RANDOM % 10) + 1 )) ) 2>&1`;

#Please note that time command will return the time in seconds as a float (i.e. there will be decimal places. e.g. 1 will be printed as 1.00).
#This will happen as time has build-in more precision than the first two methods presented here.
echo "Process finished after $totalTime seconds";

totalTimeBlock=`time (
	sleep $(( (RANDOM % 10) + 1 ));
	sleep $(( (RANDOM % 10) + 1 ));
) 2>&1`;
echo "Block finished after $totalTimeBlock seconds";

exit 0;

Method 4 – Using GNU time

The GNU time command runs the specified program command with the given arguments.
When time command is executed without its complete path (in our case it was /usr/bin/time), then the bash built-in time command is executed, instead of the GNU time command. To make sure we use the GNU time command, we use which to get the full path of the time command.
Please note that time command will return the time in seconds as a float (i.e. there will be decimal places. e.g. 1 will be printed as 1.00).

[download id=”2158″]


#!/bin/bash
#The time command runs the specified program command with the given arguments.
#When time command is executed without its complete path (in our case it was /usr/bin/time), then the bash built-in time command is executed, instead of the GNU time command.
#To make sure we use the GNU time command, we use which to get the full path of the time command.
time=`which time`;

#We pick a random number between 1 and 10.
#Then we delay the execution for that amount of seconds.
#We change the output format of time to print elapsed real time in seconds.
totalTime="$( $time -f '%e' sleep $(( (RANDOM % 10) + 1 )) 2>&1 1>/dev/null )";

#Please note that time command will return the time in seconds as a float (i.e. there will be decimal places. e.g. 1 will be printed as 1.00).
#This will happen as time has build-in more precision than the first two methods presented here.
echo "Process finished after $totalTime seconds";

exit 0;

Notes

RANDOM internal variable

Each time RANDOM internal variable is referenced, a random integer between 0 and 32767 is generated.

By using the RANDOM variable in this command $(( (RANDOM % 10) + 1 )); we perform a modulo on the random value with the static value 10. This way we force the range of valid values to be between 0 and 9.
Later, we add 1 to that value to shift the range to be between 1 and 10.