bash


Get execution time in seconds

The following methods demonstrate different methods on how to compute the time a potion of code or script take to complete their execution.

Time Methods - Full Examples (69 downloads)

 

Method 1 – Using date

The following example will calculate the execution time in seconds by subtracting the system date and time in seconds since 1970-01-01 00:00:00 UTC once right before the script goes to the computation part and once right after.

In order to get the system date and time in seconds since 1970-01-01 00:00:00 UTC we use the command date +%s.

Time Methods - Full Examples (69 downloads)
#!/bin/bash

#Print the system date and time in seconds since 1970-01-01 00:00:00 UTC
startTime=$(date +%s);

#We pick a random number between 1 and 10.
#Then we delay the execution for that amount of seconds.
sleep $(( (RANDOM % 10) + 1 ));

endTime=$(date +%s);

#Subtract endTime from startTime to get the total execution time
totalTime=$(($endTime-$startTime));

echo "Process finished after $totalTime seconds";

exit 0;

Method 2 – Using bash internal SECONDS variable

The following example will calculate the execution time in seconds by reseting the bash internal variable SECONDS to 0, forcing the shell to continue counting from there.

Time Methods - Full Examples (69 downloads)
#!/bin/bash

#This variable expands to the number of seconds since the shell was started.
#We set it to 0, forcing the shell to continue counting from there.
SECONDS=0;

#We pick a random number between 1 and 10.
#Then we delay the execution for that amount of seconds.
sleep $(( (RANDOM % 10) + 1 ));

echo "Process finished after $SECONDS seconds";

exit 0;

Method 3 – Using bash time

The following example uses the bash time command, which reports the time consumed by a pipeline’s execution.
When time command is executed without its complete path, then the bash built-in time command is executed, instead of the GNU time command. We will use the bash time command in this example and we will use it to run a whole block of commands.
Please note that time command will return the time in seconds as a float (i.e. there will be decimal places. e.g. 1 will be printed as 1.00).

Time Methods - Full Examples (69 downloads)
#!/bin/bash

#The bash time command reports the time consumed by pipeline's execution
#When time command is executed without its complete path, then the bash built-in time command is executed, instead of the GNU time command.
#We will use the bash time command in this example and we will use it to run a whole block of commands.

#We change the output format of time to print elapsed real time in seconds.
TIMEFORMAT="%E";
#We pick a random number between 1 and 10.
#Then we delay the execution for that amount of seconds.
totalTime=`time ( sleep $(( (RANDOM % 10) + 1 )) ) 2>&1`;

#Please note that time command will return the time in seconds as a float (i.e. there will be decimal places. e.g. 1 will be printed as 1.00).
#This will happen as time has build-in more precision than the first two methods presented here.
echo "Process finished after $totalTime seconds";

totalTimeBlock=`time (
	sleep $(( (RANDOM % 10) + 1 ));
	sleep $(( (RANDOM % 10) + 1 ));
) 2>&1`;
echo "Block finished after $totalTimeBlock seconds";

exit 0;

Method 4 – Using GNU time

The GNU time command runs the specified program command with the given arguments.
When time command is executed without its complete path (in our case it was /usr/bin/time), then the bash built-in time command is executed, instead of the GNU time command. To make sure we use the GNU time command, we use which to get the full path of the time command.
Please note that time command will return the time in seconds as a float (i.e. there will be decimal places. e.g. 1 will be printed as 1.00).

Time Methods - Full Examples (69 downloads)
#!/bin/bash
#The time command runs the specified program command with the given arguments.
#When time command is executed without its complete path (in our case it was /usr/bin/time), then the bash built-in time command is executed, instead of the GNU time command.
#To make sure we use the GNU time command, we use which to get the full path of the time command.
time=`which time`;

#We pick a random number between 1 and 10.
#Then we delay the execution for that amount of seconds.
#We change the output format of time to print elapsed real time in seconds.
totalTime="$( $time -f '%e' sleep $(( (RANDOM % 10) + 1 )) 2>&1 1>/dev/null )";

#Please note that time command will return the time in seconds as a float (i.e. there will be decimal places. e.g. 1 will be printed as 1.00).
#This will happen as time has build-in more precision than the first two methods presented here.
echo "Process finished after $totalTime seconds";

exit 0;

Notes

RANDOM internal variable

Each time RANDOM internal variable is referenced, a random integer between 0 and 32767 is generated.

By using the RANDOM variable in this command $(( (RANDOM % 10) + 1 )); we perform a modulo on the random value with the static value 10. This way we force the range of valid values to be between 0 and 9.
Later, we add 1 to that value to shift the range to be between 1 and 10.


cecho – a function to print using different colors in bash

The following script can be used to print colored text in bash.
You can use it in any script without copy pasting everything in it by executing the following command source cecho.sh.
Doing so, it will load to your script the functions that are defined in cecho.sh, making them available for you to use (something like including code in C, with some caveats).

cecho.sh (compressed) (68 downloads)
#!/bin/bash

# The following function prints a text using custom color
# -c or --color define the color for the print. See the array colors for the available options.
# -n or --noline directs the system not to print a new line after the content.
# Last argument is the message to be printed.
cecho () {

    declare -A colors;
    colors=(\
        ['black']='\E[0;47m'\
        ['red']='\E[0;31m'\
        ['green']='\E[0;32m'\
        ['yellow']='\E[0;33m'\
        ['blue']='\E[0;34m'\
        ['magenta']='\E[0;35m'\
        ['cyan']='\E[0;36m'\
        ['white']='\E[0;37m'\
    );

    local defaultMSG="No message passed.";
    local defaultColor="black";
    local defaultNewLine=true;

    while [[ $# -gt 1 ]];
    do
    key="$1";

    case $key in
        -c|--color)
            color="$2";
            shift;
        ;;
        -n|--noline)
            newLine=false;
        ;;
        *)
            # unknown option
        ;;
    esac
    shift;
    done

    message=${1:-$defaultMSG};   # Defaults to default message.
    color=${color:-$defaultColor};   # Defaults to default color, if not specified.
    newLine=${newLine:-$defaultNewLine};

    echo -en "${colors[$color]}";
    echo -en "$message";
    if [ "$newLine" = true ] ; then
        echo;
    fi
    tput sgr0; #  Reset text attributes to normal without clearing screen.

    return;
}

warning () {

    cecho -c 'yellow' "[email protected]";
}

error () {

    cecho -c 'red' "[email protected]";
}

information () {

    cecho -c 'blue' "[email protected]";
}

 

Usage

Function cecho accepts the options to set the color and to control if a new line should be print.
Parameter -c or --color define the color for the print. See the array colors for the available options.
Parameter -n or --noline directs the system not to print a new line after the content.
The last parameter is the string message to be printed.
Functions warning, error and information are using cecho to print in color.
These three functions always print a new line and they have hardcoded one color set for each.

Example

#Get the name of the script currently being executed

scriptName=$(basename $(test -L "$0" && readlink "$0" || echo "$0"));

#Get the directory where the script currently being executed resides

scriptDirDIR=$(cd $(dirname "$0") && pwd);

#Print in blue color with no new line

cecho -n -c 'blue' "$scriptDir";

#Print in red color with a new line following the message

cecho -c 'red' "$scriptName";

#Using the information() function to print in blue followed by a new line

information ‘End of script’;


Convert a list of integers from MySQL to a Bash array

The following code will connect to a MySQL server, it will get a list of integers and convert the results to a bash array that will be iterated using a for loop and they will be printed using zero padding.

IDS_RAW=$(mysql --user="myuser" --password="mypass" --host="db.example.com" --database="mydb" --batch --compress --skip-column-names --execute="
  SELECT
    Id
  FROM
    Users
  WHERE
    Status = 0;
");

OLDIFS=$IFS;
IFS=$'\n' command eval;
'IDS=($IDS_RAW)';
IFS=$OLDIFS;

echo "Will process the following user IDs '${IDS[@]}'";

for ID in "${IDS[@]}"; do
  LEADING_ZERO=$(printf %08d $ID);
  echo "ID $LEADING_ZERO";
done;

Bash get script file name and location

The following code will populate the variables SCRIPT_NAME and SCRIPT_DIR with the name of the script currently being execute and the location this script is in:

SCRIPT_NAME=$(basename $(test -L "$0" && readlink "$0" || echo "$0"));
SCRIPT_DIR=$(cd $(dirname "$0") && pwd);

Notes for SCRIPT_NAME:

  • $0 expands to the name of the shell or shell script
  • test -L "$0" checks that input is a file that exists and is a symbolic link
  • && readlink "$0" will be executed if the above statement is true and it will print the resolved symbolic link
  • || echo "$0" will be executed if the test for symbolic link fails
  • finally, basename will strip directory and suffix from whatever is returned from the above statements

Notes for SCRIPT_DIR:

  • Will not resolve the correct folder if the last component of the path is a symbolic link (symlink). It will return the location of the symlink instead of the location of the file the symlink is pointing to
  • cd will return 0 if it successfully navigates to a directory or 1 when it fails to navigate to the directory
  • cd "$( dirname "$0" )" will use dirname to strip the last component from the expanded name and try to navigate to that location
  • if the above cd fails, we get the current location using && pwd. pwd will print name of current/working directory

In case you have a problem with $0, it is overwritten or the above function is called by a child script in another folder you can replace $0 with ${BASH_SOURCE[0]}.

SCRIPT_NAME=$(basename $(test -L "${BASH_SOURCE[0]}" && readlink "${BASH_SOURCE[0]}" || echo "${BASH_SOURCE[0]}"));
SCRIPT_DIR=$(cd $(dirname "${BASH_SOURCE[0]}") && pwd);

Bash: Extract data from files both filtering filename, the path and doing internal processing

The following code will find all files that match the pattern 2016_*_*.log (all the log files for the year 2016).

To avoid finding log files from other services than the Web API service, we filter only the files that their path contains the folder webapi. Specifically, we used "/ServerLogs/*/webapi/*" with the following command to match all files that are under the folder /ServerLogs/ and somewhere in the path there is another folder named webapi, we do that to match files that are like /ServerLogs/Production/01/webapi/* only. The way we coded our regular expression, it will not match if there is a folder called webapi directly under the /ServerLogs/ (e.g. /ServerLogs/webapi/*).

For each result, we execute an awk script that will split the lines using the comma (FS=",";) character, then check if the line contains exactly 4 tokens (if (NF == 4) {). Later, we get the 4th token and check if it contains the substring "MASTER=" (if (match($4,"MASTER=")) {), if it does contain it we split it using the space character and assign the result to the variable named tokens. From tokens, we get the first token and use substr to remove the first character. Finally, we use the formatted result to create an array where the keys are the values we just created and it is used as a hashmap to keep record of all unique strings. In the end clause, we print all the elements of our hash map.

Finally, we sort all the results from all the awk executions and remove duplicates using sort --unique.

find /ServerLogs/ \
    -iname "2016_*_*.log" \
    -ipath "/ServerLogs/*/webapi/*" \
    -exec awk '
        BEGIN {
            FS=",";
        }
        {
            if (NF == 4) {
                if (match($4,"MASTER=")) {
                    split($4, tokens, " ");
                    instances[substr(tokens[1], 2)];
                }
            }
        }
        END {
            for (element in instances) {
                print element;
            }
        }
    ' \
    '{}' \; | sort --unique;

Following is the same code in one line.

 find /ServerLogs/ -iname "2016_*_*.log" -ipath "/ServerLogs/*/webapi/*" -exec awk 'BEGIN {FS=",";} {if (NF == 4) {if (match($4,"MASTER=")){split($4, tokens, " "); instances[substr(tokens[1], 2)];}}} END {for (element in instances) {print element;}}' '{}' \; | sort --unique 

Another way

Another way to do similar functionality would be the following

find /ServerLogs/ \
    -iname "2016_*_*.log" \
    -ipath "/ServerLogs/*/webapi/*" \
    -exec sh -c '
        grep "MASTER=" -s "$0" | awk "BEGIN {FS=\",\";} NF==4" | cut -d "," -f4 | cut -c 3- | cut -d " " -f1 | sort --unique
    ' \
    '{}' \; | sort --unique;

What we changed is the -exec part. Instead of calling a awk script, we create a new sub-shell using sh -c, then we define the source to be executed inside the single codes and we pass as the first parameter of the shell the filename that matched.

Inside the shell, we find all lines that contain the string MASTER= using the grep command. Later we filter out all lines that do not have four columns when we tokenize using the comma character using awk. Then, we get the 4th column using cut and delimiter the comma. We remove the first two characters of the input string using cut -c 3- and later we get only the first column by reusing cut and changing the delimiter to be the space character. With those results we perform a sort that eliminates duplicates and we pass the results to the parent process to perform other operations.

Following is the same code in one line

find /ServerLogs/ -iname "2016_*_*.log" -ipath "/ServerLogs/*/webapi/*" -exec sh -c 'grep "MASTER=" -s "$0" | awk "BEGIN {FS=\",\";} NF==4" | cut -d "," -f4 | cut -c 3- | cut -d " " -f1 | sort --unique' '{}' \; | sort --unique;


Bash/FFMPEG: Batch resize .mp4 videos to fixed resolution

We needed to shrink a bunch of mp4 videos so that they would have the same size as the screen of an android device.
We did that both to save space on the internal memory of the device and to make the device perform as efficient as possible as it would not have to shrink the video on the fly.

The command we used was the following:

find . -type f -name "*.mp4" -exec bash -c 'FILE="$1"; ffmpeg -i "${FILE}" -s 1280x720 -acodec copy -y "${FILE%.mp4}.shrink.mp4";' _ '{}' \;

What this command does is the following:

  • Find all files in current folder (and sub-folders) that have the extension .mp4
  • For each file, create a new bash instance in which it will call ffmpeg taking as first parameter the filename that matched
  • -i "${FILE}"ffmpeg will take as input the filename we matched
  • -s 1280x720 – Then change the video size to 1280x720
  • -acodec copy – It will keep the audio as is
  • -y "${FILE%.mp4}.shrink.mp4 – Finally, create a new file (or overwrite existing) that has the extension .shrink.mp4 in the same folder

HOWTO: Make Terminator Terminal Act Like Guake Terminal in Ubuntu 16.04 LTS (XenialXerus) Desktop edition x64 bit architecture 1

We propose an alternative solution to making terminator act like guake that requires two additional packages: xdotool and wmctrl.

Our proposal will launch terminator if there is not instance running.

In Ubuntu you can install the needed packages from the official repositories using sudo apt-get install xdotool wmctrl.

Using a text editor, create in you home folder a file named nano ~/toggle_visibility.sh and copy there the contents of the following chunk of code. You can also use nano, from a terminal issue nano ~/toggle_visibility.sh, then paste the code and hit CTRL+X to exit. When prompted if you want to save press ‘Y’ and hit enter.

#!/bin/bash

#The purpose of this script is to allow the user to toggle the visibility of (almost) any window.
#Please note it will work on the first match, so if there are multiple instances of an application it would be a random window of them the one to be affected.
#Usually it will control the window with the smallest PID.

#Checking that all dependencies are met, since we cannot proceed without them.
declare -a DEPENDENCIES=("xdotool" "wmctrl");
declare -a MANAGERS=("dnf" "apt-get");

for DEPENDENCY in ${DEPENDENCIES[@]}; do
    echo -n "Checking if $DEPENDENCY is available";
    if hash $DEPENDENCY 2>/dev/null; then
        echo "- OK, Found";
    else
        echo "- ERROR, Not Found in $PATH";
        for MANAGER in ${MANAGERS[@]}; do
            if hash $MANAGER 2>/dev/null; then
                echo -n "$DEPENDENCY is missing, would you like to try and install it via $MANAGER now? [Y/N] (default is Y): ";
                read ANSWER;
                if [[ "$ANSWER" == "Y" || "$ANSWER" == "y" || "$ANSWER" == "" ]]; then
                    sudo "$MANAGER" install "$DEPENDENCY";
                else
                    echo "Terminating";
                    exit -1;
                fi
            fi
        done
    fi
done

APPLICATION="$1";
FULL_COMMAND="$2";

#Checking if the application name provided by the user exists
if ! hash $APPLICATION 2>/dev/null; then
    echo -e "$APPLICATION does not seem to be a valid executable\nTerminating";
    exit -2;
fi

#Checking if the application is running.
PID=$(pgrep -u `whoami` -f "$FULL_COMMAND" | head -n 1);

#If the application is not running, we will try to launch it.
if [ -z $PID ]; then
  echo "$FULL_COMMAND not running, launching it..";
    $FULL_COMMAND;
else
    #Since the application has a live instance, we can proceed with the rest of the code.
    #We will get the PID of the application that is currently focused, if it is not the application we passed as parameter we will change the focus to that. In the other case, we will minimize the application.
  echo -n "$FULL_COMMAND instance found - ";
    FOCUSED=$(xdotool getactivewindow getwindowpid);
    if [[ $PID == $FOCUSED ]]; then
    echo "It was focused so we are minimizing it";
        #We minimize the active window which we know in this case that it is the application we passed as parameter.
        xdotool getactivewindow windowminimize;
    else
    echo "We are setting the focus on it";
        #We set the focus to the application we passed as parameter. If it is minimized it will be raised as well.
        wmctrl -x -R $APPLICATION;
    fi
fi

exit 0

Afterwards, you need to make the script an executable so you should issue chmod +x ~/toggle_visibility.sh to do that.

Then, execute ~/toggle_visibility.sh in your terminal once. We need to do that in order to install any missing dependencies for the tool.

Finally, you need to create a custom shortcut that will call the script using the key combination you like at any point.

To complete the procedure:

  1. Go to ‘System Settings’ either by clicking on the menu on the top right corner that looks like a light bulb or by issuing the following in a terminal unity-control-center to start the unity control center.
  2. In the newly appeared window, click on the ‘keyboard’ icon that is in the category ‘Hardware’.
  3. After that, click on the tab ‘Shortcuts’
  4. and on the left list, click on custom shortcuts.
  5. You will see a button with the + sign right next to the list, click that.
  6. In the dialog box that will appear enter the following:
    – In the name field enter anything you like. e.g ‘Toggle Terminator Visibility’
    – In the command field enter /home/<USER>/toggle_visibility.sh terminator "^/usr/bin/python /usr/bin/terminator$" where <USER> enter your own username.
    – Click apply.
  7. You will see a new row with two columns with the name you just set in the first column. Click on the second column, where it should say ‘Disabled’ and the press the key combination you want for toggling terminator e.g F12

You are ready to go 🙂

Just try the key combination you just provided and terminator will appear in front of you. Pressing it once more it will hide it.

NOTE: Please keep in mind that the above script can be used for other applications as well. In step 7, we gave as parameter the name of the application to be used, if you change that you could use it with other applications like Firefox.


Grep lines that do not begin with ‘#’ or ‘;’

Recently, we wanted to modify  the squid configuration file, which is really really big!

wc -l /etc/squid/squid.conf
7898 /etc/squid/squid.conf

We wanted to find all active rules that are enabled to modify our proxy server. Out of those ~8K lines less than 20 are actually active configuration, the rest is documentation.

To find all active configuration lines we needed to find all lines that:

  • are not empty
  • do not start with #
  • do not start with ;

To do this we used the following grep command

grep "^[^#;]" /etc/squid/squid.conf

The first ^ refers to the beginning of the line, this way if in a line there is some configuration and after that there is a comment it will not be excluded by mistake. The rest, [^#;] matches any character which is not # or ;.

This is what was actually in my configuration file (out of ~8K lines)

acl SSL_ports port 443
acl Safe_ports port 80        # http
acl Safe_ports port 21        # ftp
acl Safe_ports port 443        # https
acl Safe_ports port 70        # gopher
acl Safe_ports port 210        # wais
acl Safe_ports port 1025-65535    # unregistered ports
acl Safe_ports port 280        # http-mgmt
acl Safe_ports port 488        # gss-http
acl Safe_ports port 591        # filemaker
acl Safe_ports port 777        # multiling http
acl CONNECT method CONNECT
http_access deny !Safe_ports
http_access deny CONNECT !SSL_ports
http_access allow localhost manager
http_access deny manager
http_access allow localhost
http_access deny all
http_port 3128
coredump_dir /var/spool/squid
refresh_pattern ^ftp:        1440    20%    10080
refresh_pattern ^gopher:    1440    0%    1440
refresh_pattern -i (/cgi-bin/|\?) 0    0%    0
refresh_pattern (Release|Packages(.gz)*)$      0       20%     2880
refresh_pattern .        0    20%    4320

Bash Function to print out the files and the lines that contain a needle

The following code will create a function in bash that accepts two parameters (1: the place to search in, 2: the value to search for).

You can place it in your ~/.bashrc file to have it available whenever you open a bash shell.

#1. Copy/paste the below lines in your .bashrc

#takes 2 parameters (1: the haystack to search in, 2: the needle)
# Will print out the files and the lines that contain the needle
xfind(){
  FIND_VAR="$2";
  STACK="$1";
  if [ -f "$STACK" ] || [ -d "$STACK" ]; then
    find "$STACK" \
      -exec grep --color "$FIND_VAR" -sl '{}' \; \
      -exec grep "$FIND_VAR" -s '{}' \;
  else
    echo "ERROR: No file or folder with the name '$STACK' exist";
  fi
}
#2. Run source ~/.bashrc -- to reload 

Usage examples:

xfind . "bar";
xfind /etc/ "conf";

Fedora/Bash: Get the IP of enp0s3

Following is a small snippet that will print on screen the IP of enp0s3 (or any other device if you change the name) while in Fedora.
As you will see, it is not a very sound solution as it depends on the structure of the output of ifconfig enp0s3.

Nevertheless is works (for Fedora at least)! 🙂

ifconfig enp0s3 | grep "inet " | sed -e 's/^[[:space:]]*//' -e 's/[[:space:]]*$//' | cut -d ' ' -f 2

What this line does is: first it prints out the configuration information for enp0s3, then finds the line that contains the inet, then using sed it will trim the result (in other words, it will remove all leading and all trailing white-space from the pipe), finally cut gets the second column of the data after separating the line using the space symbol.

The Fedora version that was used for this tutorial is

$cat /etc/fedora-release 
Fedora release 23 (Twenty Three)

The version of ifconfig for this tutorial is

$ifconfig --version
net-tools 2.10-alpha

In case you want to assign the IP of enp0s3 to a variable, you can easily do as follows

IP=`ifconfig enp0s3 | grep "inet " | sed -e 's/^[[:space:]]*//' -e 's/[[:space:]]*$//' | cut -d ' ' -f 2`;