Bash


Bash: Remove the last character from each line

The following script, uses rev and cut to remove the last character from each line in a pipe.
rev utility reverses lines character-wise.
cut removes sections  from each of line.
It is a very simple script where we reverse the line once, remove the first character (which was the last one in the original form of the line) and finally we reverse the line back with the last character missing.


echo -e "hi\nHI" | rev | cut -c 2- | rev;

# Will produce:
h
H

 


Bash: Print time stamp in front of every line in a pipe

Recently, we received a binary that collected data from a web service and it printed them on screen.
The binary did not print a time stamp in front of each line so we had to improvise of a way to add the time stamp to the logs without modifying the binary.

The solution we came to was to use awk to prepend the time stamp in front of every line using a pipe.
Specifically, our solution was the following:


server_application 2>&1 | awk '{ print strftime("%Y-%m-%d %H:%M:%S"), $0; fflush(); }'

What we did there was to start our binary server_application, redirect stderr to stdout (using 2>&1) so that we will have only one stream and then we read the lines one by one using awk and printed the time stamp right before the line ($0) using strftime.
The strftime() function formats the broken-down time according to the format specification format.
fflushforces a write of all user-space buffered data for the given output or update stream via the stream’s underlying write function. We call it at each line to make sure that we do not cause additional delay in presenting the data due to buffering limitations caused by our prints.

Example


$ echo -e "hi\nHI" 2>&1 | awk '{ print strftime("%Y-%m-%d %H:%M:%S"), $0; fflush(); }'
2017-06-21 20:33:41 hi
2017-06-21 20:33:41 HI


How to execute `find` that ignores .git directories

Trying to find a source code file by its content using find and -exec grep, can some times result in getting results from the repository .git folders as well.

This behavior not only does it provide results you do not need but it also makes your search slower.
Below, we propose a couple of solutions on how to make a more efficient search.

Example 1: Ignore all .git folders no matter where they are in the search path

For find to ignore all .git folders, even if they appear on the first level of directories or any in-between until the last one, add -not -path '*/\.git*' to your command as in the example below.
This parameter will instruct find to filter out any file that has anywhere in its path the folder .git. This is very helpful in case a project has dependencies in other projects (repositories) that are part of the internal structure.


find . -type f -not -path '*/\.git/*';

Note, if you are using svn use:


find . -type f -not -path '*/\.svn/*';

Example 2: Ignore all hidden files and folders

To ignore all hidden files and folders from your find results add -not -path '*/\.*' to your command.


find . -not -path '*/\.*';

This parameter instructs find to ignore any file that has anywhere in its path the string /. which is any hidden file or folder in the search path!


How to get the pid of the last executed command that was sent to the background in a bash shell

Recently we came to the need of writing a bash script that needed periodically to check if a specific process, that was started by the script, had ended and restart it (something like watchdog but with not so many features).

To achieve this, we used the one of the shell special parameters, the $!. Like all other special parameters $! may only be referenced and the user cannot make an assignment to it.

($!) Expands to the process ID of the job most recently placed into the background, whether executed as an asynchronous command or using the bg builtin command.

From GNU.org: https://www.gnu.org/software/bash/manual/html_node/Special-Parameters.html#index-_0021-1

Example of Usage

In this example we wanted to get the PID of the application called server to be used later on in the script.


server &
echo $!; #This will print the process ID of the 'server' application


File permissions change date

Recently we wanted to check when did the permissions of a specific file changed.
Unfortunately, there exists no such flag and we do not have a 100% working solution for it.

What we did was to check the last modification time of the file status information (ctime) using the ls -lc command.
This command could indicate the last permissions change time but it is not a reliable source as it represents the modification time of other elements as well.

The modification time of the file status information (ctime) gets updated when any inode information regarding the file changes.
This means that the modification time of the file status information (ctime) will get updated when any of the following changes:

  • owner – The numeric user ID (UID) of the file’s owner.
  • group – The numeric group ID (GID) of the file’s group.
  • link count – The number of links to the file.
  • mode – The bit string that indicated the permissions and privileges
  • serial – The serial number of the file.
  • device – The numeric ID of the device containing the file.

Explanation of ls parameters

  • The parameter -c of the ls command when used with the -l will show ctime and sort by name.
  • The parameter -c of the ls command when used with the -l and the -t will show ctime and sort by ctime (newest first).

Example that demonstrates that we get different values in the time column of -l when -c is used

$ ls -lc ~/.ssh/
total 28
-rwx------. 1 george george  225 May 16 17:05 config
-rwx------. 1 george george 1743 Jun  2 13:36 id_rsa
-rwxrwx---. 1 george george  405 May 16 17:05 id_rsa.pub
-rwxrwx---. 1 george george   32 May 16 17:05 Details.txt
-rw-r--r--. 1 george george 9155 May 30 14:32 known_hosts

$ ls -l ~/.ssh/
total 28
-rwx------. 1 george george  225 Mar 22 11:36 config
-rwx------. 1 george george 1743 Jan 25 10:22 id_rsa
-rwxrwx---. 1 george george  405 Jan 25 10:22 id_rsa.pub
-rwxrwx---. 1 george george   32 Jan 25 10:22 Details.txt
-rw-r--r--. 1 george george 9155 May 30 14:32 known_hosts

Create a .tar file with different compression methods

The following commands will create .tar archives  and compress them using the different methods that are available. We provide multiple solutions, each one for a different type of .tar archive depending on the compression method that is desired.

For .tar archives

tar -c -f archive.tar $FILES_TO_ARCHIVE;

For .tar.bz2 archives

tar -c -j -f archive.tar.bz2 $FILES_TO_ARCHIVE;

For .tar.xz archives

tar -c -J -f archive.tar.xz $FILES_TO_ARCHIVE;

For .tar.gz and .tgz archives

tar -c -z -f archive.tar.gz $FILES_TO_ARCHIVE;

tar Parameters Legend

  • -z or --gzip instructs tar to filter the archive through gzip
  • -j or --bzip2 filters the archive through bzip2
  • -J or --xz filters the archive through xz
  • -f or --file=OUTPUT uses the archive file OUTPUT
  • -c or --create a new archive

Bonus Example: Create a tar.xz archive using the current date in the archive name

The following command will create an archive out of the folders Folder1 and Folder2 and then it will compress it to the .tar.xz format.
The filename of the archive will contain the current date in the format YYYY-MM-DD.

tar -c -J  -f archive.`date +%F`.tar.xz Folder1 Folder2;

The above command will result in something similar to:

archive.2017-06-04.tar.xz

Script to clone all git repositories from all projects of a privately hosted Bitbucket server

The following script can download all git repositories from all of the projects that you have access to on a privately hosted Bitbucket server.

The execution work-flow of this script is as follows:

  1. It will ask for your username (the one you use to login on the Bitbucket server)
  2. Then it will ask for your password, the password will not be visible on screen as you type because we disabled the echo functionality for that step.
  3. Later, you will be prompted to provide the URL of the server, for this step be sure to define the protocol if it is http or https and the correct port number as well (e.g. https://bitbucket.bytefreaks.net:7990)
  4. Finally, you will be requested to give the location to where the repositories should be cloned to.
  5. Then the script will connect to the server, get the list of projects that you have access to and for each project retrieve the repositories of the project and clone them in the designated folder.
clone.all.hosted.bitbucket.sh (compressed) (34 downloads)

#!/bin/bash

echo -n "Username: ";
read username;
echo -n "Password: "; 
#Disabling echo, so that password will not be visible on screen
read -s password
#Enabling echo
echo

echo -n "Server (e.g https://repository.bytefreaks.net:7990): ";
read server;
echo -n "Location to clone repositories in: ";
read location;

mkdir -p "$location";
cd "$location";

#Getting all projects
curl --user "$username:$password" "$server/rest/api/1.0/projects/" | \
  grep -oP '"key":"\K\w+' | xargs -I {} -n 1 -I_project -- sh -c \
    "curl --user \"$username:$password\" \"$server/rest/api/1.0/projects/_project/repos\" | grep -o '\"ssh:[^ ,]\+' | xargs -L1 git clone";

exit 0;

clone.all.hosted.bitbucket.sh (compressed) (34 downloads)

 

Notes for the future:

  • Separate the cloned repositories per project
  • Support for people that have hundreds of projects and/or hundreds of repositories using the paging functionality

papouch: TMU – USB thermometer

Today, we found in stock some USB thermometers by papouch, which we decided to put to use.
We wanted to create a small bash script that would take the measurements from the thermometers and log them along with the system date/time.
After doing some minor research we got to the product website, where it had a lot of useful information about the device, device drivers and source code which can utilize the device on a Windows machine.

Unfortunately for us, there was no source code for a simple bash script on Linux.

Before we continue, lets fill our heads with some information on the device:

TMU is a simple thermometer with a USB interface. The thermometer uses the USB interface for communication and also as a power source. It measures temperatures from –55 °C to +125 °C (with 0.1 °C resolution). The communication utilizes a simple ASCII protocol. Temperature values are transmitted in degrees Celsius; no numerical conversion is necessary.

–From https://www.papouch.com/en/shop/product/tmu-usb-thermometer/

The operating system on our machine was GNU/Linux CentOS 7, after plugging in the devices, we issued the command lsusb from which we saw that the OS had recognized the devices.
From the manual we read that the interface for communication of the device with the computer is implemented via a serial port.
The configuration parameters of the serial port that the device creates were the following:

COMMUNICATION PROTOCOL
TMU cannot receive instructions, it can only send out the temperature values in regular time intervals (approx. 10 seconds).
The temperature is send in a format that is compatible with the Spinel protocol.
The thermometer’s serial line parameters are:

Speed : 9,600 Baud
Number of data bits : 8
Parity : none
Number of stop-bits : 1

— From https://www.papouch.com/en/shop/product/tmu-usb-thermometer/tmu_en.pdf/_downloadFile.php

Since the newly attached devices were USB-to-Serial devices, we knew that they would create ttyUSBx devices in the /dev folder.
Indeed, after checking into the /dev folder, there were two newly created devices ttyUSB0 and ttyUSB1, one for each device.

We tried to connect to the devices using various methods and attempted to redirect the output so that we could parse it.
To our surprise, the data would ‘disappear’ from the pipe…
We could see the data on the screen when we had no pipes following and we could even replace the \r character with \n so that each new information block would appear in a new line. But, whenever we tried to do additional formatting, e.g. remove all characters that are not part of the temperature description, the whole data would vanish..

Our solution

For us process substitution did the trick!
Process substitution feeds the output of a process into the stdin of another process.
We redirected the stdout that was being generated while reading the data from the serial port to another process from where we were able to normally process them.

The following example, reads the data from the serial port, from each line it discards all characters except for characters at the positions 6 until 11 where the temperature information is presented according to the documentation.

sudo sh -c "cat < /dev/ttyUSB0" 1> >(while read line; do echo $line | cut -c6-11; done);

The above command would turn data of this format:

*B1E1+026.0
*B1E1+026.1

To this format:

+026.0
+026.1

And so we could start the development of our script.

Our script

The following script will prepend the current date and time on each line (right before the temperature reading).

 sudo sh -c "cat < /dev/ttyUSB0" 1> >(while read line; do echo $line | cut -c6-11 | xargs -L 1 echo `date`; done); 

Another solution, using miniterm.py

It has come to our attention that some times the thermometers do no work as expected using the cat command.
So, we propose an alternative using miniterm.py.
miniterm.py is a very simple serial terminal and is part of pySerial.

 miniterm.py --echo --eol CR --quiet /dev/ttyUSB0 1> >(while read line; do echo $line | cut -c6-11 | xargs -L 1 echo `date`; done); 

Some details on the format from the manual:

The protocol format is shown in this example.
Example (the data are sent without the space characters from the TMU)

*B1E1+026.1
  • 1 Byte; Prefix: the character *
  • 1 Byte; Format code: the character B
  • 1 Byte; The address of the thermometer: the character 1
  • 2 Bytes; Device instruction code: the characters E1
  • 6 Bytes; Actual temperature value. It can be number from –055.0 to +125.0 or string Err.
    An ASCII string representing the temperature value including the sign. If there is a thermal sensor’s error, the Err string is transmitted.
  • 1 Byte; Terminating character: Carriage Return (Decimal: 13, Hex: 0Dh, Binary: 00001101, Character \r)

 


Find files that were created, modified or accessed in the last N minutes

Find all files in $my_folder that their status changed in the last 60 minutes

find $my_folder -cmin -60

Find all files in $my_folder that their data were modified in the last 60 minutes

find $my_folder -mmin -60

Find all files in $my_folder that they were accessed in the last 60 minutes

find $my_folder -amin -60

Please remember to use negative values for the minutes. e.g. use -60 and not 60.

More examples

Find all files in $my_folder that their status changed in the last 60 minutes AND they were accessed in the last 10 minutes

find $my_folder -cmin -60 -amin -10

Find all files in $my_folder that their status changed in the last 60 minutes OR they were accessed in the last 10 minutes

find $my_folder \( -cmin -60 -o -amin -10 \)

Notes on find command

  • -cmin n Matches files which their status was last changed n minutes ago.
  • -mmin n Matches files which which data was last modified n minutes ago.
  • -amin n Matches files which they were last accessed n minutes ago.
  • -o is the logical Or operator. The second expression  is not evaluated if the first expression is true.

How to add automatically all empty folders in git repository 1

Since you are searching for this issue, you must have realised that git does not support storing empty folders/directories.

Currently the design of the Git index (staging area) only permits files to be listed, and nobody competent enough to make the change to allow empty directories has cared enough about this situation to remedy it.

Directories are added automatically when adding files inside them. That is, directories never have to be added to the repository, and are not tracked on their own.
— From https://git.wiki.kernel.org/index.php/Git_FAQ#Can_I_add_empty_directories.3F

All the content is stored as tree and blob objects, with trees corresponding to UNIX directory entries and blobs corresponding more or less to inodes or file contents. A single tree object contains one or more tree entries, each of which contains a SHA-1 pointer to a blob or subtree with its associated mode, type, and filename.
— From https://git-scm.com/book/en/v2/Git-Internals-Git-Objects

Below we propose two solutions, depending on how you want to use those empty folders.

Solution A – The folders will always be empty

There are scenarios where the empty folders should always remain empty on git no matter what the local copy has inside them.
Such a scenario would be, wanting to add on git folders where you will build your objects and/or put temporary/cached data.
In such scenarios it is important to have the structure available but never to add those files in git.

To achieve that, we add a .gitignore file in every empty folder containing the following:

# git does not allow empty directories.
# Yet, we need to add this empty directory on git.
# To achieve that, we created this .gitignore file, so that the directory will not be empty thus enabling us to commit it.
# Since we want all generated files/folders in this directory to be ignored by git, we add a rule for this.
*
# And then add an exception for this specifc file (so that we can commit it).
!.gitignore
.gitignore (compressed) (72 downloads)

The above .gitignore file, instructs git to add this file on the repository, and thus add the folder itself while ignoring ALL other files.
You can always update your .gitignore file to allow additional files to be added on the repository at any time.
This way you will never get prompted for temporary files that they were modified/created as part of the status of your repository.

Automation

A way to achieve this automatically, and place a copy of the .gitignore file in every empty folder would be to use the following command to copy an existing .gitignore file in all empty folders.

find $PATH_TO_REPOSITORY -type d ! -path "*.git*" -empty -exec cp .gitignore '{}'/ \;

The above command assumes that there is a .gitignore file in the folder that we are executing from, which it will copy in every empty directory inside the folder that the variable $PATH_TO_REPOSITORY is pointing to.

.gitignore (compressed) (72 downloads)

Solution B – Files will be added eventually to the folders

There are scenarios where the empty folders will be filled at a later stage and we want allow those files on git.
Such a scenario would be, wanting to add on git folders where right now are empty but in some time we will add new source code or resources there.

To achieve that, we add an empty .gitkeep file in every empty folder.

The above .gitkeep file is nothing more than a placeholder.
It is not documented, because it’s not a feature of Git.
It’s a dummy file, so git will not process the empty directory, since git tracks only files.
Once you add other files to the folder, you can safely delete it.
This way you will always get prompted for files that they were modified/created as part of the status of your repository.

Automation

A way to achieve this automatically, and place a copy of the .gitkeep file in every empty folder would be to use the following command to create an empty .gitkeep file in all empty folders.

find $PATH_TO_REPOSITORY -type d ! -path "*.git*" -empty -exec touch '{}'/.gitkeep \;

The above command will create in every empty directory inside the folder that the variable $PATH_TO_REPOSITORY is pointing to a new .gitkeep file.

Finally, push the changes to the git repository

After you create/copy the files, navigate to the repository, add all the new files to the commit, commit them and push them to the repository.

cd $PATH_TO_REPOSITORY;
# Create a new branch
git checkout -b empty_folders;
# Add all modified files to the next commit.
git add .;
git commit -m "Minor change: Adding all empty folders to the repository.";
git push -u origin empty_folders;