Bash


Create a .tar file with different compression methods

The following commands will create .tar archives  and compress them using the different methods that are available. We provide multiple solutions, each one for a different type of .tar archive depending on the compression method that is desired.

For .tar archives

tar -c -f archive.tar $FILES_TO_ARCHIVE;

For .tar.bz2 archives

tar -c -j -f archive.tar.bz2 $FILES_TO_ARCHIVE;

For .tar.xz archives

tar -c -J -f archive.tar.xz $FILES_TO_ARCHIVE;

For .tar.gz and .tgz archives

tar -c -z -f archive.tar.gz $FILES_TO_ARCHIVE;

tar Parameters Legend

  • -z or --gzip instructs tar to filter the archive through gzip
  • -j or --bzip2 filters the archive through bzip2
  • -J or --xz filters the archive through xz
  • -f or --file=OUTPUT uses the archive file OUTPUT
  • -c or --create a new archive

Bonus Example: Create a tar.xz archive using the current date in the archive name

The following command will create an archive out of the folders Folder1 and Folder2 and then it will compress it to the .tar.xz format.
The filename of the archive will contain the current date in the format YYYY-MM-DD.

tar -c -J  -f archive.`date +%F`.tar.xz Folder1 Folder2;

The above command will result in something similar to:

archive.2017-06-04.tar.xz
Advertisements

Script to clone all git repositories from all projects of a privately hosted Bitbucket server 1

The following script can download all git repositories from all of the projects that you have access to on a privately hosted Bitbucket server.

The execution work-flow of this script is as follows:

  1. It will ask for your username (the one you use to login on the Bitbucket server)
  2. Then it will ask for your password, the password will not be visible on screen as you type because we disabled the echo functionality for that step.
  3. Later, you will be prompted to provide the URL of the server, for this step be sure to define the protocol if it is http or https and the correct port number as well (e.g. https://bitbucket.bytefreaks.net:7990)
  4. Finally, you will be requested to give the location to where the repositories should be cloned to.
  5. Then the script will connect to the server, get the list of projects that you have access to and for each project retrieve the repositories of the project and clone them in the designated folder.
clone.all.hosted.bitbucket.sh (compressed) (387 downloads)
#!/bin/bash

echo -n "Username: ";
read username;
echo -n "Password: "; 
#Disabling echo, so that password will not be visible on screen
read -s password
#Enabling echo
echo

echo -n "Server (e.g https://repository.bytefreaks.net:7990): ";
read server;
echo -n "Location to clone repositories in: ";
read location;

mkdir -p "$location";
cd "$location";

#Getting all projects
curl --user "$username:$password" "$server/rest/api/1.0/projects/" | \
  grep -oP '"key":"\K\w+' | xargs -I {} -n 1 -I_project -- sh -c \
    "curl --user \"$username:$password\" \"$server/rest/api/1.0/projects/_project/repos\" | grep -o '\"ssh:[^ ,]\+' | xargs -L1 git clone";

exit 0;

clone.all.hosted.bitbucket.sh (compressed) (387 downloads)

 

Notes for the future:

  • Separate the cloned repositories per project
  • Support for people that have hundreds of projects and/or hundreds of repositories using the paging functionality

papouch: TMU – USB thermometer

Today, we found in stock some USB thermometers by papouch, which we decided to put to use.
We wanted to create a small bash script that would take the measurements from the thermometers and log them along with the system date/time.
After doing some minor research we got to the product website, where it had a lot of useful information about the device, device drivers and source code which can utilize the device on a Windows machine.

Unfortunately for us, there was no source code for a simple bash script on Linux.

Before we continue, lets fill our heads with some information on the device:

TMU is a simple thermometer with a USB interface. The thermometer uses the USB interface for communication and also as a power source. It measures temperatures from –55 °C to +125 °C (with 0.1 °C resolution). The communication utilizes a simple ASCII protocol. Temperature values are transmitted in degrees Celsius; no numerical conversion is necessary.

–From https://www.papouch.com/en/shop/product/tmu-usb-thermometer/

The operating system on our machine was GNU/Linux CentOS 7, after plugging in the devices, we issued the command lsusb from which we saw that the OS had recognized the devices.
From the manual we read that the interface for communication of the device with the computer is implemented via a serial port.
The configuration parameters of the serial port that the device creates were the following:

COMMUNICATION PROTOCOL
TMU cannot receive instructions, it can only send out the temperature values in regular time intervals (approx. 10 seconds).
The temperature is send in a format that is compatible with the Spinel protocol.
The thermometer’s serial line parameters are:

Speed : 9,600 Baud
Number of data bits : 8
Parity : none
Number of stop-bits : 1

— From https://www.papouch.com/en/shop/product/tmu-usb-thermometer/tmu_en.pdf/_downloadFile.php

Since the newly attached devices were USB-to-Serial devices, we knew that they would create ttyUSBx devices in the /dev folder.
Indeed, after checking into the /dev folder, there were two newly created devices ttyUSB0 and ttyUSB1, one for each device.

We tried to connect to the devices using various methods and attempted to redirect the output so that we could parse it.
To our surprise, the data would ‘disappear’ from the pipe…
We could see the data on the screen when we had no pipes following and we could even replace the \r character with \n so that each new information block would appear in a new line. But, whenever we tried to do additional formatting, e.g. remove all characters that are not part of the temperature description, the whole data would vanish..

Our solution

For us process substitution did the trick!
Process substitution feeds the output of a process into the stdin of another process.
We redirected the stdout that was being generated while reading the data from the serial port to another process from where we were able to normally process them.

The following example, reads the data from the serial port, from each line it discards all characters except for characters at the positions 6 until 11 where the temperature information is presented according to the documentation.

sudo sh -c "cat < /dev/ttyUSB0" 1> >(while read line; do echo $line | cut -c6-11; done);

The above command would turn data of this format:

*B1E1+026.0
*B1E1+026.1

To this format:

+026.0
+026.1

And so we could start the development of our script.

Our script

The following script will prepend the current date and time on each line (right before the temperature reading).

 sudo sh -c "cat < /dev/ttyUSB0" 1> >(while read line; do echo $line | cut -c6-11 | xargs -L 1 echo `date`; done); 

Another solution, using miniterm.py

It has come to our attention that some times the thermometers do no work as expected using the cat command.
So, we propose an alternative using miniterm.py.
miniterm.py is a very simple serial terminal and is part of pySerial.

 miniterm.py --echo --eol CR --quiet /dev/ttyUSB0 1> >(while read line; do echo $line | cut -c6-11 | xargs -L 1 echo `date`; done); 

Some details on the format from the manual:

The protocol format is shown in this example.
Example (the data are sent without the space characters from the TMU)

*B1E1+026.1
  • 1 Byte; Prefix: the character *
  • 1 Byte; Format code: the character B
  • 1 Byte; The address of the thermometer: the character 1
  • 2 Bytes; Device instruction code: the characters E1
  • 6 Bytes; Actual temperature value. It can be number from –055.0 to +125.0 or string Err.
    An ASCII string representing the temperature value including the sign. If there is a thermal sensor’s error, the Err string is transmitted.
  • 1 Byte; Terminating character: Carriage Return (Decimal: 13, Hex: 0Dh, Binary: 00001101, Character \r)

 


Find files that were created, modified or accessed in the last N minutes

Find all files in $my_folder that their status changed in the last 60 minutes

find $my_folder -cmin -60

Find all files in $my_folder that their data were modified in the last 60 minutes

find $my_folder -mmin -60

Find all files in $my_folder that they were accessed in the last 60 minutes

find $my_folder -amin -60

Please remember to use negative values for the minutes. e.g. use -60 and not 60.

More examples

Find all files in $my_folder that their status changed in the last 60 minutes AND they were accessed in the last 10 minutes

find $my_folder -cmin -60 -amin -10

Find all files in $my_folder that their status changed in the last 60 minutes OR they were accessed in the last 10 minutes

find $my_folder \( -cmin -60 -o -amin -10 \)

Notes on find command

  • -cmin n Matches files which their status was last changed n minutes ago.
  • -mmin n Matches files which which data was last modified n minutes ago.
  • -amin n Matches files which they were last accessed n minutes ago.
  • -o is the logical Or operator. The second expression  is not evaluated if the first expression is true.

How to add automatically all empty folders in git repository 4

Since you are searching for this issue, you must have realised that git does not support storing empty folders/directories.

Currently the design of the Git index (staging area) only permits files to be listed, and nobody competent enough to make the change to allow empty directories has cared enough about this situation to remedy it.

Directories are added automatically when adding files inside them. That is, directories never have to be added to the repository, and are not tracked on their own.
— From https://git.wiki.kernel.org/index.php/Git_FAQ#Can_I_add_empty_directories.3F

All the content is stored as tree and blob objects, with trees corresponding to UNIX directory entries and blobs corresponding more or less to inodes or file contents. A single tree object contains one or more tree entries, each of which contains a SHA-1 pointer to a blob or subtree with its associated mode, type, and filename.
— From https://git-scm.com/book/en/v2/Git-Internals-Git-Objects

Below we propose two solutions, depending on how you want to use those empty folders.

Solution A – The folders will always be empty

There are scenarios where the empty folders should always remain empty on git no matter what the local copy has inside them.
Such a scenario would be, wanting to add on git folders where you will build your objects and/or put temporary/cached data.
In such scenarios it is important to have the structure available but never to add those files in git.

To achieve that, we add a .gitignore file in every empty folder containing the following:

# git does not allow empty directories.
# Yet, we need to add this empty directory on git.
# To achieve that, we created this .gitignore file, so that the directory will not be empty thus enabling us to commit it.
# Since we want all generated files/folders in this directory to be ignored by git, we add a rule for this.
*
# And then add an exception for this specifc file (so that we can commit it).
!.gitignore
.gitignore (compressed) (349 downloads)

The above .gitignore file, instructs git to add this file on the repository, and thus add the folder itself while ignoring ALL other files.
You can always update your .gitignore file to allow additional files to be added on the repository at any time.
This way you will never get prompted for temporary files that they were modified/created as part of the status of your repository.

Automation

A way to achieve this automatically, and place a copy of the .gitignore file in every empty folder would be to use the following command to copy an existing .gitignore file in all empty folders.

find $PATH_TO_REPOSITORY -type d ! -path "*.git*" -empty -exec cp .gitignore '{}'/ \;

The above command assumes that there is a .gitignore file in the folder that we are executing from, which it will copy in every empty directory inside the folder that the variable $PATH_TO_REPOSITORY is pointing to.

.gitignore (compressed) (349 downloads)

Solution B – Files will be added eventually to the folders

There are scenarios where the empty folders will be filled at a later stage and we want allow those files on git.
Such a scenario would be, wanting to add on git folders where right now are empty but in some time we will add new source code or resources there.

To achieve that, we add an empty .gitkeep file in every empty folder.

The above .gitkeep file is nothing more than a placeholder.
It is not documented, because it’s not a feature of Git.
It’s a dummy file, so git will not process the empty directory, since git tracks only files.
Once you add other files to the folder, you can safely delete it.
This way you will always get prompted for files that they were modified/created as part of the status of your repository.

Automation

A way to achieve this automatically, and place a copy of the .gitkeep file in every empty folder would be to use the following command to create an empty .gitkeep file in all empty folders.

find $PATH_TO_REPOSITORY -type d ! -path "*.git*" -empty -exec touch '{}'/.gitkeep \;

The above command will create in every empty directory inside the folder that the variable $PATH_TO_REPOSITORY is pointing to a new .gitkeep file.

Finally, push the changes to the git repository

After you create/copy the files, navigate to the repository, add all the new files to the commit, commit them and push them to the repository.

cd $PATH_TO_REPOSITORY;
# Create a new branch
git checkout -b empty_folders;
# Add all modified files to the next commit.
git add .;
git commit -m "Minor change: Adding all empty folders to the repository.";
git push -u origin empty_folders;

Delete all empty directories and all directories containing empty directories

Assuming you have a complex filesystem from which you want to delete all empty directories and all directories containing empty directories recursively, while leaving intact the rest.
You can use the following command.

find . -type d -empty -delete;

The configuration we used is the following:

  • -type d restricts the results to directories only
  • -empty restricts to empty directories only
  • -delete removes the directories that matched

The above code will delete all directories that are either empty or contain empty directories in the current directory.
It will successfully delete all empty directories even if they contain a large number of empty directories in any structure inside them.


Status of an executing dd 1

Recently, we were cloning a large hard disk on another using dd.
This operation took a really long time, at some point we got curious on what the status of the execution was.
Due to the minimal output dd offers, there was no indication for us whether the system was still copying and if it had a long way to go or not.

Fortunately, the developers of dd added a feature where sending a USR1 signal to a running dd process makes it print I/O statistics to standard error and then resume copying.

To achieve that we used a second terminal and followed these steps:

  1. We used pgrep to look up the running process based on its name and get the dd running process ID (PID): pgrep ^dd$ .
  2. We passed that PID to kill -USR1 which triggered the printing of the statistics on the terminal where dd was executing: kill -USR1 $(pgrep ^dd$).

Solution

kill -USR1 $(pgrep ^dd$);

Bonus

Additionally, we wanted to have dd statistics printed automatically every minute.
To achieve that, we used watchwatch executes a program periodically, showing it’s output in full-screen.
We defined the interval in seconds using the parameter -n. (Please note that, the command will not allow less than 0.1 second interval.)

In the end, our command became as follows:

watch -n 60 kill -USR1 $(pgrep ^dd$)

The above command was sending a USR1 signal to dd via the kill application every minute (60 seconds) forcing it to print on standard output the I/O statistics.

Example

On terminal 1, we executed the command dd if=/dev/sda of=/dev/sdb;, which will copy disk sda over sdb.

On terminal 2, we executed the command kill -USR1 $(pgrep ^dd$);, which forced dd to print I/O statistics back on terminal 1.

0+49728 records in
7218+0 records out
3695616 bytes (3.7 MB) copied, 2.85812 s, 1.3 MB/s
0+78673 records in
11443+0 records out
5858816 bytes (5.9 MB) copied, 4.49477 s, 1.3 MB/s
0+99003 records in
14386+0 records out
7365632 bytes (7.4 MB) copied, 5.75575 s, 1.3 MB/s
^C0+172104 records in
24918+0 records out
12758016 bytes (13 MB) copied, 10.197 s, 1.3 MB/s

Using .tgz files 1

Create

To create a .tgz file, we used tar with the following parameters -czf:

  • -c or --create will create a new archive.
  • -z– or --gzip or --gunzip or --ungzip will filter the archive through gzip and compress it.
  • -f or --file=ARCHIVE will use archive file or device ARCHIVE. If this option is not given, tar will first examine the environment variable TAPE. If it is set, its value will be used as the archive name. Otherwise, tar will assume the compiled-in default.

Example:

tar -czf $ARCHIVE_FILE_NAME.tgz $PATH_TO_COMPRESS;

Please note that the order of the parameters will not change the result.

Extract

To extract a .tgz or .tar.gz file using tar we used the following parameters -xzf:

  • -x or --extract --get will extract the files from the archive. Arguments are optional. When given, they specify names of the archive members to be extracted.
  • -z– or --gzip or --gunzip or --ungzip will filter the archive through gzip and decompress it.
  • -f or --file=ARCHIVE will use archive file or device ARCHIVE. If this option is not given, tar will first examine the environment variable TAPE. If it is set, its value will be used as the archive name. Otherwise, tar will assume the compiled-in default.

Example:

tar -xzf $ARCHIVE_FILE_NAME.tgz;


GNU/Linux find: Get results relative to the directory searching in, instead of directory shell is in

Recently we wanted to create a list of files that could be found in a specific folder.
For that list we wanted the paths of the files to be relative to the folder we were searching in, instead of them being relative to the folder our shell was currently in.

To achieve that, we used cd to navigate into that folder and searched from there locally.
We used a sub-shell to achieve this, which was not needed, but because we did not want to change the current directory of our shell, it was needed.

The command was as follows:

(cd toThe/Path/WeAre/Interested/In && find .)

instead of:

find toThe/Path/WeAre/Interested/In

Since we were interested in getting all files, we did not put any filters on find.
Of course you can use find normally and modify it as you please.

Finally, since we wanted the list of files to be saved in a text file, we redirected the output of the above command to a file in the current working directory

(cd toThe/Path/WeAre/Interested/In && find .) > interestingFiles.txt

Copy a symbolic link without traversing it

Recently we needed to copy a Symbolic Link on a disk image we would deploy on an embedded device.
For this reason it was important for us to copy the Symbolic Link and not the file it was pointing to as that link would become valid once the machine would boot from the image.

To achieve that we used -P which instructs cp to never follow symbolic links in source. In other words it would not traverse the symbolic link and copy the symbolic link itself.

Notes:

  • --no-dereference is the same as -P.
  • -P uses a capital P.