Bash


How to add automatically all empty folders in git repository 4

Since you are searching for this issue, you must have realised that git does not support storing empty folders/directories.

Currently the design of the Git index (staging area) only permits files to be listed, and nobody competent enough to make the change to allow empty directories has cared enough about this situation to remedy it.

Directories are added automatically when adding files inside them. That is, directories never have to be added to the repository, and are not tracked on their own.
— From https://git.wiki.kernel.org/index.php/Git_FAQ#Can_I_add_empty_directories.3F

All the content is stored as tree and blob objects, with trees corresponding to UNIX directory entries and blobs corresponding more or less to inodes or file contents. A single tree object contains one or more tree entries, each of which contains a SHA-1 pointer to a blob or subtree with its associated mode, type, and filename.
— From https://git-scm.com/book/en/v2/Git-Internals-Git-Objects

Below we propose two solutions, depending on how you want to use those empty folders.

Solution A – The folders will always be empty

There are scenarios where the empty folders should always remain empty on git no matter what the local copy has inside them.
Such a scenario would be, wanting to add on git folders where you will build your objects and/or put temporary/cached data.
In such scenarios it is important to have the structure available but never to add those files in git.

To achieve that, we add a .gitignore file in every empty folder containing the following:

# git does not allow empty directories.
# Yet, we need to add this empty directory on git.
# To achieve that, we created this .gitignore file, so that the directory will not be empty thus enabling us to commit it.
# Since we want all generated files/folders in this directory to be ignored by git, we add a rule for this.
*
# And then add an exception for this specifc file (so that we can commit it).
!.gitignore
.gitignore (compressed) (324 downloads)

The above .gitignore file, instructs git to add this file on the repository, and thus add the folder itself while ignoring ALL other files.
You can always update your .gitignore file to allow additional files to be added on the repository at any time.
This way you will never get prompted for temporary files that they were modified/created as part of the status of your repository.

Automation

A way to achieve this automatically, and place a copy of the .gitignore file in every empty folder would be to use the following command to copy an existing .gitignore file in all empty folders.

find $PATH_TO_REPOSITORY -type d ! -path "*.git*" -empty -exec cp .gitignore '{}'/ \;

The above command assumes that there is a .gitignore file in the folder that we are executing from, which it will copy in every empty directory inside the folder that the variable $PATH_TO_REPOSITORY is pointing to.

.gitignore (compressed) (324 downloads)

Solution B – Files will be added eventually to the folders

There are scenarios where the empty folders will be filled at a later stage and we want allow those files on git.
Such a scenario would be, wanting to add on git folders where right now are empty but in some time we will add new source code or resources there.

To achieve that, we add an empty .gitkeep file in every empty folder.

The above .gitkeep file is nothing more than a placeholder.
It is not documented, because it’s not a feature of Git.
It’s a dummy file, so git will not process the empty directory, since git tracks only files.
Once you add other files to the folder, you can safely delete it.
This way you will always get prompted for files that they were modified/created as part of the status of your repository.

Automation

A way to achieve this automatically, and place a copy of the .gitkeep file in every empty folder would be to use the following command to create an empty .gitkeep file in all empty folders.

find $PATH_TO_REPOSITORY -type d ! -path "*.git*" -empty -exec touch '{}'/.gitkeep \;

The above command will create in every empty directory inside the folder that the variable $PATH_TO_REPOSITORY is pointing to a new .gitkeep file.

Finally, push the changes to the git repository

After you create/copy the files, navigate to the repository, add all the new files to the commit, commit them and push them to the repository.

cd $PATH_TO_REPOSITORY;
# Create a new branch
git checkout -b empty_folders;
# Add all modified files to the next commit.
git add .;
git commit -m "Minor change: Adding all empty folders to the repository.";
git push -u origin empty_folders;
Advertisements

Delete all empty directories and all directories containing empty directories

Assuming you have a complex filesystem from which you want to delete all empty directories and all directories containing empty directories recursively, while leaving intact the rest.
You can use the following command.

find . -type d -empty -delete;

The configuration we used is the following:

  • -type d restricts the results to directories only
  • -empty restricts to empty directories only
  • -delete removes the directories that matched

The above code will delete all directories that are either empty or contain empty directories in the current directory.
It will successfully delete all empty directories even if they contain a large number of empty directories in any structure inside them.


Status of an executing dd 1

Recently, we were cloning a large hard disk on another using dd.
This operation took a really long time, at some point we got curious on what the status of the execution was.
Due to the minimal output dd offers, there was no indication for us whether the system was still copying and if it had a long way to go or not.

Fortunately, the developers of dd added a feature where sending a USR1 signal to a running dd process makes it print I/O statistics to standard error and then resume copying.

To achieve that we used a second terminal and followed these steps:

  1. We used pgrep to look up the running process based on its name and get the dd running process ID (PID): pgrep ^dd$ .
  2. We passed that PID to kill -USR1 which triggered the printing of the statistics on the terminal where dd was executing: kill -USR1 $(pgrep ^dd$).

Solution

kill -USR1 $(pgrep ^dd$);

Bonus

Additionally, we wanted to have dd statistics printed automatically every minute.
To achieve that, we used watchwatch executes a program periodically, showing it’s output in full-screen.
We defined the interval in seconds using the parameter -n. (Please note that, the command will not allow less than 0.1 second interval.)

In the end, our command became as follows:

watch -n 60 kill -USR1 $(pgrep ^dd$)

The above command was sending a USR1 signal to dd via the kill application every minute (60 seconds) forcing it to print on standard output the I/O statistics.

Example

On terminal 1, we executed the command dd if=/dev/sda of=/dev/sdb;, which will copy disk sda over sdb.

On terminal 2, we executed the command kill -USR1 $(pgrep ^dd$);, which forced dd to print I/O statistics back on terminal 1.

0+49728 records in
7218+0 records out
3695616 bytes (3.7 MB) copied, 2.85812 s, 1.3 MB/s
0+78673 records in
11443+0 records out
5858816 bytes (5.9 MB) copied, 4.49477 s, 1.3 MB/s
0+99003 records in
14386+0 records out
7365632 bytes (7.4 MB) copied, 5.75575 s, 1.3 MB/s
^C0+172104 records in
24918+0 records out
12758016 bytes (13 MB) copied, 10.197 s, 1.3 MB/s

Using .tgz files 1

Create

To create a .tgz file, we used tar with the following parameters -czf:

  • -c or --create will create a new archive.
  • -z– or --gzip or --gunzip or --ungzip will filter the archive through gzip and compress it.
  • -f or --file=ARCHIVE will use archive file or device ARCHIVE. If this option is not given, tar will first examine the environment variable TAPE. If it is set, its value will be used as the archive name. Otherwise, tar will assume the compiled-in default.

Example:

tar -czf $ARCHIVE_FILE_NAME.tgz $PATH_TO_COMPRESS;

Please note that the order of the parameters will not change the result.

Extract

To extract a .tgz or .tar.gz file using tar we used the following parameters -xzf:

  • -x or --extract --get will extract the files from the archive. Arguments are optional. When given, they specify names of the archive members to be extracted.
  • -z– or --gzip or --gunzip or --ungzip will filter the archive through gzip and decompress it.
  • -f or --file=ARCHIVE will use archive file or device ARCHIVE. If this option is not given, tar will first examine the environment variable TAPE. If it is set, its value will be used as the archive name. Otherwise, tar will assume the compiled-in default.

Example:

tar -xzf $ARCHIVE_FILE_NAME.tgz;


GNU/Linux find: Get results relative to the directory searching in, instead of directory shell is in

Recently we wanted to create a list of files that could be found in a specific folder.
For that list we wanted the paths of the files to be relative to the folder we were searching in, instead of them being relative to the folder our shell was currently in.

To achieve that, we used cd to navigate into that folder and searched from there locally.
We used a sub-shell to achieve this, which was not needed, but because we did not want to change the current directory of our shell, it was needed.

The command was as follows:

(cd toThe/Path/WeAre/Interested/In && find .)

instead of:

find toThe/Path/WeAre/Interested/In

Since we were interested in getting all files, we did not put any filters on find.
Of course you can use find normally and modify it as you please.

Finally, since we wanted the list of files to be saved in a text file, we redirected the output of the above command to a file in the current working directory

(cd toThe/Path/WeAre/Interested/In && find .) > interestingFiles.txt

Copy a symbolic link without traversing it

Recently we needed to copy a Symbolic Link on a disk image we would deploy on an embedded device.
For this reason it was important for us to copy the Symbolic Link and not the file it was pointing to as that link would become valid once the machine would boot from the image.

To achieve that we used -P which instructs cp to never follow symbolic links in source. In other words it would not traverse the symbolic link and copy the symbolic link itself.

Notes:

  • --no-dereference is the same as -P.
  • -P uses a capital P.

 


Delete all files and keep the directory structure

Scenario

You have a complex folder structure and you want to remove all files and at the same time keep all folders intact.

We will present one method, using two variations of it that can achieve the above.
The method uses the GNU find command to find all files and delete them one by one.

Variation A

find . ! -type d -exec rm '{}' \;

This above command will search in the current directory and sub-directories for anything that is not a folder and then it will delete them.

  • find . – Searches in this folder, since we did not define depth, it will search in all sub-folders as well
  • ! -type dtype d instructs find to match all Directories, by adding the ! in front of the instruction it negates the result and instructs find to match anything but the Directories
  • -exec rm '{}' \; – for every result, the command after exec is executed. The filename replaces '{}' so that the results get deleted one by one.

Variation B

find . ! -type d -delete

In this example, we replaced -exec rm '{}' \; with the simpler to remember directive of -delete.


Get execution time in seconds

The following methods demonstrate different methods on how to compute the time a potion of code or script take to complete their execution.

Time Methods - Full Examples (175 downloads)

 

Method 1 – Using date

The following example will calculate the execution time in seconds by subtracting the system date and time in seconds since 1970-01-01 00:00:00 UTC once right before the script goes to the computation part and once right after.

In order to get the system date and time in seconds since 1970-01-01 00:00:00 UTC we use the command date +%s.

Time Methods - Full Examples (175 downloads)
#!/bin/bash

#Print the system date and time in seconds since 1970-01-01 00:00:00 UTC
startTime=$(date +%s);

#We pick a random number between 1 and 10.
#Then we delay the execution for that amount of seconds.
sleep $(( (RANDOM % 10) + 1 ));

endTime=$(date +%s);

#Subtract endTime from startTime to get the total execution time
totalTime=$(($endTime-$startTime));

echo "Process finished after $totalTime seconds";

exit 0;

Method 2 – Using bash internal SECONDS variable

The following example will calculate the execution time in seconds by reseting the bash internal variable SECONDS to 0, forcing the shell to continue counting from there.

Time Methods - Full Examples (175 downloads)
#!/bin/bash

#This variable expands to the number of seconds since the shell was started.
#We set it to 0, forcing the shell to continue counting from there.
SECONDS=0;

#We pick a random number between 1 and 10.
#Then we delay the execution for that amount of seconds.
sleep $(( (RANDOM % 10) + 1 ));

echo "Process finished after $SECONDS seconds";

exit 0;

Method 3 – Using bash time

The following example uses the bash time command, which reports the time consumed by a pipeline’s execution.
When time command is executed without its complete path, then the bash built-in time command is executed, instead of the GNU time command. We will use the bash time command in this example and we will use it to run a whole block of commands.
Please note that time command will return the time in seconds as a float (i.e. there will be decimal places. e.g. 1 will be printed as 1.00).

Time Methods - Full Examples (175 downloads)
#!/bin/bash

#The bash time command reports the time consumed by pipeline's execution
#When time command is executed without its complete path, then the bash built-in time command is executed, instead of the GNU time command.
#We will use the bash time command in this example and we will use it to run a whole block of commands.

#We change the output format of time to print elapsed real time in seconds.
TIMEFORMAT="%E";
#We pick a random number between 1 and 10.
#Then we delay the execution for that amount of seconds.
totalTime=`time ( sleep $(( (RANDOM % 10) + 1 )) ) 2>&1`;

#Please note that time command will return the time in seconds as a float (i.e. there will be decimal places. e.g. 1 will be printed as 1.00).
#This will happen as time has build-in more precision than the first two methods presented here.
echo "Process finished after $totalTime seconds";

totalTimeBlock=`time (
	sleep $(( (RANDOM % 10) + 1 ));
	sleep $(( (RANDOM % 10) + 1 ));
) 2>&1`;
echo "Block finished after $totalTimeBlock seconds";

exit 0;

Method 4 – Using GNU time

The GNU time command runs the specified program command with the given arguments.
When time command is executed without its complete path (in our case it was /usr/bin/time), then the bash built-in time command is executed, instead of the GNU time command. To make sure we use the GNU time command, we use which to get the full path of the time command.
Please note that time command will return the time in seconds as a float (i.e. there will be decimal places. e.g. 1 will be printed as 1.00).

Time Methods - Full Examples (175 downloads)
#!/bin/bash
#The time command runs the specified program command with the given arguments.
#When time command is executed without its complete path (in our case it was /usr/bin/time), then the bash built-in time command is executed, instead of the GNU time command.
#To make sure we use the GNU time command, we use which to get the full path of the time command.
time=`which time`;

#We pick a random number between 1 and 10.
#Then we delay the execution for that amount of seconds.
#We change the output format of time to print elapsed real time in seconds.
totalTime="$( $time -f '%e' sleep $(( (RANDOM % 10) + 1 )) 2>&1 1>/dev/null )";

#Please note that time command will return the time in seconds as a float (i.e. there will be decimal places. e.g. 1 will be printed as 1.00).
#This will happen as time has build-in more precision than the first two methods presented here.
echo "Process finished after $totalTime seconds";

exit 0;

Notes

RANDOM internal variable

Each time RANDOM internal variable is referenced, a random integer between 0 and 32767 is generated.

By using the RANDOM variable in this command $(( (RANDOM % 10) + 1 )); we perform a modulo on the random value with the static value 10. This way we force the range of valid values to be between 0 and 9.
Later, we add 1 to that value to shift the range to be between 1 and 10.


convertAndPrintSeconds – Convert seconds to minutes, hours and days in bash 1

The following code can be used to convert some time in seconds to minutes, hours and days in bash.
It will print on screen the converted values only if they are not 0. i.e If the resulting days is 0, it will not print the text for days at all.
You can use it in any script without copy pasting everything in it by executing the following command source ./convertAndPrintSeconds.sh.
Doing so, it will load to your script the function that is defined in convertAndPrintSeconds.sh, making it available for you to use (something like including code in C, with some caveats).

convertAndPrintSeconds.sh (compressed) (176 downloads)
#!/bin/bash

convertAndPrintSeconds() {

    local totalSeconds=$1;
    local seconds=$((totalSeconds%60));
    local minutes=$((totalSeconds/60%60));
    local hours=$((totalSeconds/60/60%24));
    local days=$((totalSeconds/60/60/24));
    (( $days > 0 )) && printf '%d days ' $days;
    (( $hours > 0 )) && printf '%d hours ' $hours;
    (( $minutes > 0 )) && printf '%d minutes ' $minutes;
    (( $days > 0 || $hours > 0 || $minutes > 0 )) && printf 'and ';
    printf '%d seconds\n' $seconds;
}

Usage

Function convertAndPrintSeconds accepts one parameter, the positive integer number that represents the seconds count we want to convert.

Example

$ source ./time.sh 
$ convertAndPrintSeconds 10
10 seconds
$ convertAndPrintSeconds 100
1 minutes and 40 seconds
$ convertAndPrintSeconds 1000
16 minutes and 40 seconds
$ convertAndPrintSeconds 10000
2 hours 46 minutes and 40 seconds
$ convertAndPrintSeconds 100000
1 days 3 hours 46 minutes and 40 seconds
$ convertAndPrintSeconds 1000000
11 days 13 hours 46 minutes and 40 seconds

cecho – a function to print using different colors in bash

The following script can be used to print colored text in bash.
You can use it in any script without copy pasting everything in it by executing the following command source cecho.sh.
Doing so, it will load to your script the functions that are defined in cecho.sh, making them available for you to use (something like including code in C, with some caveats).

cecho.sh (compressed) (161 downloads)
#!/bin/bash

# The following function prints a text using custom color
# -c or --color define the color for the print. See the array colors for the available options.
# -n or --noline directs the system not to print a new line after the content.
# Last argument is the message to be printed.
cecho () {

    declare -A colors;
    colors=(\
        ['black']='\E[0;47m'\
        ['red']='\E[0;31m'\
        ['green']='\E[0;32m'\
        ['yellow']='\E[0;33m'\
        ['blue']='\E[0;34m'\
        ['magenta']='\E[0;35m'\
        ['cyan']='\E[0;36m'\
        ['white']='\E[0;37m'\
    );

    local defaultMSG="No message passed.";
    local defaultColor="black";
    local defaultNewLine=true;

    while [[ $# -gt 1 ]];
    do
    key="$1";

    case $key in
        -c|--color)
            color="$2";
            shift;
        ;;
        -n|--noline)
            newLine=false;
        ;;
        *)
            # unknown option
        ;;
    esac
    shift;
    done

    message=${1:-$defaultMSG};   # Defaults to default message.
    color=${color:-$defaultColor};   # Defaults to default color, if not specified.
    newLine=${newLine:-$defaultNewLine};

    echo -en "${colors[$color]}";
    echo -en "$message";
    if [ "$newLine" = true ] ; then
        echo;
    fi
    tput sgr0; #  Reset text attributes to normal without clearing screen.

    return;
}

warning () {

    cecho -c 'yellow' "[email protected]";
}

error () {

    cecho -c 'red' "[email protected]";
}

information () {

    cecho -c 'blue' "[email protected]";
}

 

Usage

Function cecho accepts the options to set the color and to control if a new line should be print.
Parameter -c or --color define the color for the print. See the array colors for the available options.
Parameter -n or --noline directs the system not to print a new line after the content.
The last parameter is the string message to be printed.
Functions warning, error and information are using cecho to print in color.
These three functions always print a new line and they have hardcoded one color set for each.

Example

#Get the name of the script currently being executed

scriptName=$(basename $(test -L "$0" && readlink "$0" || echo "$0"));

#Get the directory where the script currently being executed resides

scriptDirDIR=$(cd $(dirname "$0") && pwd);

#Print in blue color with no new line

cecho -n -c 'blue' "$scriptDir";

#Print in red color with a new line following the message

cecho -c 'red' "$scriptName";

#Using the information() function to print in blue followed by a new line

information ‘End of script’;