Bash

Find all git repositories and perform a pull operation on them.

 in Bash  tagged cron / find all / git by

The following command will find all git projects in your home folder and perform a pull operation on them.

find ~ -name ".git" -type d -exec bash -c "echo '{}' && cd '{}'/.. && git pull" \;

The above command is based on finding the .git folders that exist in any clone of a git repository. Once a .git folder is found, it will navigate to its parent folder where it will perform the pull request.

Bonus – Automate the procedure using a cron job

The following entry in crontab allows us to periodically perform a pull request on all the cloned repositories that we have in a specific folder. Specifically, it will perform this operation once every five minutes.

*/5    *    *    *    *    cd /home/bytefreaks/Projects; find . -name ".git" -type d -exec bash -c "echo '{}' && cd '{}'/.. && git pull" \; &> /tmp/bf.git.log

Please note that it would be easier to use an ssh key that does not have a password for this automation.
If you do not, the you will need to either pass the password via this configuration line (not recommended) or have a key agent running to provide the password for the key.

Redirecting standard error (stderr)

 in Bash  tagged redirect / stderr / stdout by

The following command will redirect stderr to a different file than the one stdout is redirected to:

command >log.txt 2>errors.txt;

In case you want to redirect stderr to stdout (&1), and then redirect stdout to a file you can use the following setup:

command >mixed-log.txt 2>&1;

The following command will have the same effect as the previous one, the difference between them is the way they are implemented. This time we will redirect both the stdout and stderr to a file:

command &> mixed-log.txt;

Bash: Remove the last character from each line 1

 in Bash  tagged cut / last character / remove / rev by

The following script, uses rev and cut to remove the last character from each line in a pipe.
rev utility reverses lines character-wise.
cut removes sections  from each of line.
It is a very simple script where we reverse the line once, remove the first character (which was the last one in the original form of the line) and finally we reverse the line back with the last character missing.


echo -e "hi\nHI" | rev | cut -c 2- | rev;

# Will produce:
h
H

 

Bash: Print time stamp in front of every line in a pipe

 in Bash  tagged awk / fflush / stamp / strftime / time / timestamp by

Recently, we received a binary that collected data from a web service and it printed them on screen.
The binary did not print a time stamp in front of each line so we had to improvise of a way to add the time stamp to the logs without modifying the binary.

The solution we came to was to use awk to prepend the time stamp in front of every line using a pipe.
Specifically, our solution was the following:


server_application 2>&1 | awk '{ print strftime("%Y-%m-%d %H:%M:%S"), $0; fflush(); }'

What we did there was to start our binary server_application, redirect stderr to stdout (using 2>&1) so that we will have only one stream and then we read the lines one by one using awk and printed the time stamp right before the line ($0) using strftime.
The strftime() function formats the broken-down time according to the format specification format.
fflushforces a write of all user-space buffered data for the given output or update stream via the stream’s underlying write function. We call it at each line to make sure that we do not cause additional delay in presenting the data due to buffering limitations caused by our prints.

Example


$ echo -e "hi\nHI" 2>&1 | awk '{ print strftime("%Y-%m-%d %H:%M:%S"), $0; fflush(); }'
2017-06-21 20:33:41 hi
2017-06-21 20:33:41 HI