Bash


ffmpeg: Create a video countdown

The code below was used to generate the video countdown timers that are available in the following playlist using ffmpeg:

#This example will create a 3 second video, with 100 frames per second and it will print the elapsed and remaining times using a two second accuracy.
fps=100;
seconds=3;
mantissaDigits=2;
upperFont=600;
lowerFont=100;
ffmpeg -loop 1 -i ~/Pictures/Black-Background.png -c:v libx264 -r $fps -t $seconds -pix_fmt yuv420p -vf "fps=$fps,drawtext=fontfile='/usr/share/fonts/urw-base35/C059-Bold.otf':fontcolor=yellow:fontsize=$upperFont:x=(w-text_w)/2:y=(h-text_h)/2:text='%{eif\:($seconds-t)\:d}.%{eif\:(mod($seconds-t, 1)*pow(10,$mantissaDigits))\:d\:$mantissaDigits}',drawtext=fontfile='/usr/share/fonts/urw-base35/C059-Bold.otf':fontcolor=yellow:fontsize=$lowerFont:x=(w-text_w)/2:y=((h-text_h)/2)+$upperFont:text='Elapsed\: %{eif\:(t)\:d}.%{eif\:(mod(t, 1)*pow(10,$mantissaDigits))\:d\:$mantissaDigits}'" "$seconds seconds countdown timer.mp4";

Notes:

  • We used a single black frame for the background that defined the size of the video frame as well.
  • Using the fps variable we defined the number of Frames per Second for the video.
  • The seconds variable defined the number of seconds the duration of the video should be.
  • The mantissaDigits variable defines how many decimal digits should be shown after the dot.
  • upperFont and lowerFont define the size of the fonts in the upper row and the lower one respectively.
  • We used the drawtext directive twice to write to the frames.

Notes on the first drawtext:

  • fontfile='/usr/share/fonts/urw-base35/C059-Bold.otf' defines the font to be used for the text.
  • fontcolor=yellow defines the color of the font of the text.
  • fontsize=$upperFont defines the size of the font of the text.
  • x=(w-text_w)/2 defines the X-coordinate of the location for the text on the frame, here we center the text horizontally on the frame.
  • y=(h-text_h)/2 defines the Y-coordinate of the location for the text on the frame, here we center the text vertically on the frame.
  • text='%{eif\:($seconds-t)\:d}.%{eif\:(mod($seconds-t, 1)*pow(10,$mantissaDigits))\:d\:$mantissaDigits}' We print the remaining seconds for the video to finish with specific decimal digit accuracy.

Notes on the second drawtext:

  • drawtext=fontfile='/usr/share/fonts/urw-base35/C059-Bold.otf' defines the font to be used for the text.
  • fontcolor=yellow defines the color of the font of the text.
  • fontsize=$lowerFont defines the size of the font of the text.
  • x=(w-text_w)/2 defines the X-coordinate of the location for the text on the frame, here we center the text horizontally on the frame.
  • y=((h-text_h)/2)+$upperFont defines the Y-coordinate of the location for the text on the frame, here shift the text from the vertical center  of the frame.
  • text='Elapsed\: %{eif\:(t)\:d}.%{eif\:(mod(t, 1)*pow(10,$mantissaDigits))\:d\:$mantissaDigits}' We print the elapsed seconds since the video started with specific decimal digit accuracy.

Perform diff on two folders

To perform a recursive diff on all the files of two folders we just need to add the -r (or --recursive) parameter that recursively compares any subdirectories found.

To avoid needless messages from the tool, we can also use the -q (or --brief) parameter that reports only when files differ.

Example of performing diff on two folders recursively while preventing needless messages.

diff -rq aFolder someOtherFolder;


How to find lines that contain only lowercase characters

To print all lines that contain only lower case characters, we used the following regular expression in grep:

egrep '^[[:lower:]]+$' <file>;
#If you do not have egrep, use
grep -e '^[[:lower:]]+$' <file>;

Breakdown of the above regular expression:

  • ^ instructs the regular expression parser that the pattern should always start with the beginning of the line
  • [[:lower:]] this special instruction informs us that only lower case characters can match it
  • + the plus sign causes the preceding token to be matched one or more times
  • $ signifies the end of the line

Fedora 26: C++: static linking cannot find -lstdc++ -lm and -lc

Recently, we were trying to compile a C++ application with the following compilation command on a Fedora 26 64bit :

g++ -static -O2 -lm -Wall -Wno-unused-result -std=c++14 -DCS_ACADEMY -DONLINE_JUDGE 510152025.cpp -o 510152025;

unfortunately, we got the following errors:

 /usr/bin/ld: cannot find -lstdc++
 /usr/bin/ld: cannot find -lm
 /usr/bin/ld: cannot find -lc
 collect2: error: ld returned 1 exit status

To resolve the issues, we performed the following installations to install the static versions of the glibc and libstdc libraries:

sudo dnf install glibc-static libstdc++-static -y;

 


Bash: After redirected input file is done, allow user to control application via STDIN

Recently, we needed to start an application using a script, which application had its own CLI.
After starting it, we had to feed it with some input to configure it before handing it over to the end user.

The application we used was named dog. We wrote into a plain text file (named food) the commands that we needed to feed the application and then we started the execution using a plain input redirect like so dog < food;.
Doing so, resulted into properly feeding the dog application the food data  but it would cause the application to terminate immediately after the food was done as it would also feed the EOF (End Of File) directive to dog.
Apparently, the application after receiving the EOF, it would then terminate without allowing the end user to take control.

To mitigate the problem, we used the cat command to concatenate the input file along with the stdin stream, permitting us to first feed all the data in the food file and then allow the user to input data using the standard input stream.
Our final command resulted in the following solution as below

cat <(cat food) - | dog;

Where - is the special sign for standard input stdin.
cat food can be of course replaced with other commands that produce output on the standard output (stdout).

A bad side-effect of this solution, is that we lost some functionality of the terminal including, but not limited to, using the backspace and navigation.


Find all git repositories and perform a pull operation on them.

The following command will find all git projects in your home folder and perform a pull operation on them.

find ~ -name ".git" -type d -exec bash -c "echo '{}' && cd '{}'/.. && git pull" \;

The above command is based on finding the .git folders that exist in any clone of a git repository. Once a .git folder is found, it will navigate to its parent folder where it will perform the pull request.

Bonus – Automate the procedure using a cron job

The following entry in crontab allows us to periodically perform a pull request on all the cloned repositories that we have in a specific folder. Specifically, it will perform this operation once every five minutes.

*/5    *    *    *    *    cd /home/bytefreaks/Projects; find . -name ".git" -type d -exec bash -c "echo '{}' && cd '{}'/.. && git pull" \; &> /tmp/bf.git.log

Please note that it would be easier to use an ssh key that does not have a password for this automation.
If you do not, the you will need to either pass the password via this configuration line (not recommended) or have a key agent running to provide the password for the key.

Redirecting standard error (stderr)

The following command will redirect stderr to a different file than the one stdout is redirected to:

command >log.txt 2>errors.txt;

In case you want to redirect stderr to stdout (&1), and then redirect stdout to a file you can use the following setup:

command >mixed-log.txt 2>&1;

The following command will have the same effect as the previous one, the difference between them is the way they are implemented. This time we will redirect both the stdout and stderr to a file:

command &> mixed-log.txt;


Bash: Remove the last character from each line

The following script, uses rev and cut to remove the last character from each line in a pipe.
rev utility reverses lines character-wise.
cut removes sections  from each of line.
It is a very simple script where we reverse the line once, remove the first character (which was the last one in the original form of the line) and finally we reverse the line back with the last character missing.

echo -e "hi\nHI" | rev | cut -c 2- | rev;

# Will produce:
h
H

 


Bash: Print time stamp in front of every line in a pipe

Recently, we received a binary that collected data from a web service and it printed them on screen.
The binary did not print a time stamp in front of each line so we had to improvise of a way to add the time stamp to the logs without modifying the binary.

The solution we came to was to use awk to prepend the time stamp in front of every line using a pipe.
Specifically, our solution was the following:

server_application 2>&1 | awk '{ print strftime("%Y-%m-%d %H:%M:%S"), $0; fflush(); }'

What we did there was to start our binary server_application, redirect stderr to stdout (using 2>&1) so that we will have only one stream and then we read the lines one by one using awk and printed the time stamp right before the line ($0) using strftime.
The strftime() function formats the broken-down time according to the format specification format.
fflushforces a write of all user-space buffered data for the given output or update stream via the stream’s underlying write function. We call it at each line to make sure that we do not cause additional delay in presenting the data due to buffering limitations caused by our prints.

Example

$ echo -e "hi\nHI" 2>&1 | awk '{ print strftime("%Y-%m-%d %H:%M:%S"), $0; fflush(); }'
2017-06-21 20:33:41 hi
2017-06-21 20:33:41 HI


How to execute `find` that ignores .git directories

Trying to find a source code file by its content using find and -exec grep, can some times result in getting results from the repository .git folders as well.

This behavior not only does it provide results you do not need but it also makes your search slower.
Below, we propose a couple of solutions on how to make a more efficient search.

Example 1: Ignore all .git folders no matter where they are in the search path

For find to ignore all .git folders, even if they appear on the first level of directories or any in-between until the last one, add -not -path '*/\.git*' to your command as in the example below.
This parameter will instruct find to filter out any file that has anywhere in its path the folder .git. This is very helpful in case a project has dependencies in other projects (repositories) that are part of the internal structure.

find . -type f -not -path '*/\.git/*';

Note, if you are using svn use:

find . -type f -not -path '*/\.svn/*';

Example 2: Ignore all hidden files and folders

To ignore all hidden files and folders from your find results add -not -path '*/\.*' to your command.

find . -not -path '*/\.*';

This parameter instructs find to ignore any file that has anywhere in its path the string /. which is any hidden file or folder in the search path!