Bash


How we create bootable GNU/Linux USB flash drives from terminal

A very important tool in our everyday life are the LiveUSB GNU/Linux flash drives.
We keep an updated collection of several GNU/Linux flavors/distributions (Fedora, CentOS, (L/X)Ubuntu, Kali etc.) that are used depending on the scenario.

The command we use is the following:

sudo dd bs=4M if=path/to/OS.iso of=/dev/sdX conv=fdatasync;

dd allows you to convert and copy a file and we use it to copy the ISO file of the operating system onto the USB flash drive.

Notes:

  1. You need to unmount the USB flash drive before formatting it, e.g.:
    sudo umount /dev/sdXY;
  2. You need to use the device filename and not a partition filename:
    e.g. You need to use /dev/sdX and NOT /dev/sdX1
  3. You need to use either the root account or execute the command with sudo
  4. If you do not know the filename associated with your flash drive, use an application like the following ones to determine which /dev file is mapped to the USB flash drive:
    gnome-disks; or
    lsblk; or
    sudo fdisk -l;

The parameters we use are the following:

  • bs=SIZE_IN_BYTES defines up to how many bytes should be read and written at a time.
    In our case we used 4 Megabytes (4M).
  • if=INPUT_FILE defines the file to be read, we use this parameter to point to the OS ISO file that we want to write on the USB drive.
  • of=OUTPUT_FILE defines the filename where the data is to be written in.
    In GNU/Linux, devices are accessible like files as well so we used /dev/sdX here that happened to be the device file assigned to our USB device.
  • conv=CONVS converts the file as per the comma separated symbol list
    fdatasync physically writes output file data before finishing, we use this parameter to be sure that all I/O operations are done well before dd terminates, this way we are certain that our USB device will be ready to use as soon as the application is done.

ffmpeg: Create a video countdown

The code below was used to generate the video countdown timers that are available in the following playlist using ffmpeg:

#This example will create a 3 second video, with 100 frames per second and it will print the elapsed and remaining times using a two second accuracy.
fps=100;
seconds=3;
mantissaDigits=2;
upperFont=600;
lowerFont=100;
ffmpeg -loop 1 -i ~/Pictures/Black-Background.png -c:v libx264 -r $fps -t $seconds -pix_fmt yuv420p -vf "fps=$fps,drawtext=fontfile='/usr/share/fonts/urw-base35/C059-Bold.otf':fontcolor=yellow:fontsize=$upperFont:x=(w-text_w)/2:y=(h-text_h)/2:text='%{eif\:($seconds-t)\:d}.%{eif\:(mod($seconds-t, 1)*pow(10,$mantissaDigits))\:d\:$mantissaDigits}',drawtext=fontfile='/usr/share/fonts/urw-base35/C059-Bold.otf':fontcolor=yellow:fontsize=$lowerFont:x=(w-text_w)/2:y=((h-text_h)/2)+$upperFont:text='Elapsed\: %{eif\:(t)\:d}.%{eif\:(mod(t, 1)*pow(10,$mantissaDigits))\:d\:$mantissaDigits}'" "$seconds seconds countdown timer.mp4";

Notes:

  • We used a single black frame for the background that defined the size of the video frame as well.
  • Using the fps variable we defined the number of Frames per Second for the video.
  • The seconds variable defined the number of seconds the duration of the video should be.
  • The mantissaDigits variable defines how many decimal digits should be shown after the dot.
  • upperFont and lowerFont define the size of the fonts in the upper row and the lower one respectively.
  • We used the drawtext directive twice to write to the frames.

Notes on the first drawtext:

  • fontfile='/usr/share/fonts/urw-base35/C059-Bold.otf' defines the font to be used for the text.
  • fontcolor=yellow defines the color of the font of the text.
  • fontsize=$upperFont defines the size of the font of the text.
  • x=(w-text_w)/2 defines the X-coordinate of the location for the text on the frame, here we center the text horizontally on the frame.
  • y=(h-text_h)/2 defines the Y-coordinate of the location for the text on the frame, here we center the text vertically on the frame.
  • text='%{eif\:($seconds-t)\:d}.%{eif\:(mod($seconds-t, 1)*pow(10,$mantissaDigits))\:d\:$mantissaDigits}' We print the remaining seconds for the video to finish with specific decimal digit accuracy.

Notes on the second drawtext:

  • drawtext=fontfile='/usr/share/fonts/urw-base35/C059-Bold.otf' defines the font to be used for the text.
  • fontcolor=yellow defines the color of the font of the text.
  • fontsize=$lowerFont defines the size of the font of the text.
  • x=(w-text_w)/2 defines the X-coordinate of the location for the text on the frame, here we center the text horizontally on the frame.
  • y=((h-text_h)/2)+$upperFont defines the Y-coordinate of the location for the text on the frame, here shift the text from the vertical center  of the frame.
  • text='Elapsed\: %{eif\:(t)\:d}.%{eif\:(mod(t, 1)*pow(10,$mantissaDigits))\:d\:$mantissaDigits}' We print the elapsed seconds since the video started with specific decimal digit accuracy.

Perform diff on two folders

To perform a recursive diff on all the files of two folders we just need to add the -r (or --recursive) parameter that recursively compares any subdirectories found.

To avoid needless messages from the tool, we can also use the -q (or --brief) parameter that reports only when files differ.

Example of performing diff on two folders recursively while preventing needless messages.

diff -rq aFolder someOtherFolder;


How to find lines that contain only lowercase characters

To print all lines that contain only lower case characters, we used the following regular expression in grep:

egrep '^[[:lower:]]+$' <file>;
#If you do not have egrep, use
grep -e '^[[:lower:]]+$' <file>;

Breakdown of the above regular expression:

  • ^ instructs the regular expression parser that the pattern should always start with the beginning of the line
  • [[:lower:]] this special instruction informs us that only lower case characters can match it
  • + the plus sign causes the preceding token to be matched one or more times
  • $ signifies the end of the line

Fedora 26: C++: static linking cannot find -lstdc++ -lm and -lc

Recently, we were trying to compile a C++ application with the following compilation command on a Fedora 26 64bit :

g++ -static -O2 -lm -Wall -Wno-unused-result -std=c++14 -DCS_ACADEMY -DONLINE_JUDGE 510152025.cpp -o 510152025;

unfortunately, we got the following errors:

 /usr/bin/ld: cannot find -lstdc++
 /usr/bin/ld: cannot find -lm
 /usr/bin/ld: cannot find -lc
 collect2: error: ld returned 1 exit status

To resolve the issues, we performed the following installations to install the static versions of the glibc and libstdc libraries:

sudo dnf install glibc-static libstdc++-static -y;

 


Bash: After redirected input file is done, allow user to control application via STDIN

Recently, we needed to start an application using a script, which application had its own CLI.
After starting it, we had to feed it with some input to configure it before handing it over to the end user.

The application we used was named dog. We wrote into a plain text file (named food) the commands that we needed to feed the application and then we started the execution using a plain input redirect like so dog < food;.
Doing so, resulted into properly feeding the dog application the food data  but it would cause the application to terminate immediately after the food was done as it would also feed the EOF (End Of File) directive to dog.
Apparently, the application after receiving the EOF, it would then terminate without allowing the end user to take control.

To mitigate the problem, we used the cat command to concatenate the input file along with the stdin stream, permitting us to first feed all the data in the food file and then allow the user to input data using the standard input stream.
Our final command resulted in the following solution as below

cat <(cat food) - | dog;

Where - is the special sign for standard input stdin.
cat food can be of course replaced with other commands that produce output on the standard output (stdout).

A bad side-effect of this solution, is that we lost some functionality of the terminal including, but not limited to, using the backspace and navigation.


Find all git repositories and perform a pull operation on them.

The following command will find all git projects in your home folder and perform a pull operation on them.

find ~ -name ".git" -type d -exec bash -c "echo '{}' && cd '{}'/.. && git pull" \;

The above command is based on finding the .git folders that exist in any clone of a git repository. Once a .git folder is found, it will navigate to its parent folder where it will perform the pull request.

Bonus – Automate the procedure using a cron job

The following entry in crontab allows us to periodically perform a pull request on all the cloned repositories that we have in a specific folder. Specifically, it will perform this operation once every five minutes.

*/5    *    *    *    *    cd /home/bytefreaks/Projects; find . -name ".git" -type d -exec bash -c "echo '{}' && cd '{}'/.. && git pull" \; &> /tmp/bf.git.log

Please note that it would be easier to use an ssh key that does not have a password for this automation.
If you do not, the you will need to either pass the password via this configuration line (not recommended) or have a key agent running to provide the password for the key.

Redirecting standard error (stderr)

The following command will redirect stderr to a different file than the one stdout is redirected to:

command >log.txt 2>errors.txt;

In case you want to redirect stderr to stdout (&1), and then redirect stdout to a file you can use the following setup:

command >mixed-log.txt 2>&1;

The following command will have the same effect as the previous one, the difference between them is the way they are implemented. This time we will redirect both the stdout and stderr to a file:

command &> mixed-log.txt;


Bash: Remove the last character from each line

The following script, uses rev and cut to remove the last character from each line in a pipe.
rev utility reverses lines character-wise.
cut removes sections  from each of line.
It is a very simple script where we reverse the line once, remove the first character (which was the last one in the original form of the line) and finally we reverse the line back with the last character missing.

echo -e "hi\nHI" | rev | cut -c 2- | rev;

# Will produce:
h
H