Bash


Fedora 26: C++: static linking cannot find -lstdc++ -lm and -lc

Recently, we were trying to compile a C++ application with the following compilation command on a Fedora 26 64bit :

g++ -static -O2 -lm -Wall -Wno-unused-result -std=c++14 -DCS_ACADEMY -DONLINE_JUDGE 510152025.cpp -o 510152025;

unfortunately, we got the following errors:

 /usr/bin/ld: cannot find -lstdc++
 /usr/bin/ld: cannot find -lm
 /usr/bin/ld: cannot find -lc
 collect2: error: ld returned 1 exit status

To resolve the issues, we performed the following installations to install the static versions of the glibc and libstdc libraries:

sudo dnf install glibc-static libstdc++-static -y;

 


Bash: After redirected input file is done, allow user to control application via STDIN

Recently, we needed to start an application using a script, which application had its own CLI.
After starting it, we had to feed it with some input to configure it before handing it over to the end user.

The application we used was named dog. We wrote into a plain text file (named food) the commands that we needed to feed the application and then we started the execution using a plain input redirect like so dog < food;.
Doing so, resulted into properly feeding the dog application the food data  but it would cause the application to terminate immediately after the food was done as it would also feed the EOF (End Of File) directive to dog.
Apparently, the application after receiving the EOF, it would then terminate without allowing the end user to take control.

To mitigate the problem, we used the cat command to concatenate the input file along with the stdin stream, permitting us to first feed all the data in the food file and then allow the user to input data using the standard input stream.
Our final command resulted in the following solution as below

cat <(cat food) - | dog;

Where - is the special sign for standard input stdin.
cat food can be of course replaced with other commands that produce output on the standard output (stdout).

A bad side-effect of this solution, is that we lost some functionality of the terminal including, but not limited to, using the backspace and navigation.


Find all git repositories and perform a pull operation on them.

The following command will find all git projects in your home folder and perform a pull operation on them.

find ~ -name ".git" -type d -exec bash -c "echo '{}' && cd '{}'/.. && git pull" \;

The above command is based on finding the .git folders that exist in any clone of a git repository. Once a .git folder is found, it will navigate to its parent folder where it will perform the pull request.

Bonus – Automate the procedure using a cron job

The following entry in crontab allows us to periodically perform a pull request on all the cloned repositories that we have in a specific folder. Specifically, it will perform this operation once every five minutes.

*/5    *    *    *    *    cd /home/bytefreaks/Projects; find . -name ".git" -type d -exec bash -c "echo '{}' && cd '{}'/.. && git pull" \; &> /tmp/bf.git.log

Please note that it would be easier to use an ssh key that does not have a password for this automation.
If you do not, the you will need to either pass the password via this configuration line (not recommended) or have a key agent running to provide the password for the key.

Redirecting standard error (stderr)

The following command will redirect stderr to a different file than the one stdout is redirected to:

command >log.txt 2>errors.txt;

In case you want to redirect stderr to stdout (&1), and then redirect stdout to a file you can use the following setup:

command >mixed-log.txt 2>&1;

The following command will have the same effect as the previous one, the difference between them is the way they are implemented. This time we will redirect both the stdout and stderr to a file:

command &> mixed-log.txt;


Bash: Remove the last character from each line

The following script, uses rev and cut to remove the last character from each line in a pipe.
rev utility reverses lines character-wise.
cut removes sections  from each of line.
It is a very simple script where we reverse the line once, remove the first character (which was the last one in the original form of the line) and finally we reverse the line back with the last character missing.

echo -e "hi\nHI" | rev | cut -c 2- | rev;

# Will produce:
h
H

 


Bash: Print time stamp in front of every line in a pipe

Recently, we received a binary that collected data from a web service and it printed them on screen.
The binary did not print a time stamp in front of each line so we had to improvise of a way to add the time stamp to the logs without modifying the binary.

The solution we came to was to use awk to prepend the time stamp in front of every line using a pipe.
Specifically, our solution was the following:

server_application 2>&1 | awk '{ print strftime("%Y-%m-%d %H:%M:%S"), $0; fflush(); }'

What we did there was to start our binary server_application, redirect stderr to stdout (using 2>&1) so that we will have only one stream and then we read the lines one by one using awk and printed the time stamp right before the line ($0) using strftime.
The strftime() function formats the broken-down time according to the format specification format.
fflushforces a write of all user-space buffered data for the given output or update stream via the stream’s underlying write function. We call it at each line to make sure that we do not cause additional delay in presenting the data due to buffering limitations caused by our prints.

Example

$ echo -e "hi\nHI" 2>&1 | awk '{ print strftime("%Y-%m-%d %H:%M:%S"), $0; fflush(); }'
2017-06-21 20:33:41 hi
2017-06-21 20:33:41 HI


How to execute `find` that ignores .git directories

Trying to find a source code file by its content using find and -exec grep, can some times result in getting results from the repository .git folders as well.

This behavior not only does it provide results you do not need but it also makes your search slower.
Below, we propose a couple of solutions on how to make a more efficient search.

Example 1: Ignore all .git folders no matter where they are in the search path

For find to ignore all .git folders, even if they appear on the first level of directories or any in-between until the last one, add -not -path '*/\.git*' to your command as in the example below.
This parameter will instruct find to filter out any file that has anywhere in its path the folder .git. This is very helpful in case a project has dependencies in other projects (repositories) that are part of the internal structure.

find . -type f -not -path '*/\.git/*';

Note, if you are using svn use:

find . -type f -not -path '*/\.svn/*';

Example 2: Ignore all hidden files and folders

To ignore all hidden files and folders from your find results add -not -path '*/\.*' to your command.

find . -not -path '*/\.*';

This parameter instructs find to ignore any file that has anywhere in its path the string /. which is any hidden file or folder in the search path!


How to get the pid of the last executed command that was sent to the background in a bash shell

Recently we came to the need of writing a bash script that needed periodically to check if a specific process, that was started by the script, had ended and restart it (something like watchdog but with not so many features).

To achieve this, we used the one of the shell special parameters, the $!. Like all other special parameters $! may only be referenced and the user cannot make an assignment to it.

($!) Expands to the process ID of the job most recently placed into the background, whether executed as an asynchronous command or using the bg builtin command.

From GNU.org: https://www.gnu.org/software/bash/manual/html_node/Special-Parameters.html#index-_0021-1

Example of Usage

In this example we wanted to get the PID of the application called server to be used later on in the script.

server &
echo $!; #This will print the process ID of the 'server' application


File permissions change date

Recently we wanted to check when did the permissions of a specific file changed.
Unfortunately, there exists no such flag and we do not have a 100% working solution for it.

What we did was to check the last modification time of the file status information (ctime) using the ls -lc command.
This command could indicate the last permissions change time but it is not a reliable source as it represents the modification time of other elements as well.

The modification time of the file status information (ctime) gets updated when any inode information regarding the file changes.
This means that the modification time of the file status information (ctime) will get updated when any of the following changes:

  • owner – The numeric user ID (UID) of the file’s owner.
  • group – The numeric group ID (GID) of the file’s group.
  • link count – The number of links to the file.
  • mode – The bit string that indicated the permissions and privileges
  • serial – The serial number of the file.
  • device – The numeric ID of the device containing the file.

Explanation of ls parameters

  • The parameter -c of the ls command when used with the -l will show ctime and sort by name.
  • The parameter -c of the ls command when used with the -l and the -t will show ctime and sort by ctime (newest first).

Example that demonstrates that we get different values in the time column of -l when -c is used

$ ls -lc ~/.ssh/
total 28
-rwx------. 1 george george  225 May 16 17:05 config
-rwx------. 1 george george 1743 Jun  2 13:36 id_rsa
-rwxrwx---. 1 george george  405 May 16 17:05 id_rsa.pub
-rwxrwx---. 1 george george   32 May 16 17:05 Details.txt
-rw-r--r--. 1 george george 9155 May 30 14:32 known_hosts

$ ls -l ~/.ssh/
total 28
-rwx------. 1 george george  225 Mar 22 11:36 config
-rwx------. 1 george george 1743 Jan 25 10:22 id_rsa
-rwxrwx---. 1 george george  405 Jan 25 10:22 id_rsa.pub
-rwxrwx---. 1 george george   32 Jan 25 10:22 Details.txt
-rw-r--r--. 1 george george 9155 May 30 14:32 known_hosts

Create a .tar file with different compression methods

The following commands will create .tar archives  and compress them using the different methods that are available. We provide multiple solutions, each one for a different type of .tar archive depending on the compression method that is desired.

For .tar archives

tar -c -f archive.tar $FILES_TO_ARCHIVE;

For .tar.bz2 archives

tar -c -j -f archive.tar.bz2 $FILES_TO_ARCHIVE;

For .tar.xz archives

tar -c -J -f archive.tar.xz $FILES_TO_ARCHIVE;

For .tar.gz and .tgz archives

tar -c -z -f archive.tar.gz $FILES_TO_ARCHIVE;

tar Parameters Legend

  • -z or --gzip instructs tar to filter the archive through gzip
  • -j or --bzip2 filters the archive through bzip2
  • -J or --xz filters the archive through xz
  • -f or --file=OUTPUT uses the archive file OUTPUT
  • -c or --create a new archive

Bonus Example: Create a tar.xz archive using the current date in the archive name

The following command will create an archive out of the folders Folder1 and Folder2 and then it will compress it to the .tar.xz format.
The filename of the archive will contain the current date in the format YYYY-MM-DD.

tar -c -J  -f archive.`date +%F`.tar.xz Folder1 Folder2;

The above command will result in something similar to:

archive.2017-06-04.tar.xz