time


Bash: Print time stamp in front of every line in a pipe

Recently, we received a binary that collected data from a web service and it printed them on screen.
The binary did not print a time stamp in front of each line so we had to improvise of a way to add the time stamp to the logs without modifying the binary.

The solution we came to was to use awk to prepend the time stamp in front of every line using a pipe.
Specifically, our solution was the following:

server_application 2>&1 | awk '{ print strftime("%Y-%m-%d %H:%M:%S"), $0; fflush(); }'

What we did there was to start our binary server_application, redirect stderr to stdout (using 2>&1) so that we will have only one stream and then we read the lines one by one using awk and printed the time stamp right before the line ($0) using strftime.
The strftime() function formats the broken-down time according to the format specification format.
fflushforces a write of all user-space buffered data for the given output or update stream via the stream’s underlying write function. We call it at each line to make sure that we do not cause additional delay in presenting the data due to buffering limitations caused by our prints.

Example

$ echo -e "hi\nHI" 2>&1 | awk '{ print strftime("%Y-%m-%d %H:%M:%S"), $0; fflush(); }'
2017-06-21 20:33:41 hi
2017-06-21 20:33:41 HI

Advertisements

C: Code to time execution with accuracy greater than a second

The following application computes the time needed for a process to finish using the method clock().
The result of the application is the time in seconds as a floating number (where 1.0 = 1 second).
It provides greater accuracy than seconds as the estimation is done using processor time used by the program.

#include <stdio.h>
#include <time.h>
#include <stdlib.h>
#include <limits.h>

int main()
{

    /* clock_t clock(void)
     The clock() function returns an approximation of processor time used by the program.
     The value returned is the CPU time used so far as a clock_t,
     to get the number of seconds used, divide by CLOCKS_PER_SEC.
     On error it returns -1. */
    const clock_t start = clock();

    /* svoid srand(unsigned int __seed)
     The srand() function sets its argument as the seed for a new sequence of pseudo-random
     integers to be returned by rand(). These sequences are repeatable by calling srand() with the
     same seed value.
     If no seed value is provided, the rand() function is automatically seeded with a value of 1. */
    /* time_t time(time_t *__timer)
     time() returns the time since the Epoch (00:00:00 UTC, January 1, 1970), measured in seconds.
     If the __timer variable is not NULL, the return value is also stored there. */
    srand(time(NULL));
    unsigned long i;
    for (i = 0; i < 10000000; i++)
    {
        /* int rand(void)
         The rand() function returns a pseudo-random integer in the range 0 to RAND_MAX inclusive. */
        rand();
    }
    const clock_t end = clock();

    /* ISO/IEC 9899:1999 7.23.1: Components of time
    The macro `CLOCKS_PER_SEC' is an expression with type `clock_t' that is
    the number per second of the value returned by the `clock' function. */
    /* CAE XSH, Issue 4, Version 2: <time.h>
    The value of CLOCKS_PER_SEC is required to be 1 million on all
    XSI-conformant systems. */
    const float seconds = (float) (end - start) / CLOCKS_PER_SEC;

    printf("Seconds elapsed %f\n", seconds);
    return 0;
}

Find files that were created, modified or accessed in the last N minutes

Find all files in $my_folder that their status changed in the last 60 minutes

find $my_folder -cmin -60

Find all files in $my_folder that their data were modified in the last 60 minutes

find $my_folder -mmin -60

Find all files in $my_folder that they were accessed in the last 60 minutes

find $my_folder -amin -60

Please remember to use negative values for the minutes. e.g. use -60 and not 60.

More examples

Find all files in $my_folder that their status changed in the last 60 minutes AND they were accessed in the last 10 minutes

find $my_folder -cmin -60 -amin -10

Find all files in $my_folder that their status changed in the last 60 minutes OR they were accessed in the last 10 minutes

find $my_folder \( -cmin -60 -o -amin -10 \)

Notes on find command

  • -cmin n Matches files which their status was last changed n minutes ago.
  • -mmin n Matches files which which data was last modified n minutes ago.
  • -amin n Matches files which they were last accessed n minutes ago.
  • -o is the logical Or operator. The second expression  is not evaluated if the first expression is true.

Get execution time in seconds

The following methods demonstrate different methods on how to compute the time a potion of code or script take to complete their execution.

Time Methods - Full Examples (186 downloads)

 

Method 1 – Using date

The following example will calculate the execution time in seconds by subtracting the system date and time in seconds since 1970-01-01 00:00:00 UTC once right before the script goes to the computation part and once right after.

In order to get the system date and time in seconds since 1970-01-01 00:00:00 UTC we use the command date +%s.

Time Methods - Full Examples (186 downloads)
#!/bin/bash

#Print the system date and time in seconds since 1970-01-01 00:00:00 UTC
startTime=$(date +%s);

#We pick a random number between 1 and 10.
#Then we delay the execution for that amount of seconds.
sleep $(( (RANDOM % 10) + 1 ));

endTime=$(date +%s);

#Subtract endTime from startTime to get the total execution time
totalTime=$(($endTime-$startTime));

echo "Process finished after $totalTime seconds";

exit 0;

Method 2 – Using bash internal SECONDS variable

The following example will calculate the execution time in seconds by reseting the bash internal variable SECONDS to 0, forcing the shell to continue counting from there.

Time Methods - Full Examples (186 downloads)
#!/bin/bash

#This variable expands to the number of seconds since the shell was started.
#We set it to 0, forcing the shell to continue counting from there.
SECONDS=0;

#We pick a random number between 1 and 10.
#Then we delay the execution for that amount of seconds.
sleep $(( (RANDOM % 10) + 1 ));

echo "Process finished after $SECONDS seconds";

exit 0;

Method 3 – Using bash time

The following example uses the bash time command, which reports the time consumed by a pipeline’s execution.
When time command is executed without its complete path, then the bash built-in time command is executed, instead of the GNU time command. We will use the bash time command in this example and we will use it to run a whole block of commands.
Please note that time command will return the time in seconds as a float (i.e. there will be decimal places. e.g. 1 will be printed as 1.00).

Time Methods - Full Examples (186 downloads)
#!/bin/bash

#The bash time command reports the time consumed by pipeline's execution
#When time command is executed without its complete path, then the bash built-in time command is executed, instead of the GNU time command.
#We will use the bash time command in this example and we will use it to run a whole block of commands.

#We change the output format of time to print elapsed real time in seconds.
TIMEFORMAT="%E";
#We pick a random number between 1 and 10.
#Then we delay the execution for that amount of seconds.
totalTime=`time ( sleep $(( (RANDOM % 10) + 1 )) ) 2>&1`;

#Please note that time command will return the time in seconds as a float (i.e. there will be decimal places. e.g. 1 will be printed as 1.00).
#This will happen as time has build-in more precision than the first two methods presented here.
echo "Process finished after $totalTime seconds";

totalTimeBlock=`time (
	sleep $(( (RANDOM % 10) + 1 ));
	sleep $(( (RANDOM % 10) + 1 ));
) 2>&1`;
echo "Block finished after $totalTimeBlock seconds";

exit 0;

Method 4 – Using GNU time

The GNU time command runs the specified program command with the given arguments.
When time command is executed without its complete path (in our case it was /usr/bin/time), then the bash built-in time command is executed, instead of the GNU time command. To make sure we use the GNU time command, we use which to get the full path of the time command.
Please note that time command will return the time in seconds as a float (i.e. there will be decimal places. e.g. 1 will be printed as 1.00).

Time Methods - Full Examples (186 downloads)
#!/bin/bash
#The time command runs the specified program command with the given arguments.
#When time command is executed without its complete path (in our case it was /usr/bin/time), then the bash built-in time command is executed, instead of the GNU time command.
#To make sure we use the GNU time command, we use which to get the full path of the time command.
time=`which time`;

#We pick a random number between 1 and 10.
#Then we delay the execution for that amount of seconds.
#We change the output format of time to print elapsed real time in seconds.
totalTime="$( $time -f '%e' sleep $(( (RANDOM % 10) + 1 )) 2>&1 1>/dev/null )";

#Please note that time command will return the time in seconds as a float (i.e. there will be decimal places. e.g. 1 will be printed as 1.00).
#This will happen as time has build-in more precision than the first two methods presented here.
echo "Process finished after $totalTime seconds";

exit 0;

Notes

RANDOM internal variable

Each time RANDOM internal variable is referenced, a random integer between 0 and 32767 is generated.

By using the RANDOM variable in this command $(( (RANDOM % 10) + 1 )); we perform a modulo on the random value with the static value 10. This way we force the range of valid values to be between 0 and 9.
Later, we add 1 to that value to shift the range to be between 1 and 10.