Ετήσια αρχεία: 2014


Use awk to print the last N columns of a file or a pipe

In this post we will describe a way to print the last N number of columns in awk.

We will use this code as example, where we will print the last 2 columns only:

awk '{n = 2; for (--n; n >= 0; n--){ printf "%s\t",$(NF-n)} print ""}';
'

In the awk script we use the variable n to control how many columns we want to print. In the above example we initialized it  to the value 2 as that is the number of columns we want printed.

After, we use a for loop to iterate over the fields (in this case the last two fields) and we print them to the screen using printf "%s\t",$(NF-n) to avoid printing the new line character and to separate them with a tab character.

NF is a special variable in awk that holds the total number of fields available on that line. If you do not change the delimiter, then it will hold the number of words on the line.

$(NF-n) is the way we ask awk to gives us the variable value that is n places before the last.

Outside the loop we print "" to print the new line character between input rows.

Examples:

If we want to print the last two columns of the ls -l command we can do it as follows:

ls -l | awk '{i = 2; for (--i; i >= 0; i--){ printf "%s\t",$(NF-i)} print ""}';

If we want to print the last two columns of the /etc/passwd file we can do it as follows:

awk -F ':' '{i = 2; for (--i; i >= 0; i--){ printf "%s\t",$(NF-i)} print ""}' /etc/passwd;

Note that we change the delimiter with the command line argument -F ":"

Advertisements

Pull all Git repositories you have access to

ssh [email protected] info | cut -f 2 | tail -n +3 | xargs -I {} -n 1 -I_repository -- sh -c 'cd _repository; git pull; cd ..;'

The above command will connect to the git server (git.bytefreaks.net) using  gitolite and get a list of all the repositories you have access to using ssh [email protected] info

The command should return a list similar to this:

hello bytefreaks, this is [email protected] running gitolite3 v3.5.3.1-1-gf8776f5 on git 1.7.1

 R W	Repo1
 R W	Repo2
 R W	Repo3
 R  	Repo4

From the results, we remove the first 3 lines as they contain no useful information to cloning all the repositories. From the rest of the lines, where each line contains the information for a repository we have access to, we keep the third column only as it is the one that holds the repository name as it is stored on the server.

Afterwards it will remove all columns except the second to filter the column with the repository names and will remove the first 3 lines to keep only the data we are interested in.

On the last stage of the pipe we have a list of the names of the repositories, using xargs, we assign each repository name to the _repository variable and using one result at a time, we navigate into the folder of the repository using cd and call the pull command.

Note: We assume that all repositories are in the current folder as children and each one is in a sub-folder of its own which is named as the repository is.


Clone all repositories you have access to over ssh

ssh [email protected] info | cut -f 2 | tail -n +3 | xargs -I {} -n 1 git clone ssh://[email protected]/{}

The above command will connect to the git server (git.bytefreaks.net) that is using  gitolite and get a list of all the repositories you have access to using ssh [email protected] info

The command should return a list similar to this:

hello bytefreaks, this is [email protected] running gitolite3 v3.5.3.1-1-gf8776f5 on git 1.7.1

 R W	Repo1
 R W	Repo2
 R W	Repo3
 R  	Repo4

From the results, we remove the first 3 lines as they contain no useful information to cloning all the repositories. From the rest of the lines, where each line contains the information for a repository we have access to, we keep the third column only as it is the one that holds the repository name as it is stored on the server.

On the last stage of the pipe we have a list of the names of the repositories, using xargs, we assign each repository name to the special variable {} and processing one result at a time we clone the git repository to the current directory under the folder that is named as the repository.


ffmpeg: Extract audio from .WEBM to .OGG 3

If you need to extract the audio from an .WEBM movie file to an .OGG audio file you can  execute the following:

FILE="the-file-you-want-to-process.webm";
ffmpeg -i "${FILE}" -vn -y "${FILE%.webm}.ogg"

The first command will assign the file name to a variable, we do this to avoid typing errors in the second command where we might want to use the same name for the audio file.

The second command, will use ffmpeg to extract the audio. The -i flag, indicates the file name of the input. We used the flag -vn that will instruct ffmpeg to disable video recording. The -acodec flag will set the output audio codec to vorbis. The -y flag will overwrite output file without asking, so be careful when you use it.

In case we want to automatically process (batch process) all .WEBM video files in a folder we can use the following:

for FILE in *.webm;
do
    echo -e "Processing video '\e[32m$FILE\e[0m'";
    ffmpeg -i "${FILE}" -vn -y "${FILE%.webm}.ogg"
done

The above script will find all .WEBM files in the folder and process them one after the other.

 

UPDATE:

The following command will find all webm files that are in the current directory and in all sub-folders and extract the audio to ogg format.

find . -type f -iname "*.webm" -exec bash -c 'FILE="$1"; ffmpeg -i "${FILE}" -vn -y "${FILE%.webm}.ogg";' _ '{}' \;

The filename of the audio file will be the same as the webm video with the correct extension. The webm extension will be removed and replaced by the ogg extension e.g hi.webm will become hi.ogg

 


ffmpeg: Extract audio from .WEBM to .AAC 2

If you need to extract the audio from an .WEBM movie file to an .AAC audio file you can  execute the following:

FILE="the-file-you-want-to-process.webm";
ffmpeg -i "${FILE}" -vn -ab 128k -ar 44100 -strict -2 -y "${FILE%.webm}.aac";

The first command will assign the file name to a variable, we do this to avoid typing errors in the second command where we might want to use the same name for the audio file.

The second command, will use ffmpeg to extract the audio. The -i flag, indicates the file name of the input. We used the flag -vn that will instruct ffmpeg to disable video recording. The -ab flag will set the bit rate to 128k. The -ar flag will set the audio sample rate to 441000 Hz.  The flags -strict -2 are required as the native FFmpeg AAC encoder that is included with ffmpeg is considered an experimental encoder. The -y flag will overwrite output file without asking, so be careful when you use it.

In case we want to automatically process (batch process) all .WEBM video files in a folder we can use the following:

for FILE in *.webm; do
    echo -e "Processing video '\e[32m$FILE\e[0m'";
    ffmpeg -i "${FILE}" -vn -ab 128k -ar 44100 -strict -2 -y "${FILE%.webm}.aac";
done;

The above script will find all .WEBM files in the folder and process them one after the other.

 

UPDATE:

The following command will find all webm files that are in the current directory and in all sub-folders and extract the audio to aac format.

find . -type f -iname "*.webm" -exec bash -c 'FILE="$1"; ffmpeg -i "${FILE}" -vn -ab 128k -ar 44100 -strict -2 -y "${FILE%.webm}.aac";' _ '{}' \;

The filename of the audio file will be the same as the webm video with the correct extension. The webm extension will be removed and replaced by the aac extension e.g hi.webm will become hi.aac

 


ffmpeg: Extract audio from .WEBM to .MP3 6

If you need to extract the audio from an .WEBM movie file to an .MP3 audio file you can  execute the following:

FILE="the-file-you-want-to-process.webm";
ffmpeg -i "${FILE}" -vn -ab 128k -ar 44100 -y "${FILE%.webm}.mp3";

The first command will assign the file name to a variable, we do this to avoid typing errors in the second command where we might want to use the same name for the audio file.

The second command, will use ffmpeg to extract the audio. The -i flag, indicates the file name of the input. We used the flag -vn that will instruct ffmpeg to disable video recording. The -ab flag will set the bit rate to 128k. The -ar flag will set the audio sample rate to 441000 Hz.  The -y flag will overwrite output file without asking, so be careful when you use it.

In case we want to automatically process (batch process) all .WEBM video files in a folder we can use the following:

for FILE in *.webm; do
    echo -e "Processing video '\e[32m$FILE\e[0m'";
    ffmpeg -i "${FILE}" -vn -ab 128k -ar 44100 -y "${FILE%.webm}.mp3";
done;

The above script will find all .WEBM files in the folder and process them one after the other.

 

UPDATE:

The following command will find all webm files that are in the current directory and in all sub-folders and extract the audio to mp3 format.

find . -type f -iname "*.webm" -exec bash -c 'FILE="$1"; ffmpeg -i "${FILE}" -vn -ab 128k -ar 44100 -y "${FILE%.webm}.mp3";' _ '{}' \;

The filename of the audio file will be the same as the webm video with the correct extension. The webm extension will be removed and replaced by the mp3 extension e.g hi.webm will become hi.mp3

 


ffmpeg: Extract audio from .MP4 to .OGG 1

If you need to extract the audio from an .MP4 movie file to an .OGG audio file you can  execute the following:

FILE="the-file-you-want-to-process.mp4";
ffmpeg -i "${FILE}" -vn -acodec libvorbis -y "${FILE%.mp4}.ogg"

The first command will assign the file name to a variable, we do this to avoid typing errors in the second command where we might want to use the same name for the audio file.

The second command, will use ffmpeg to extract the audio. The -i flag, indicates the file name of the input. We used the flag -vn that will instruct ffmpeg to disable video recording. The -acodec flag will set the output audio codec to vorbis. The -y flag will overwrite output file without asking, so be careful when you use it.

In case we want to automatically process (batch process) all .MP4 video files in a folder we can use the following:

for FILE in *.mp4;
do
    echo -e "Processing video '\e[32m$FILE\e[0m'";
    ffmpeg -i "${FILE}" -vn -acodec libvorbis -y "${FILE%.mp4}.ogg";
done

The above script will find all .MP4 files in the folder and process them one after the other.

 

UPDATE:

The following command will find all mp4 files that are in the current directory and in all sub-folders and extract the audio to ogg format.

find . -type f -iname "*.mp4" -exec bash -c 'FILE="$1"; ffmpeg -i "${FILE}" -vn -acodec libvorbis -y "${FILE%.mp4}.ogg";' _ '{}' \;

The filename of the audio file will be the same as the mp4 video with the correct extension. The mp4 extension will be removed and replaced by the ogg extension e.g hi.mp4 will become hi.ogg