Monthly Archives: April 2016


Git: Delete all local branches

The following command will:

  • print all branches that were merged to master
  • then filter out the branch named master and the branch you are currently switched to
  • and finally, it will delete the rest (one branch at a time).
git branch --merged master | grep -v -e "\*" -e "master" | xargs git branch -D

Tip:

To cleanup any remote-tracking references that no longer exist on the remote use the following:

git fetch --prune

Bash/FFMPEG: Batch resize .mp4 videos to fixed resolution

We needed to shrink a bunch of mp4 videos so that they would have the same size as the screen of an android device.
We did that both to save space on the internal memory of the device and to make the device perform as efficient as possible as it would not have to shrink the video on the fly.

The command we used was the following:

find . -type f -name "*.mp4" -exec bash -c 'FILE="$1"; ffmpeg -i "${FILE}" -s 1280x720 -acodec copy -y "${FILE%.mp4}.shrink.mp4";' _ '{}' \;

What this command does is the following:

  • Find all files in current folder (and sub-folders) that have the extension .mp4
  • For each file, create a new bash instance in which it will call ffmpeg taking as first parameter the filename that matched
  • -i "${FILE}"ffmpeg will take as input the filename we matched
  • -s 1280x720 – Then change the video size to 1280x720
  • -acodec copy – It will keep the audio as is
  • -y "${FILE%.mp4}.shrink.mp4 – Finally, create a new file (or overwrite existing) that has the extension .shrink.mp4 in the same folder

PHP: Convert JavaScript-escaped Unicode characters to HTML hex references

There are cases where one might receive in PHP, escaped Unicode characters from the client side JavaScript. According to the RFC it is normal for JavaScript to convert characters to that format and in effect that we receive any character in the escaped format of \uXXXX in PHP.

Any character may be escaped.
If the character is in the Basic Multilingual Plane (U+0000 through U+FFFF),
then it may be represented as a six-character sequence:
a reverse solidus, followed by the lowercase letter u, followed by four hexadecimal digits that encode the character's code point.
The hexadecimal letters A though F can be upper or lowercase.

A sample input you might receive could look like this George\u2019s treasure box instead of George’s treasure box.

This kind of input should not be stored as is as it does not make sense to the HTML language, instead we should fix it up using preg_replace.

$decoded = preg_replace('/\\\\u([a-fA-F0-9]{4})/', '&#x\\1;', $input);

The above command will look for all instances of \uXXXX in the $input and it will replace each one with the appropriate character using the XXXX value that it will match.

What this part '/\\\\u([a-fA-F0-9]{4})/' of the code do is the following:

  • \\\\ – Find the character \ in the string, the reason we have four \ instead of one, is because it has special meaning in the regular expression and we have to escape it. For that reason we need to use two of them and get \\. After that, we need to escape each of them again due to the special meaning they have in PHP and we end up with four of them.
  • u – The previous step must be followed by a u character.
  • ([a-fA-F0-9]{4}) – After the previous step has matched, we need to match 4 characters. Each of them must be either a character from A-Z or a-z or 0-9.

This part '&#x\\1;' will:

  • &#x – Is a constant string that will print the characters &#x. These characters will instruct HTML to print the character that will occur using hexadecimal entity reference that will follow.
  • \\1 – Contains the reference of the 1st parenthesized pattern. In this case we only have a parenthesis around the XXXX part of the \uXXXX so \\1 will be replaced with the XXXX value.

HOWTO: Make Terminator Terminal Act Like Guake Terminal in Ubuntu 16.04 LTS (The easy ways) 1

First way to make terminator toggle its visibility using the F12 key (like guake)

  • Start terminator
  • Right click anywhere in the terminal area and click on the Preferences option

terminator-01

  • In the new window, click on the Keybindings tab and scroll down until you find the line that has the following information:
    Name : hide_window
    Action : Toggle window visibility

terminator-02

  • Click on the Keybinding column (3rd column), the value will change to New accelerator..., hit the key combination you want to be used to toggle the visibility of terminator. If you want the same behavior as guake, hit F12. You will see that the value in the Keybinding column will change to F12.
  • Hit the close button to close the settings window.
  • In the terminal try the key you just set (e.g F12) to see if it works. If it doesn’t work and in the case of F12 writes on the terminal a ~, close terminator and re-open it for the changes to get applied.

Second way to make terminator toggle its visibility using the F12 key (like guake)

  • Create the folder tree ~/.config/terminator (maybe it exists already). Please note that the . in front of config is purposely there, it is the way to hide a folder.
  • In the folder create a file named config (the full path would be ~/.config/terminator/config) and put the following as content:
[global_config]
[keybindings]
  hide_window = F12
[layouts]
  [[default]]
    [[[child1]]]
      parent = window0
      type = Terminal
    [[[window0]]]
      parent = ""
      type = Window
[plugins]
[profiles]
  [[default]]
  • Save the file and start terminator, pressing the F12 key should hide the terminal, pressing it once more should make it reappear.

HOWTO: Make Terminator Terminal Act Like Guake Terminal in Ubuntu 16.04 LTS (XenialXerus) Desktop edition x64 bit architecture 1

We propose an alternative solution to making terminator act like guake that requires two additional packages: xdotool and wmctrl.

Our proposal will launch terminator if there is not instance running.

In Ubuntu you can install the needed packages from the official repositories using sudo apt-get install xdotool wmctrl.

Using a text editor, create in you home folder a file named nano ~/toggle_visibility.sh and copy there the contents of the following chunk of code. You can also use nano, from a terminal issue nano ~/toggle_visibility.sh, then paste the code and hit CTRL+X to exit. When prompted if you want to save press ‘Y’ and hit enter.

#!/bin/bash

#The purpose of this script is to allow the user to toggle the visibility of (almost) any window.
#Please note it will work on the first match, so if there are multiple instances of an application it would be a random window of them the one to be affected.
#Usually it will control the window with the smallest PID.

#Checking that all dependencies are met, since we cannot proceed without them.
declare -a DEPENDENCIES=("xdotool" "wmctrl");
declare -a MANAGERS=("dnf" "apt-get");

for DEPENDENCY in ${DEPENDENCIES[@]}; do
    echo -n "Checking if $DEPENDENCY is available";
    if hash $DEPENDENCY 2>/dev/null; then
        echo "- OK, Found";
    else
        echo "- ERROR, Not Found in $PATH";
        for MANAGER in ${MANAGERS[@]}; do
            if hash $MANAGER 2>/dev/null; then
                echo -n "$DEPENDENCY is missing, would you like to try and install it via $MANAGER now? [Y/N] (default is Y): ";
                read ANSWER;
                if [[ "$ANSWER" == "Y" || "$ANSWER" == "y" || "$ANSWER" == "" ]]; then
                    sudo "$MANAGER" install "$DEPENDENCY";
                else
                    echo "Terminating";
                    exit -1;
                fi
            fi
        done
    fi
done

APPLICATION="$1";
FULL_COMMAND="$2";

#Checking if the application name provided by the user exists
if ! hash $APPLICATION 2>/dev/null; then
    echo -e "$APPLICATION does not seem to be a valid executable\nTerminating";
    exit -2;
fi

#Checking if the application is running.
PID=$(pgrep -u `whoami` -f "$FULL_COMMAND" | head -n 1);

#If the application is not running, we will try to launch it.
if [ -z $PID ]; then
  echo "$FULL_COMMAND not running, launching it..";
    $FULL_COMMAND;
else
    #Since the application has a live instance, we can proceed with the rest of the code.
    #We will get the PID of the application that is currently focused, if it is not the application we passed as parameter we will change the focus to that. In the other case, we will minimize the application.
  echo -n "$FULL_COMMAND instance found - ";
    FOCUSED=$(xdotool getactivewindow getwindowpid);
    if [[ $PID == $FOCUSED ]]; then
    echo "It was focused so we are minimizing it";
        #We minimize the active window which we know in this case that it is the application we passed as parameter.
        xdotool getactivewindow windowminimize;
    else
    echo "We are setting the focus on it";
        #We set the focus to the application we passed as parameter. If it is minimized it will be raised as well.
        wmctrl -x -R $APPLICATION;
    fi
fi

exit 0

Afterwards, you need to make the script an executable so you should issue chmod +x ~/toggle_visibility.sh to do that.

Then, execute ~/toggle_visibility.sh in your terminal once. We need to do that in order to install any missing dependencies for the tool.

Finally, you need to create a custom shortcut that will call the script using the key combination you like at any point.

To complete the procedure:

  1. Go to ‘System Settings’ either by clicking on the menu on the top right corner that looks like a light bulb or by issuing the following in a terminal unity-control-center to start the unity control center.
  2. In the newly appeared window, click on the ‘keyboard’ icon that is in the category ‘Hardware’.
  3. After that, click on the tab ‘Shortcuts’
  4. and on the left list, click on custom shortcuts.
  5. You will see a button with the + sign right next to the list, click that.
  6. In the dialog box that will appear enter the following:
    – In the name field enter anything you like. e.g ‘Toggle Terminator Visibility’
    – In the command field enter /home/<USER>/toggle_visibility.sh terminator "^/usr/bin/python /usr/bin/terminator$" where <USER> enter your own username.
    – Click apply.
  7. You will see a new row with two columns with the name you just set in the first column. Click on the second column, where it should say ‘Disabled’ and the press the key combination you want for toggling terminator e.g F12

You are ready to go 🙂

Just try the key combination you just provided and terminator will appear in front of you. Pressing it once more it will hide it.

NOTE: Please keep in mind that the above script can be used for other applications as well. In step 7, we gave as parameter the name of the application to be used, if you change that you could use it with other applications like Firefox.


Grep lines that do not begin with ‘#’ or ‘;’

Recently, we wanted to modify  the squid configuration file, which is really really big!

wc -l /etc/squid/squid.conf
7898 /etc/squid/squid.conf

We wanted to find all active rules that are enabled to modify our proxy server. Out of those ~8K lines less than 20 are actually active configuration, the rest is documentation.

To find all active configuration lines we needed to find all lines that:

  • are not empty
  • do not start with #
  • do not start with ;

To do this we used the following grep command

grep "^[^#;]" /etc/squid/squid.conf

The first ^ refers to the beginning of the line, this way if in a line there is some configuration and after that there is a comment it will not be excluded by mistake. The rest, [^#;] matches any character which is not # or ;.

This is what was actually in my configuration file (out of ~8K lines)

acl SSL_ports port 443
acl Safe_ports port 80        # http
acl Safe_ports port 21        # ftp
acl Safe_ports port 443        # https
acl Safe_ports port 70        # gopher
acl Safe_ports port 210        # wais
acl Safe_ports port 1025-65535    # unregistered ports
acl Safe_ports port 280        # http-mgmt
acl Safe_ports port 488        # gss-http
acl Safe_ports port 591        # filemaker
acl Safe_ports port 777        # multiling http
acl CONNECT method CONNECT
http_access deny !Safe_ports
http_access deny CONNECT !SSL_ports
http_access allow localhost manager
http_access deny manager
http_access allow localhost
http_access deny all
http_port 3128
coredump_dir /var/spool/squid
refresh_pattern ^ftp:        1440    20%    10080
refresh_pattern ^gopher:    1440    0%    1440
refresh_pattern -i (/cgi-bin/|\?) 0    0%    0
refresh_pattern (Release|Packages(.gz)*)$      0       20%     2880
refresh_pattern .        0    20%    4320

Copy all databases to another host

The following command will use mysqldump to create a dump of all available databases in the OLD_HOST that are available for the user OLD_USER.
The results will be imported to another server via the mysql pipe.

OLD_USER="myUser"; OLD_PASS="myPASS"; OLD_HOST="myHost";
NEW_USER="myUserNEW"; NEW_PASS="myPASSNEW"; NEW_HOST="myHostNEW";
mysqldump -u "$OLD_USER" -p"$OLD_PASS" -h "$OLD_HOST" --all-databases | mysql -h "$NEW_HOST" -u "$NEW_USER" -p"$NEW_PASS";

The user must have the LOCK TABLES privilege for the above command to work or else you will get the following error.

mysqldump: Got error: 1044: "Access denied for user 'OLD_USER'@'OLD_HOST' to database 'DBNAME'" when using LOCK TABLES

In case you cannot give the privilege to the user, then use the parameter --single-transaction to mitigate the problem. The command changes as follows.

OLD_USER="myUser"; OLD_PASS="myPASS"; OLD_HOST="myHost";
NEW_USER="myUserNEW"; NEW_PASS="myPASSNEW"; NEW_HOST="myHostNEW";
mysqldump -u "$OLD_USER" -p"$OLD_PASS" -h "$OLD_HOST" --single-transaction --all-databases | mysql -h "$NEW_HOST" -u "$NEW_USER" -p"$NEW_PASS";

In case you want to copy only specific databases, use the following

OLD_USER="myUser"; OLD_PASS="myPASS"; OLD_HOST="myHost"; OLD_DBS=("DB1" "DB2");
NEW_USER="myUserNEW"; NEW_PASS="myPASSNEW"; NEW_HOST="myHostNEW";
mysqldump -u "$OLD_USER" -p"$OLD_PASS" -h "$OLD_HOST" "${OLD_DBS[@]}" | mysql -h "$NEW_HOST" -u "$NEW_USER" -p"$NEW_PASS";

In case you want to copy only specific tables from a database, use the following

OLD_USER="myUser"; OLD_PASS="myPASS"; OLD_HOST="myHost"; OLD_DB="DB1"; OLD_TABLES=("TBL1" "TBL2");
NEW_USER="myUserNEW"; NEW_PASS="myPASSNEW"; NEW_HOST="myHostNEW"; NEW_DB="NewDB";
mysqldump -u "$OLD_USER" -p"$OLD_PASS" -h "$OLD_HOST" "$OLD_DB" "${OLD_TABLES[@]}" | mysql -h "$NEW_HOST" -u "$NEW_USER" -p"$NEW_PASS" "$NEW_DB";

Export/Backup all MySQL databases

The following command will use mysqldump to create a dump of all available databases in the HOST that are available for the user USER.
The results will be found in a file that begins with the current date and will contain the hostname as part of the name.

USER="myUser"; PASS="myPASS"; HOST="myHost"; mysqldump -u "$USER" -p"$PASS" -h "$HOST" --all-databases > "`date +%F`-backup-all.$HOST.sql"

The user must have the LOCK TABLES privilege for the above command to work or else you will get the following error.

mysqldump: Got error: 1044: "Access denied for user 'USER'@'HOST' to database 'DBNAME'" when using LOCK TABLES

In case you cannot give the privilege to the user, then use the parameter --single-transaction to mitigate the problem. The command changes as follows.

USER="myUser"; PASS="myPASS"; HOST="myHost"; mysqldump -u "$USER" -p"$PASS" -h "$HOST" --single-transaction --all-databases > "`date +%F`-backup-all.$HOST.sql"

Bash Function to print out the files and the lines that contain a needle

The following code will create a function in bash that accepts two parameters (1: the place to search in, 2: the value to search for).

You can place it in your ~/.bashrc file to have it available whenever you open a bash shell.

#1. Copy/paste the below lines in your .bashrc

#takes 2 parameters (1: the haystack to search in, 2: the needle)
# Will print out the files and the lines that contain the needle
xfind(){
  FIND_VAR="$2";
  STACK="$1";
  if [ -f "$STACK" ] || [ -d "$STACK" ]; then
    find "$STACK" \
      -exec grep --color "$FIND_VAR" -sl '{}' \; \
      -exec grep "$FIND_VAR" -s '{}' \;
  else
    echo "ERROR: No file or folder with the name '$STACK' exist";
  fi
}
#2. Run source ~/.bashrc -- to reload 

Usage examples:

xfind . "bar";
xfind /etc/ "conf";

Recursively change extension of multiple files using find

Assuming you have a whole bunch of files that you need to change their extension from one to another, you can use the following commands after setting the values for the BEFORE and the AFTER variables to the values you need.

BEFORE='.txt'; AFTER='.csv'; find . -type f -name "*$BEFORE" -exec bash -c 'mv "$1" "${1%$2}$3"' _ '{}' "$BEFORE" "$AFTER" \;

What the above will do is: after setting the two input variables it will call find in the current directory (with recursion) and find all files that their suffix is the value you set in BEFORE. Then, for each match it will create a new shell terminal in which it will rename the file by removing the old suffix and then attaching the new one. We pass the input variables as parameters to the new shell and that is why inside the code of the shell we are using variables $1, $2 and $3. The reason we had to issue a new shell is because we wanted to reuse the ‘{}’ variable.