Ubuntu: Headless wireshark (or wireshark from terminal)

Recently, we wanted to use wireshark on an Ubuntu through ssh and no X-Server forwarding enabled.
After a quick search we found tshark.

TShark is a network protocol analyzer. It lets you capture packet data from a live network, or read packets from a previously saved capture file, either printing a decoded form of those packets to the standard output or writing the packets to a file. TShark‘s native capture file format is pcap format, which is also the format used by tcpdump and various other tools.
Without any options set, TShark will work much like tcpdump. It will use the pcap library to capture traffic from the first available network interface and displays a summary line on stdout for each received packet.
TShark is able to detect, read and write the same capture files that are supported by Wireshark.

From: man tshark

Install tshark on Ubuntu

sudo apt-get install tshark -y;

Using tshark to capture all traffic on eth0 to a pcap file

sudo tshark -i eth0 -w something.pcap;

Note: If you just want to capture network traffic on a network interface and not use the additional features wireshark has to offer, you can also use tcpdumpas follows

#The following command will create a files that has in its name the current date and time using the date function.
sudo tcpdump -i eth0 -w "data.`date +%Y-%m-%d\ %H.%M`.pcap";

Bash: Remove the last character from each line

The following script, uses rev and cut to remove the last character from each line in a pipe.
rev utility reverses lines character-wise.
cut removes sections  from each of line.
It is a very simple script where we reverse the line once, remove the first character (which was the last one in the original form of the line) and finally we reverse the line back with the last character missing.

echo -e "hi\nHI" | rev | cut -c 2- | rev;

# Will produce:


Bash: Switch positions between all characters in odd positions with characters in even positions

The following awk script allowed us to switch position of all characters placed in odd numbered positions with their next neighboring even numbered position characters.
In detail what it does is to create a for loop that skips one character every time and then it prints each pair in reverse order (it will print the second character first, then the first one, then the fourth and so on).

echo "123456789" | awk -vFS= '{for (i = 1; i <= NF; i+=2) {printf $(i+1)$i""} printf "\n"}';

# Will produce 214365879

echo "1234567890" | awk -vFS= '{for (i = 1; i <= NF; i+=2) {printf $(i+1)$i""} printf "\n"}';

# Will produce 2143658709

Please note that we set the built-in variable FS (The input field separator which is a space by default) to the empty string so that each character is treated like a different field by NF (The number of fields in the current input record).


Bash: Print time stamp in front of every line in a pipe

Recently, we received a binary that collected data from a web service and it printed them on screen.
The binary did not print a time stamp in front of each line so we had to improvise of a way to add the time stamp to the logs without modifying the binary.

The solution we came to was to use awk to prepend the time stamp in front of every line using a pipe.
Specifically, our solution was the following:

server_application 2>&1 | awk '{ print strftime("%Y-%m-%d %H:%M:%S"), $0; fflush(); }'

What we did there was to start our binary server_application, redirect stderr to stdout (using 2>&1) so that we will have only one stream and then we read the lines one by one using awk and printed the time stamp right before the line ($0) using strftime.
The strftime() function formats the broken-down time according to the format specification format.
fflushforces a write of all user-space buffered data for the given output or update stream via the stream’s underlying write function. We call it at each line to make sure that we do not cause additional delay in presenting the data due to buffering limitations caused by our prints.


$ echo -e "hi\nHI" 2>&1 | awk '{ print strftime("%Y-%m-%d %H:%M:%S"), $0; fflush(); }'
2017-06-21 20:33:41 hi
2017-06-21 20:33:41 HI

How to execute `find` that ignores .git directories

Trying to find a source code file by its content using find and -exec grep, can some times result in getting results from the repository .git folders as well.

This behavior not only does it provide results you do not need but it also makes your search slower.
Below, we propose a couple of solutions on how to make a more efficient search.

Example 1: Ignore all .git folders no matter where they are in the search path

For find to ignore all .git folders, even if they appear on the first level of directories or any in-between until the last one, add -not -path '*/\.git*' to your command as in the example below.
This parameter will instruct find to filter out any file that has anywhere in its path the folder .git. This is very helpful in case a project has dependencies in other projects (repositories) that are part of the internal structure.

find . -type f -not -path '*/\.git/*';

Note, if you are using svn use:

find . -type f -not -path '*/\.svn/*';

Example 2: Ignore all hidden files and folders

To ignore all hidden files and folders from your find results add -not -path '*/\.*' to your command.

find . -not -path '*/\.*';

This parameter instructs find to ignore any file that has anywhere in its path the string /. which is any hidden file or folder in the search path!

How to capture all network traffic of a single process

Method A: Using strace

If the process is not started yet, you can start it as a new process and monitor it using the following

strace -f -e trace=network -s 10000 <PROCESS WITH ARGUMENTS>;

If the process is already started and you know its PID you can use the following

strace -f -e trace=network -s 10000 -p <PID>;

strace is a very helpful utility that can be used to trace system calls and signals.

Parameters Legend:

  • -f Instructs strace to trace all child processes as they are created by the currently traced processes as a result of the fork, vfork and clone system calls.
    Note that -p PID -f will attach all threads of process PID if it is multi-threaded, not only thread with id PID.
  • -e trace=%network strace will trace all the network related system calls only if used alone.
  • -s strsize Specifies the maximum string size to print (the default is 32). Note that filenames are not considered strings and are always printed in full.
  • -p PID Attaches strace to the process with the process ID PID and starts tracing. The trace may be terminated at any time by a keyboard interrupt signal (CTRL-C). strace will respond by detaching itself from the traced process(es) leaving it (them) to continue running. Multiple -p options can be used to attach to many processes in addition to command (which is optional if at least one -p option is given). -p "`pidof APPLICATION`" syntax is supported.

In the simplest case strace runs the specified command until it exits. It intercepts and records the system calls which are called by a process and the signals which are received by a process. The name of each system call, its arguments and its return value are printed on standard error or to the file specified with the -o option.
strace is a useful diagnostic, instructional, and debugging tool. System administrators, diagnosticians and trouble-shooters will find it invaluable for solving problems with programs for which the source is not readily available since they do not need to be recompiled in order to trace them. Students, hackers and the overly-curious will find that a great deal can be learned about a system and its system calls by tracing even ordinary programs. And programmers will find that since system calls and signals are events that happen at the user/kernel inter‐ face, a close examination of this boundary is very useful for bug isolation, sanity checking and attempting to capture race conditions.

From: man strace

Method B: Using an isolated network namespace and Wireshark

Please note that this method might not work for all kernels.
It was tested on Fedora 25 (64 bit) with success.

Create the test network namespace

A network namespace is logically another copy of the network stack, with its own routes, firewall rules, and network devices.
With the following command we will create a network namespace called test.

sudo ip netns add test;

ip netns add NAME Creates a new named network namespace.
If NAME is available in /var/run/netns/ this command creates a new network namespace and assigns to it then name NAME.

Create two virtual network interfaces (veth0 and veth1) for our Virtual eXtended LAN (VXLAN)

The following command will create veth0 and veth1 virtual network interfaces.

sudo ip link add veth0 type veth peer name veth1;

ip link add with no link argument specified to a physical device to operate on, it adds a VXLAN virtual link.
Note: veth1 will act as a gateway later on.

Change the active namespace of the veth0 interface

With the following command, we move veth0 to our test network namespace.

sudo ip link set veth0 netns test;

Some devices are not allowed to change network namespace: loopback, bridge, ppp, wireless. These are network namespace local devices. In such case ip tool will return Invalid argument error. It is possible to find out if device is local to a single network namespace by checking netns-local flag in the output of the ethtool:
ethtool -k DEVICE;
To change network namespace for wireless devices the iw tool can be used. But it allows to change network namespace only for physical devices and by process PID.
From man ip-link

Configure the IP addresses of the virtual interfaces

#Set the IP of veth0 to and veth0 to
sudo ip netns exec test ifconfig veth0 up netmask;
sudo ifconfig veth1 up netmask;

Configure the routing in the test namespace

The following command will set the default gateway for veth0 to be the IP which is the address we gave veth1 in the previous step.

sudo ip netns exec test route add default gw dev veth0;

Then we make sure ip_forward is active by issuing the following command

sudo sh -c 'echo 1 > /proc/sys/net/ipv4/ip_forward';

Then, we establish a NAT rule to forward all the traffic of test network namespace to a physical device

sudo iptables -t nat -A POSTROUTING -s -o <PHYSICAL DEVICE e.g. eth0> -j SNAT --to-source <PHYSICAL DEVICE IP>;

Actually using the isolated network namespace

Add root to the list of users that is allowed to start an X application

xhost +si:localuser:root;

Start wireshark in your test network namespace

sudo ip netns exec test wireshark &

In wireshark start monitoring the data on the device veth0.

Finally, start the application you wish to monitor its network traffic

sudo ip netns exec test firefox;

How to get the pid of the last executed command that was sent to the background in a bash shell

Recently we came to the need of writing a bash script that needed periodically to check if a specific process, that was started by the script, had ended and restart it (something like watchdog but with not so many features).

To achieve this, we used the one of the shell special parameters, the $!. Like all other special parameters $! may only be referenced and the user cannot make an assignment to it.

($!) Expands to the process ID of the job most recently placed into the background, whether executed as an asynchronous command or using the bg builtin command.


Example of Usage

In this example we wanted to get the PID of the application called server to be used later on in the script.

server &
echo $!; #This will print the process ID of the 'server' application

How to set a static IP Address from the Command Line in GNU/Linux using ip addr and ip route

Assuming you want to make the following changes to the network device eth0

  1. Change the IP to the static value
  2. Set the Subnet Mask to
  3. Set the Default Gateway for the device to be

and you want to avoid using ifconfig and route that are obsolete you can perform these changes using the following two commands

sudo ip addr add dev eth0;
sudo ip route add default via dev eth0;

Please note that the netmask is given in CIDR notation (it is the /24 right after the IP of the device in the ip addr command).

A subnet mask (netmask) is a bitmask that encodes the prefix length in quad-dotted notation: 32 bits, starting with a number of 1 bits equal to the prefix length, ending with 0 bits, and encoded in four-part dotted-decimal format: A subnet mask encodes the same information as a prefix length, but predates the advent of CIDR. In CIDR notation, the prefix bits are always contiguous, whereas subnet masks may specify non-contiguous bits.

From Wikipedia:

File permissions change date

Recently we wanted to check when did the permissions of a specific file changed.
Unfortunately, there exists no such flag and we do not have a 100% working solution for it.

What we did was to check the last modification time of the file status information (ctime) using the ls -lc command.
This command could indicate the last permissions change time but it is not a reliable source as it represents the modification time of other elements as well.

The modification time of the file status information (ctime) gets updated when any inode information regarding the file changes.
This means that the modification time of the file status information (ctime) will get updated when any of the following changes:

  • owner – The numeric user ID (UID) of the file’s owner.
  • group – The numeric group ID (GID) of the file’s group.
  • link count – The number of links to the file.
  • mode – The bit string that indicated the permissions and privileges
  • serial – The serial number of the file.
  • device – The numeric ID of the device containing the file.

Explanation of ls parameters

  • The parameter -c of the ls command when used with the -l will show ctime and sort by name.
  • The parameter -c of the ls command when used with the -l and the -t will show ctime and sort by ctime (newest first).

Example that demonstrates that we get different values in the time column of -l when -c is used

$ ls -lc ~/.ssh/
total 28
-rwx------. 1 george george  225 May 16 17:05 config
-rwx------. 1 george george 1743 Jun  2 13:36 id_rsa
-rwxrwx---. 1 george george  405 May 16 17:05
-rwxrwx---. 1 george george   32 May 16 17:05 Details.txt
-rw-r--r--. 1 george george 9155 May 30 14:32 known_hosts

$ ls -l ~/.ssh/
total 28
-rwx------. 1 george george  225 Mar 22 11:36 config
-rwx------. 1 george george 1743 Jan 25 10:22 id_rsa
-rwxrwx---. 1 george george  405 Jan 25 10:22
-rwxrwx---. 1 george george   32 Jan 25 10:22 Details.txt
-rw-r--r--. 1 george george 9155 May 30 14:32 known_hosts

How to instruct SSH use only my password and ignore my (rsa) key

Recently, we wanted to connect to a machine via SSH without using the default RSA key that was available in the client’s profile (~/.ssh/id_rsa).

We needed to avoid using the public key authentication method for two reasons:

  1. The client did not want to share the passphrase with us
  2. We did not want to move the key, not even temporarily

So, to connect via SSH while ignoring the key completely we connected using the following command

ssh -o PreferredAuthentications=keyboard-interactive,password -o PubkeyAuthentication=no [email protected];

Explanation of parameters:

  • -o Was used to give options in the format used in the configuration file (/etc/ssh/ssh_config). It is useful for specifying options for which there is no separate command-line flag available.
  • -o PreferredAuthentications can be used to change the default order of authentication and bypass the GSSAPI-based authentication, the host-based authentication, the public key authentication and the challenge-response authentication.
    -o PreferredAuthentications=keyboard-interactive,password instructs the server to perform the authentication through the keyboard-interactive method and if that method is not available to use the password method.
    The keyboard-interactive authentication method is a request for all different pieces of information needed for the authentication. The server can specify, which inputs need to be hidden when user types them and which are not.
    The password authentication is a request for a single password. There is no configuration sent by the server. So the client decides how to format the prompt.
  • -o PubkeyAuthentication=no Specifies whether to try public key authentication. By setting the value to no it disables it.