GNU/Linux


Regular expression to match any ASCII character

The following regular expression will match any ASCII character (character values [0-127]).

[\x00-\x7F]

The next regular expression makes the exact opposite match, it will match any character that is NOT ASCII (character values greater than 127).

[^\x00-\x7F]

 

gEdit - regular expression to match any ASCII character

gEdit – regular expression to match any ASCII character

gEdit - regular expression to match any Non-ASCII character

gEdit – regular expression to match any Non-ASCII character


NTFS Support on CentOS 7

Solution


sudo yum --enablerepo=extras install epel-release;
sudo yum install ntfs-3g -y;

Background – Explanation of commands

By default, CentOS does not have installed the necessary drivers to mount ntfs drives.

sudo yum --enablerepo=extras install epel-release;

To install them, you need to enable the Extra Packages for Enterprise Linux (EPEL).

Extra Packages for Enterprise Linux (or EPEL) is a Fedora Special Interest Group that creates, maintains, and manages a high quality set of additional packages for Enterprise Linux, including, but not limited to, Red Hat Enterprise Linux (RHEL), CentOS and Scientific Linux (SL), Oracle Linux (OL).

EPEL packages are usually based on their Fedora counterparts and will never conflict with or replace packages in the base Enterprise Linux distributions. EPEL uses much of the same infrastructure as Fedora, including buildsystem, bugzilla instance, updates manager, mirror manager and more.

From: https://fedoraproject.org/wiki/EPEL

You can install EPEL by running yum --enablerepo=extras install epel-release. The epel-release package is included in the CentOS Extras repository that is enabled by default. The package includes gpg keys for package signing and repository information. Installing this package for your Enterprise Linux version should allow you to use normal tools such as yum to install packages and their dependencies.

sudo yum install ntfs-3g -y;

After you’ve enabled the repository, you should be able to install the Linux NTFS userspace driver packaged in ntfs-3g. ntfs-3g is a stable, open source, GPL licensed, POSIX, read/write NTFS driver for Linux and many other operating systems. It provides safe handling of the Windows XP, Windows Server 2003, Windows 2000, Windows Vista, Windows Server 2008 and Windows 7 NTFS file systems. NTFS-3G can create, remove, rename, move files, directories, hard links, and streams; it can read and write normal and transparently compressed files, including streams and sparse files; it can handle special files like symbolic links, devices, and FIFOs, ACL, extended attributes; moreover it provides full file access right and ownership support.

To install it, we used the following command: yum install ntfs-3g -y.


Ubuntu: Headless wireshark (or wireshark from terminal)

Recently, we wanted to use wireshark on an Ubuntu through ssh and no X-Server forwarding enabled.
After a quick search we found tshark.

TShark is a network protocol analyzer. It lets you capture packet data from a live network, or read packets from a previously saved capture file, either printing a decoded form of those packets to the standard output or writing the packets to a file. TShark‘s native capture file format is pcap format, which is also the format used by tcpdump and various other tools.
Without any options set, TShark will work much like tcpdump. It will use the pcap library to capture traffic from the first available network interface and displays a summary line on stdout for each received packet.
TShark is able to detect, read and write the same capture files that are supported by Wireshark.

From: man tshark

Install tshark on Ubuntu


sudo apt-get install tshark -y;

Using tshark to capture all traffic on eth0 to a pcap file


sudo tshark -i eth0 -w something.pcap;

Note: If you just want to capture network traffic on a network interface and not use the additional features wireshark has to offer, you can also use tcpdumpas follows


#The following command will create a files that has in its name the current date and time using the date function.
sudo tcpdump -i eth0 -w "data.`date +%Y-%m-%d\ %H.%M`.pcap";


Bash: Remove the last character from each line

The following script, uses rev and cut to remove the last character from each line in a pipe.
rev utility reverses lines character-wise.
cut removes sections  from each of line.
It is a very simple script where we reverse the line once, remove the first character (which was the last one in the original form of the line) and finally we reverse the line back with the last character missing.


echo -e "hi\nHI" | rev | cut -c 2- | rev;

# Will produce:
h
H

 


Bash: Switch positions between all characters in odd positions with characters in even positions

The following awk script allowed us to switch position of all characters placed in odd numbered positions with their next neighboring even numbered position characters.
In detail what it does is to create a for loop that skips one character every time and then it prints each pair in reverse order (it will print the second character first, then the first one, then the fourth and so on).


echo "123456789" | awk -vFS= '{for (i = 1; i <= NF; i+=2) {printf $(i+1)$i""} printf "\n"}';

# Will produce 214365879

echo "1234567890" | awk -vFS= '{for (i = 1; i <= NF; i+=2) {printf $(i+1)$i""} printf "\n"}';

# Will produce 2143658709

Please note that we set the built-in variable FS (The input field separator which is a space by default) to the empty string so that each character is treated like a different field by NF (The number of fields in the current input record).

 


Bash: Print time stamp in front of every line in a pipe

Recently, we received a binary that collected data from a web service and it printed them on screen.
The binary did not print a time stamp in front of each line so we had to improvise of a way to add the time stamp to the logs without modifying the binary.

The solution we came to was to use awk to prepend the time stamp in front of every line using a pipe.
Specifically, our solution was the following:


server_application 2>&1 | awk '{ print strftime("%Y-%m-%d %H:%M:%S"), $0; fflush(); }'

What we did there was to start our binary server_application, redirect stderr to stdout (using 2>&1) so that we will have only one stream and then we read the lines one by one using awk and printed the time stamp right before the line ($0) using strftime.
The strftime() function formats the broken-down time according to the format specification format.
fflushforces a write of all user-space buffered data for the given output or update stream via the stream’s underlying write function. We call it at each line to make sure that we do not cause additional delay in presenting the data due to buffering limitations caused by our prints.

Example


$ echo -e "hi\nHI" 2>&1 | awk '{ print strftime("%Y-%m-%d %H:%M:%S"), $0; fflush(); }'
2017-06-21 20:33:41 hi
2017-06-21 20:33:41 HI


How to execute `find` that ignores .git directories

Trying to find a source code file by its content using find and -exec grep, can some times result in getting results from the repository .git folders as well.

This behavior not only does it provide results you do not need but it also makes your search slower.
Below, we propose a couple of solutions on how to make a more efficient search.

Example 1: Ignore all .git folders no matter where they are in the search path

For find to ignore all .git folders, even if they appear on the first level of directories or any in-between until the last one, add -not -path '*/\.git*' to your command as in the example below.
This parameter will instruct find to filter out any file that has anywhere in its path the folder .git. This is very helpful in case a project has dependencies in other projects (repositories) that are part of the internal structure.


find . -type f -not -path '*/\.git/*';

Note, if you are using svn use:


find . -type f -not -path '*/\.svn/*';

Example 2: Ignore all hidden files and folders

To ignore all hidden files and folders from your find results add -not -path '*/\.*' to your command.


find . -not -path '*/\.*';

This parameter instructs find to ignore any file that has anywhere in its path the string /. which is any hidden file or folder in the search path!


How to capture all network traffic of a single process

Method A: Using strace

If the process is not started yet, you can start it as a new process and monitor it using the following


strace -f -e trace=network -s 10000 <PROCESS WITH ARGUMENTS>;

If the process is already started and you know its PID you can use the following


strace -f -e trace=network -s 10000 -p <PID>;

strace is a very helpful utility that can be used to trace system calls and signals.

Parameters Legend:

  • -f Instructs strace to trace all child processes as they are created by the currently traced processes as a result of the fork, vfork and clone system calls.
    Note that -p PID -f will attach all threads of process PID if it is multi-threaded, not only thread with id PID.
  • -e trace=%network strace will trace all the network related system calls only if used alone.
  • -s strsize Specifies the maximum string size to print (the default is 32). Note that filenames are not considered strings and are always printed in full.
  • -p PID Attaches strace to the process with the process ID PID and starts tracing. The trace may be terminated at any time by a keyboard interrupt signal (CTRL-C). strace will respond by detaching itself from the traced process(es) leaving it (them) to continue running. Multiple -p options can be used to attach to many processes in addition to command (which is optional if at least one -p option is given). -p "`pidof APPLICATION`" syntax is supported.

In the simplest case strace runs the specified command until it exits. It intercepts and records the system calls which are called by a process and the signals which are received by a process. The name of each system call, its arguments and its return value are printed on standard error or to the file specified with the -o option.
strace is a useful diagnostic, instructional, and debugging tool. System administrators, diagnosticians and trouble-shooters will find it invaluable for solving problems with programs for which the source is not readily available since they do not need to be recompiled in order to trace them. Students, hackers and the overly-curious will find that a great deal can be learned about a system and its system calls by tracing even ordinary programs. And programmers will find that since system calls and signals are events that happen at the user/kernel inter‐ face, a close examination of this boundary is very useful for bug isolation, sanity checking and attempting to capture race conditions.

From: man strace

Method B: Using an isolated network namespace and Wireshark

Please note that this method might not work for all kernels.
It was tested on Fedora 25 (64 bit) with success.

Create the test network namespace

A network namespace is logically another copy of the network stack, with its own routes, firewall rules, and network devices.
With the following command we will create a network namespace called test.


sudo ip netns add test;

ip netns add NAME Creates a new named network namespace.
If NAME is available in /var/run/netns/ this command creates a new network namespace and assigns to it then name NAME.

Create two virtual network interfaces (veth0 and veth1) for our Virtual eXtended LAN (VXLAN)

The following command will create veth0 and veth1 virtual network interfaces.


sudo ip link add veth0 type veth peer name veth1;

ip link add with no link argument specified to a physical device to operate on, it adds a VXLAN virtual link.
Note: veth1 will act as a gateway later on.

Change the active namespace of the veth0 interface

With the following command, we move veth0 to our test network namespace.


sudo ip link set veth0 netns test;

Some devices are not allowed to change network namespace: loopback, bridge, ppp, wireless. These are network namespace local devices. In such case ip tool will return Invalid argument error. It is possible to find out if device is local to a single network namespace by checking netns-local flag in the output of the ethtool:
ethtool -k DEVICE;
To change network namespace for wireless devices the iw tool can be used. But it allows to change network namespace only for physical devices and by process PID.
From man ip-link

Configure the IP addresses of the virtual interfaces


#Set the IP of veth0 to 192.168.10.1 and veth0 to 192.168.10.254
sudo ip netns exec test ifconfig veth0 up 192.168.10.1 netmask 255.255.255.0;
sudo ifconfig veth1 up 192.168.10.254 netmask 255.255.255.0;

Configure the routing in the test namespace

The following command will set the default gateway for veth0 to be the IP 192.168.10.254 which is the address we gave veth1 in the previous step.


sudo ip netns exec test route add default gw 192.168.10.254 dev veth0;

Then we make sure ip_forward is active by issuing the following command


sudo sh -c 'echo 1 > /proc/sys/net/ipv4/ip_forward';

Then, we establish a NAT rule to forward all the traffic of test network namespace to a physical device


sudo iptables -t nat -A POSTROUTING -s 192.168.10.0/24 -o <PHYSICAL DEVICE e.g. eth0> -j SNAT --to-source <PHYSICAL DEVICE IP>;

Actually using the isolated network namespace

Add root to the list of users that is allowed to start an X application


xhost +si:localuser:root;

Start wireshark in your test network namespace


sudo ip netns exec test wireshark &

In wireshark start monitoring the data on the device veth0.

Finally, start the application you wish to monitor its network traffic


sudo ip netns exec test firefox;


How to get the pid of the last executed command that was sent to the background in a bash shell

Recently we came to the need of writing a bash script that needed periodically to check if a specific process, that was started by the script, had ended and restart it (something like watchdog but with not so many features).

To achieve this, we used the one of the shell special parameters, the $!. Like all other special parameters $! may only be referenced and the user cannot make an assignment to it.

($!) Expands to the process ID of the job most recently placed into the background, whether executed as an asynchronous command or using the bg builtin command.

From GNU.org: https://www.gnu.org/software/bash/manual/html_node/Special-Parameters.html#index-_0021-1

Example of Usage

In this example we wanted to get the PID of the application called server to be used later on in the script.


server &
echo $!; #This will print the process ID of the 'server' application


How to set a static IP Address from the Command Line in GNU/Linux using ip addr and ip route

Assuming you want to make the following changes to the network device eth0

  1. Change the IP to the static value 192.168.1.2
  2. Set the Subnet Mask to 255.255.255.0
  3. Set the Default Gateway for the device to be 192.168.1.1

and you want to avoid using ifconfig and route that are obsolete you can perform these changes using the following two commands


sudo ip addr add 192.168.1.2/24 dev eth0;
sudo ip route add default via 192.168.1.1 dev eth0;

Please note that the netmask is given in CIDR notation (it is the /24 right after the IP of the device in the ip addr command).

A subnet mask (netmask) is a bitmask that encodes the prefix length in quad-dotted notation: 32 bits, starting with a number of 1 bits equal to the prefix length, ending with 0 bits, and encoded in four-part dotted-decimal format: 255.255.255.0. A subnet mask encodes the same information as a prefix length, but predates the advent of CIDR. In CIDR notation, the prefix bits are always contiguous, whereas subnet masks may specify non-contiguous bits.

From Wikipedia: https://en.wikipedia.org/wiki/Classless_Inter-Domain_Routing