Digital Innovation with APIs

The CCS is organizing the following Open Seminar on Current International Technology Trends:

Presentation Title: Digital Innovation with APIs
Speaker: Saad Syed, Google
Date/Time: June 28th 2017 , 17:00-18:30 (16:30-17:00 Welcome Coffee & Registration )
Place: European University Cyprus, Room 208, 2nd, East Block

Free Admission – Registration is required
Digital Innovation with APIs -BROCHURE.pdf (33 downloads)

Open Seminars On Current International Technology Trends Series: Digital Innovation With APIs

APIs are the foundation for many Digital Native applications. Find out how this important capability will allow you to be more agile, build innovative new products/services and leverage the power of collaboration beyond your organization.

Saad Syed is the Regional Lead for Google Edge Southern EMEA. He is responsible for the Apigee API Management solution set in the region. He has worked extensively with many Leading institutions in the region on their Digital Transformation initiatives. Prior to Google Saad was the Regional Leader for Data Center at Cisco Systems for Middle East Turkey Russia and Africa.

Seminar Target Group

  • Innovators
  • Developers
  • Engineers
  • ICT Companies
  • Telecom Companies
  • Banks
  • Insurance Companies
  • Students And Professors

Free Admission – Registration is required
Digital Innovation with APIs -BROCHURE.pdf (33 downloads)


Ubuntu: Headless wireshark (or wireshark from terminal)

Recently, we wanted to use wireshark on an Ubuntu through ssh and no X-Server forwarding enabled.
After a quick search we found tshark.

TShark is a network protocol analyzer. It lets you capture packet data from a live network, or read packets from a previously saved capture file, either printing a decoded form of those packets to the standard output or writing the packets to a file. TShark‘s native capture file format is pcap format, which is also the format used by tcpdump and various other tools.
Without any options set, TShark will work much like tcpdump. It will use the pcap library to capture traffic from the first available network interface and displays a summary line on stdout for each received packet.
TShark is able to detect, read and write the same capture files that are supported by Wireshark.

From: man tshark

Install tshark on Ubuntu


sudo apt-get install tshark -y;

Using tshark to capture all traffic on eth0 to a pcap file


sudo tshark -i eth0 -w something.pcap;

Note: If you just want to capture network traffic on a network interface and not use the additional features wireshark has to offer, you can also use tcpdumpas follows


#The following command will create a files that has in its name the current date and time using the date function.
sudo tcpdump -i eth0 -w "data.`date +%Y-%m-%d\ %H.%M`.pcap";


Bash: Remove the last character from each line

The following script, uses rev and cut to remove the last character from each line in a pipe.
rev utility reverses lines character-wise.
cut removes sections  from each of line.
It is a very simple script where we reverse the line once, remove the first character (which was the last one in the original form of the line) and finally we reverse the line back with the last character missing.


echo -e "hi\nHI" | rev | cut -c 2- | rev;

# Will produce:
h
H

 


Bash: Switch positions between all characters in odd positions with characters in even positions

The following awk script allowed us to switch position of all characters placed in odd numbered positions with their next neighboring even numbered position characters.
In detail what it does is to create a for loop that skips one character every time and then it prints each pair in reverse order (it will print the second character first, then the first one, then the fourth and so on).


echo "123456789" | awk -vFS= '{for (i = 1; i <= NF; i+=2) {printf $(i+1)$i""} printf "\n"}';

# Will produce 214365879

echo "1234567890" | awk -vFS= '{for (i = 1; i <= NF; i+=2) {printf $(i+1)$i""} printf "\n"}';

# Will produce 2143658709

Please note that we set the built-in variable FS (The input field separator which is a space by default) to the empty string so that each character is treated like a different field by NF (The number of fields in the current input record).

 


Bash: Print time stamp in front of every line in a pipe

Recently, we received a binary that collected data from a web service and it printed them on screen.
The binary did not print a time stamp in front of each line so we had to improvise of a way to add the time stamp to the logs without modifying the binary.

The solution we came to was to use awk to prepend the time stamp in front of every line using a pipe.
Specifically, our solution was the following:


server_application 2>&1 | awk '{ print strftime("%Y-%m-%d %H:%M:%S"), $0; fflush(); }'

What we did there was to start our binary server_application, redirect stderr to stdout (using 2>&1) so that we will have only one stream and then we read the lines one by one using awk and printed the time stamp right before the line ($0) using strftime.
The strftime() function formats the broken-down time according to the format specification format.
fflushforces a write of all user-space buffered data for the given output or update stream via the stream’s underlying write function. We call it at each line to make sure that we do not cause additional delay in presenting the data due to buffering limitations caused by our prints.

Example


$ echo -e "hi\nHI" 2>&1 | awk '{ print strftime("%Y-%m-%d %H:%M:%S"), $0; fflush(); }'
2017-06-21 20:33:41 hi
2017-06-21 20:33:41 HI


How to execute `find` that ignores .git directories

Trying to find a source code file by its content using find and -exec grep, can some times result in getting results from the repository .git folders as well.

This behavior not only does it provide results you do not need but it also makes your search slower.
Below, we propose a couple of solutions on how to make a more efficient search.

Example 1: Ignore all .git folders no matter where they are in the search path

For find to ignore all .git folders, even if they appear on the first level of directories or any in-between until the last one, add -not -path '*/\.git*' to your command as in the example below.
This parameter will instruct find to filter out any file that has anywhere in its path the folder .git. This is very helpful in case a project has dependencies in other projects (repositories) that are part of the internal structure.


find . -type f -not -path '*/\.git/*';

Note, if you are using svn use:


find . -type f -not -path '*/\.svn/*';

Example 2: Ignore all hidden files and folders

To ignore all hidden files and folders from your find results add -not -path '*/\.*' to your command.


find . -not -path '*/\.*';

This parameter instructs find to ignore any file that has anywhere in its path the string /. which is any hidden file or folder in the search path!


How to capture all network traffic of a single process

Method A: Using strace

If the process is not started yet, you can start it as a new process and monitor it using the following


strace -f -e trace=network -s 10000 <PROCESS WITH ARGUMENTS>;

If the process is already started and you know its PID you can use the following


strace -f -e trace=network -s 10000 -p <PID>;

strace is a very helpful utility that can be used to trace system calls and signals.

Parameters Legend:

  • -f Instructs strace to trace all child processes as they are created by the currently traced processes as a result of the fork, vfork and clone system calls.
    Note that -p PID -f will attach all threads of process PID if it is multi-threaded, not only thread with id PID.
  • -e trace=%network strace will trace all the network related system calls only if used alone.
  • -s strsize Specifies the maximum string size to print (the default is 32). Note that filenames are not considered strings and are always printed in full.
  • -p PID Attaches strace to the process with the process ID PID and starts tracing. The trace may be terminated at any time by a keyboard interrupt signal (CTRL-C). strace will respond by detaching itself from the traced process(es) leaving it (them) to continue running. Multiple -p options can be used to attach to many processes in addition to command (which is optional if at least one -p option is given). -p "`pidof APPLICATION`" syntax is supported.

In the simplest case strace runs the specified command until it exits. It intercepts and records the system calls which are called by a process and the signals which are received by a process. The name of each system call, its arguments and its return value are printed on standard error or to the file specified with the -o option.
strace is a useful diagnostic, instructional, and debugging tool. System administrators, diagnosticians and trouble-shooters will find it invaluable for solving problems with programs for which the source is not readily available since they do not need to be recompiled in order to trace them. Students, hackers and the overly-curious will find that a great deal can be learned about a system and its system calls by tracing even ordinary programs. And programmers will find that since system calls and signals are events that happen at the user/kernel inter‐ face, a close examination of this boundary is very useful for bug isolation, sanity checking and attempting to capture race conditions.

From: man strace

Method B: Using an isolated network namespace and Wireshark

Please note that this method might not work for all kernels.
It was tested on Fedora 25 (64 bit) with success.

Create the test network namespace

A network namespace is logically another copy of the network stack, with its own routes, firewall rules, and network devices.
With the following command we will create a network namespace called test.


sudo ip netns add test;

ip netns add NAME Creates a new named network namespace.
If NAME is available in /var/run/netns/ this command creates a new network namespace and assigns to it then name NAME.

Create two virtual network interfaces (veth0 and veth1) for our Virtual eXtended LAN (VXLAN)

The following command will create veth0 and veth1 virtual network interfaces.


sudo ip link add veth0 type veth peer name veth1;

ip link add with no link argument specified to a physical device to operate on, it adds a VXLAN virtual link.
Note: veth1 will act as a gateway later on.

Change the active namespace of the veth0 interface

With the following command, we move veth0 to our test network namespace.


sudo ip link set veth0 netns test;

Some devices are not allowed to change network namespace: loopback, bridge, ppp, wireless. These are network namespace local devices. In such case ip tool will return Invalid argument error. It is possible to find out if device is local to a single network namespace by checking netns-local flag in the output of the ethtool:
ethtool -k DEVICE;
To change network namespace for wireless devices the iw tool can be used. But it allows to change network namespace only for physical devices and by process PID.
From man ip-link

Configure the IP addresses of the virtual interfaces


#Set the IP of veth0 to 192.168.10.1 and veth0 to 192.168.10.254
sudo ip netns exec test ifconfig veth0 up 192.168.10.1 netmask 255.255.255.0;
sudo ifconfig veth1 up 192.168.10.254 netmask 255.255.255.0;

Configure the routing in the test namespace

The following command will set the default gateway for veth0 to be the IP 192.168.10.254 which is the address we gave veth1 in the previous step.


sudo ip netns exec test route add default gw 192.168.10.254 dev veth0;

Then we make sure ip_forward is active by issuing the following command


sudo sh -c 'echo 1 > /proc/sys/net/ipv4/ip_forward';

Then, we establish a NAT rule to forward all the traffic of test network namespace to a physical device


sudo iptables -t nat -A POSTROUTING -s 192.168.10.0/24 -o <PHYSICAL DEVICE e.g. eth0> -j SNAT --to-source <PHYSICAL DEVICE IP>;

Actually using the isolated network namespace

Add root to the list of users that is allowed to start an X application


xhost +si:localuser:root;

Start wireshark in your test network namespace


sudo ip netns exec test wireshark &

In wireshark start monitoring the data on the device veth0.

Finally, start the application you wish to monitor its network traffic


sudo ip netns exec test firefox;


How to undo a Git commit that was not pushed

To undo a Git commit that was not pushed, you are given a few major options:

  1. Undo the commit but keep all changes staged
  2. Undo the commit and unstage the changes
  3. Undo the commit and lose all changes

Method 1: Undo commit and keep all files staged

In case you just want to undo the commit and change nothing more, you can use


git reset --soft HEAD~;

This is most often used to make a few changes to your latest commit and/or fix your commit message. Leaves working tree as it was before reset.
soft does not touch the index file or the working tree at all (but resets the head to the previous commit). This leaves all your changed files Changes to be committed, as git status would put it.

Method 2: Undo commit and unstage all files

In case you want to undo the last commit and unstage all the files you can use the following


git reset HEAD~;

or


git reset --mixed HEAD~;

mixed will reset the index but not the working tree (i.e., the changed files are preserved but not marked for commit) and reports what has not been updated. This is the default action.

Method 3: Undo the commit and completely remove all changes

The following method will undo the commit and revert all changes so that your state is exactly as it was before you started making changes.


git reset --hard HEAD~;

hard resets the index and working tree. Any changes to tracked files in the working tree since the previous commit are discarded.

 

Note: In case you just want to rewrite the commit message, you could use git –amend instead.


How to get the pid of the last executed command that was sent to the background in a bash shell

Recently we came to the need of writing a bash script that needed periodically to check if a specific process, that was started by the script, had ended and restart it (something like watchdog but with not so many features).

To achieve this, we used the one of the shell special parameters, the $!. Like all other special parameters $! may only be referenced and the user cannot make an assignment to it.

($!) Expands to the process ID of the job most recently placed into the background, whether executed as an asynchronous command or using the bg builtin command.

From GNU.org: https://www.gnu.org/software/bash/manual/html_node/Special-Parameters.html#index-_0021-1

Example of Usage

In this example we wanted to get the PID of the application called server to be used later on in the script.


server &
echo $!; #This will print the process ID of the 'server' application


C: Implicit declaration of function ‘read’ and ‘write’

While working on an socket-based application, we received the following warnings from the compiler:

implicit declaration of function 'read'
implicit declaration of function 'write'

read and write functions are declared in unistd.h which we forgot to include in our code.

Adding the directive


#include <unistd.h>

to the source file that used read and/or write removed the warnings.