Monthly Archives: February 2017


Find which Ports are listening on Linux using netstat

netstat prints network connections, routing tables, interface statistics, masquerade connections, and multicast memberships.

Using the parameter -l (or --listening) it will show only listening sockets/ports (which are omitted by default.).
--numeric-ports shows numerical port numbers but does not affect the resolution of host or user names (e.g. instead of showing the name ssh, it will show the value 22).

We used netstat using the following syntax to check which sockets/ports are open on the current machine:

netstat --listening --numeric-ports;

The results appeared as follows:

[[email protected] ~]$ netstat --listening --numeric-ports
 Active Internet connections (only servers)
 Proto Recv-Q Send-Q Local Address           Foreign Address         State      
 tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN     
 tcp        0      0 localhost:25            0.0.0.0:*               LISTEN     
 tcp6       0      0 [::]:44300              [::]:*                  LISTEN     
 tcp6       0      0 [::]:8080               [::]:*                  LISTEN     
 tcp6       0      0 [::]:22                 [::]:*                  LISTEN     
 tcp6       0      0 localhost:25            [::]:*                  LISTEN     
 udp        0      0 0.0.0.0:39925           0.0.0.0:*                          
 udp        0      0 0.0.0.0:24186           0.0.0.0:*                          
 udp        0      0 0.0.0.0:68              0.0.0.0:*                          
 udp        0      0 localhost:323           0.0.0.0:*                          
 udp        0      0 0.0.0.0:5353            0.0.0.0:*                          
 udp6       0      0 localhost:323           [::]:*                             
 udp6       0      0 [::]:33848              [::]:*                             
 udp6       0      0 [::]:61453              [::]:*                             
 raw6       0      0 [::]:58                 [::]:*                  7          
 Active UNIX domain sockets (only servers)
 Proto RefCnt Flags       Type       State         I-Node   Path
 unix  2      [ ACC ]     STREAM     LISTENING     22489    public/showq
 unix  2      [ ACC ]     STREAM     LISTENING     22445    public/pickup
 unix  2      [ ACC ]     STREAM     LISTENING     22449    public/cleanup
 unix  2      [ ACC ]     STREAM     LISTENING     22477    private/proxymap
 unix  2      [ ACC ]     STREAM     LISTENING     22480    private/proxywrite
 unix  2      [ ACC ]     STREAM     LISTENING     15452    /run/systemd/private
 unix  2      [ ACC ]     STREAM     LISTENING     22483    private/smtp
 unix  2      [ ACC ]     STREAM     LISTENING     22486    private/relay
 unix  2      [ ACC ]     STREAM     LISTENING     22492    private/error
 unix  2      [ ACC ]     STREAM     LISTENING     22495    private/retry
 unix  2      [ ACC ]     STREAM     LISTENING     22498    private/discard
 unix  2      [ ACC ]     STREAM     LISTENING     22501    private/local
 unix  2      [ ACC ]     STREAM     LISTENING     22504    private/virtual
 unix  2      [ ACC ]     STREAM     LISTENING     22507    private/lmtp
 unix  2      [ ACC ]     STREAM     LISTENING     22510    private/anvil
 unix  2      [ ACC ]     STREAM     LISTENING     22513    private/scache
 unix  2      [ ACC ]     STREAM     LISTENING     14445    /var/run/NetworkManager/private-dhcp
 unix  2      [ ACC ]     SEQPACKET  LISTENING     15476    /run/udev/control
 unix  2      [ ACC ]     STREAM     LISTENING     1404     /run/systemd/journal/stdout
 unix  2      [ ACC ]     STREAM     LISTENING     22452    public/qmgr
 unix  2      [ ACC ]     STREAM     LISTENING     15498    /run/lvm/lvmpolld.socket
 unix  2      [ ACC ]     STREAM     LISTENING     22474    public/flush
 unix  2      [ ACC ]     STREAM     LISTENING     22471    private/verify
 unix  2      [ ACC ]     STREAM     LISTENING     16034    /var/run/dbus/system_bus_socket
 unix  2      [ ACC ]     STREAM     LISTENING     16037    /var/run/avahi-daemon/socket
 unix  2      [ ACC ]     STREAM     LISTENING     15537    /run/lvm/lvmetad.socket
 unix  2      [ ACC ]     STREAM     LISTENING     22456    private/tlsmgr
 unix  2      [ ACC ]     STREAM     LISTENING     22459    private/rewrite
 unix  2      [ ACC ]     STREAM     LISTENING     22462    private/bounce
 unix  2      [ ACC ]     STREAM     LISTENING     22465    private/defer
 unix  2      [ ACC ]     STREAM     LISTENING     22468    private/trace

Check a specific port if it is open from a remote machine

In case you want to check a specific port if it is open from a remote machine, you can use nmap.
Using nmap to scan specific ports allows you to check if a remote machine appears to have open ports available to you.
nmap is a network exploration tool and security / port scanner.

The following example checks ports 80 and 8080 on 192.168.1.199 if they are open.

[[email protected] ~]$ nmap -vv -p 80,8080 192.168.1.199
 
 Starting Nmap 6.40 ( http://nmap.org ) at 2017-02-22 14:10 EET
 Initiating Ping Scan at 14:10
 Scanning 192.168.1.199 [2 ports]
 Completed Ping Scan at 14:10, 0.00s elapsed (1 total hosts)
 Initiating Parallel DNS resolution of 1 host. at 14:10
 Completed Parallel DNS resolution of 1 host. at 14:10, 0.00s elapsed
 Initiating Connect Scan at 14:10
 Scanning 192.168.1.199 [2 ports]
 Discovered open port 8080/tcp on 192.168.1.199
 Completed Connect Scan at 14:10, 0.00s elapsed (2 total ports)
 Nmap scan report for 192.168.1.199
 Host is up (0.000060s latency).
 Scanned at 2017-02-22 14:10:29 EET for 0s
 PORT     STATE  SERVICE
 80/tcp   closed http
 8080/tcp open   http-proxy
 
 Read data files from: /usr/bin/../share/nmap
 Nmap done: 1 IP address (1 host up) scanned in 0.03 seconds

The -vv parameter for nmap increases the verbosity of the results.
The -p parameter defines the ports to be checked.

Advertisements

About restoring a deleted Git branch

Recently, a branch was deleted from the server without it being merged.
Luckily for us, we had a local copy.

We used used the command git reflog to get access to the reference logs of the branch.
The command returned results similar to the below:

271f0084 [email protected]{0}: pull: Merge made by the 'recursive' strategy.
 0e71c0b [email protected]{1}: commit: Minor change: Cracked NSA systems
 c81624b [email protected]{2}: pull: Fast-forward
 ef1f281 [email protected]{3}: commit (amend): Deployed satelite
 1ed1a5a [email protected]{4}: commit (amend): Deployed satelite
 a8682cb [email protected]{5}: commit (amend): Deployed satelite
 e4560c8 [email protected]{6}: commit (amend): Deployed satelite
 1679a90 [email protected]{7}: commit (amend): Deployed satelite
 d27d2c9 [email protected]{8}: commit: Deployed satelite
 aaf8261 [email protected]{9}: checkout: moving from master to crack-nsa-systems
 2500e11 [email protected]{10}: clone: from ssh://[email protected]:7999/secret/bananas.git

From this information we got the hash value in front of the commit which we wanted to use to restore, which was 271f0084.

Then, we checked out that version using

git checkout 271f0084;

When we tried to push the branch back to the server the Git pre commit hooks blocked the operation.
Based on the rsa key used, we could only submit changes that were committed by the owner of that key.
The option to disable temporarily the pre commit hooks was unfortunately out of the question.
So we had to replace all author names and emails with the name of the one holding the key.

To do so we used the following command before pushing to the server once more:

git filter-branch --commit-filter '
   if [ "$GIT_COMMITTER_NAME" = "Doe, John" ];
   then
     GIT_COMMITTER_NAME="Squarepants, Bob";
     GIT_AUTHOR_NAME="Squarepants, Bob";
     GIT_COMMITTER_EMAIL="[email protected]";
     GIT_AUTHOR_EMAIL="[email protected]";
     git commit-tree "[email protected]";
   else
     git commit-tree "[email protected]";
   fi' HEAD

git filter-branch lets you rewrite Git revision history by rewriting the branches mentioned, in our case it was HEAD, applying custom filters on each revision. Those filters can modify each tree (e.g. removing a file or running a perl rewrite on all files) or information about each commit. Otherwise, all information (including original commit times or merge information) will be preserved.

HEAD is a reference to the currently checked out commit. In normal states, it’s actually a symbolic reference to the branch you have checked out.
Looking at the contents of .git/HEAD you’ll see something similar to ref: refs/heads/master.
The branch itself is a reference to the commit at the tip of the branch.


Rough notes for DJI Phantom 2 Vision +

After you connect to the Wi-Fi hot spot of the device (SSID: Phantom_XXXXXX):

System 1

ssh [email protected]
#Password for root is 19881209
[email protected]'s password: 


BusyBox v1.19.4 (2013-04-22 22:41:57 CST) built-in shell (ash)
Enter 'help' for a list of built-in commands.

 -----------------------------------------------------
 DJI-INNOVATIONS
 -----------------------------------------------------
  * FC200-Vision+        
  * [email protected]
  * 02/20/2014
  * Version 1.01
  * ART / DHCP down / ping
 -----------------------------------------------------



System is based on OpenWrt.

System 2

ssh [email protected]
#Password for root is 19881209
[email protected]'s password: 


BusyBox v1.19.4 (2013-04-22 22:41:57 CST) built-in shell (ash)
Enter 'help' for a list of built-in commands.

 -----------------------------------------------------
 DJI-INNOVATIONS
 -----------------------------------------------------
  * FC200-Vision+ RE   
  * [email protected]
  * 02/21/2014
  * Version 1.01
  * ART / DHCP down / ping
 -----------------------------------------------------

ssh -L 8080:downloads.openwrt.org:80 [email protected]

cat /etc/opkg.conf 
src/gz barrier_breaker http://downloads.openwrt.org/snapshots/trunk/ar71xx/packages
dest root /
dest ram /tmp
lists_dir ext /var/opkg-lists
option overlay_root /overlay

cat /etc/opkg.conf 
src/gz barrier_breaker http://downloads.openwrt.org/barrier_breaker/14.07/ar71xx/generic/packages/base
dest root /
dest ram /tmp
lists_dir ext /var/opkg-lists
option overlay_root /overlay
option http_proxy http://127.0.0.1:8080/

[email protected]:~# opkg update
Downloading http://downloads.openwrt.org/barrier_breaker/14.07/ar71xx/generic/packages/base/Packages.gz.
Updated list of available packages in /var/opkg-lists/barrier_breaker.
[email protected]:~# opkg install tcpdump-mini
Installing tcpdump-mini (4.5.1-4) to root...
Downloading http://downloads.openwrt.org/barrier_breaker/14.07/ar71xx/generic/packages/base/tcpdump-mini_4.5.1-4_ar71xx.ipk.
Installing libpcap (1.5.3-1) to root...
Downloading http://downloads.openwrt.org/barrier_breaker/14.07/ar71xx/generic/packages/base/libpcap_1.5.3-1_ar71xx.ipk.
Configuring libpcap.
Configuring tcpdump-mini.




BAD TRUNC
opkg install tcpdump-mini
Installing tcpdump-mini (4.5.1-4) to root...
Downloading http://downloads.openwrt.org/snapshots/trunk/ar71xx/generic/packages/base/tcpdump-mini_4.5.1-4_ar71xx.ipk.
Installing libpcap (1.7.4-1) to root...
Downloading http://downloads.openwrt.org/snapshots/trunk/ar71xx/generic/packages/base/libpcap_1.7.4-1_ar71xx.ipk.
Configuring libpcap.
//usr/lib/opkg/info/libpcap.postinst: line 4: default_postinst: not found
Configuring tcpdump-mini.
//usr/lib/opkg/info/tcpdump-mini.postinst: line 4: default_postinst: not found
Collected errors:
 * pkg_run_script: package "libpcap" postinst script returned status 127.
 * opkg_configure: libpcap.postinst returned 127.
 * pkg_run_script: package "tcpdump-mini" postinst script returned status 127.
 * opkg_configure: tcpdump-mini.postinst returned 127.


From host machine:
ssh [email protected] tcpdump -i br-lan -s0 -w - port not 22 > dji-wifi-traffic.pcap

ssh [email protected] tcpdump -n -i br-lan -s0 -w - port 2001 > dji.2001.pcap


ip.src==192.168.1.1 || ip.dst==192.168.1.1 ||ip.src==192.168.1.2 || ip.dst==192.168.1.2 || ip.src==192.168.1.10 || ip.dst==192.168.1.10

System is based on OpenWrt.


C Bit Fields: Full example code

The following two examples demonstrate the use of bit fields to reduce memory consumption of certain applications.

In the first example we create a compressed ‘bit’ struct and on the second we create a weird struct representation for bytes to show that the size of that struct is significantly less that the original one.

First example: Using bit fields to create a ‘bit’ structure

bits_bit_fields.c (compressed) (170 downloads)
#include <stdio.h>

// Weird structure to represent a 'bit'
typedef struct {
    unsigned char value;
} bit;

// 'bit' structure using bit fields
typedef struct {
    unsigned char value : 1;
} bit_bit_field;

int main( ) {

    printf( "Memory size occupied by 'bit' struct : %zu bytes\n", sizeof(bit));
    printf( "Memory size occupied by 'bit_bit_field' struct : %zu bytes\n", sizeof(bit_bit_field));

    bit bits[8];
    bit_bit_field bits_bit_field[8];

    printf( "Memory size occupied by 'bits' array : %zu bytes\n", sizeof(bits));
    printf( "Memory size occupied by 'bits_bit_field' array : %zu bytes\n", sizeof(bits_bit_field));

    // Setting the value of the first 'bit' and then printing it.
    // We will use various values for this test to show that when you set a value to a bit field
    // that is greater that the allowed size it will fill it using the last bits only.
    // It will not spill data though to neighbouring 'bits'.
    unsigned char value;
    for (value = 0; value < 4; value++)
    {
        printf("Input Value: %d\n", value);
        int bits_i;
        const int bits_length = (sizeof(bits) / sizeof(bit));
        for (bits_i = 0; bits_i < bits_length; bits_i++)
        {
            if (bits_i % 2)
            {
                bits[bits_i].value = 0;
            }
            else
            {
                bits[bits_i].value = value;
            }
            printf("%d", bits[bits_i].value);
        }
        printf("\n");

        int bits_bit_field_i;
        const int bits_bit_field_length = (sizeof(bits_bit_field) / sizeof(bit_bit_field));
        for (bits_bit_field_i = 0; bits_bit_field_i < bits_bit_field_length; bits_bit_field_i++)
        {
            if (bits_bit_field_i % 2)
            {
                bits_bit_field[bits_bit_field_i].value = 0;
            }
            else
            {
                bits_bit_field[bits_bit_field_i].value = value;
            }
            printf("%d", bits_bit_field[bits_bit_field_i].value);
        }
        printf("\n");
    }
    return 0;
}
bits_bit_fields.c (compressed) (170 downloads)

Execution output

Memory size occupied by 'bit' struct : 1 bytes
Memory size occupied by 'bit_bit_field' struct : 1 bytes
Memory size occupied by 'bits' array : 8 bytes
Memory size occupied by 'bits_bit_field' array : 8 bytes
Input Value: 0
00000000
00000000
Input Value: 1
10101010
10101010
Input Value: 2
20202020
00000000
Input Value: 3
30303030
10101010

Second example: Using bit fields to create a ‘byte’ structure where each ‘bit’ is another named member

bytes_bit_fields.c (compressed) (165 downloads)
#include <stdio.h>

// Weird structure to represent a 'byte'
typedef struct {
    unsigned char bit_0;
    unsigned char bit_1;
    unsigned char bit_2;
    unsigned char bit_3;
    unsigned char bit_4;
    unsigned char bit_5;
    unsigned char bit_6;
    unsigned char bit_7;
} byte;

// 'byte' structure using bit fields
// Unfortunately we cannot declare an array where the values are bit fields,
// so we have to declare each member separately.
// We instruct the compiler to use only one bit per element.
typedef struct {
    unsigned char bit_0 : 1;
    unsigned char bit_1 : 1;
    unsigned char bit_2 : 1;
    unsigned char bit_3 : 1;
    unsigned char bit_4 : 1;
    unsigned char bit_5 : 1;
    unsigned char bit_6 : 1;
    unsigned char bit_7 : 1;
} byte_bit_field;

int main( ) {

    printf( "Memory size occupied by 'byte' struct : %zu bytes\n", sizeof(byte));
    printf( "Memory size occupied by 'byte_bit_field' struct : %zu bytes\n", sizeof(byte_bit_field));

    byte bytes[8];
    byte_bit_field bytes_bit_field[8];

    printf( "Memory size occupied by 'bytes' array : %zu bytes\n", sizeof(bytes));
    printf( "Memory size occupied by 'bytes_bit_field' array : %zu bytes\n", sizeof(bytes_bit_field));

    // Setting the value of the first 'bit' and then printing it.
    // We will use various values for this test to show that when you set a value to a bit field
    // that is greater that the allowed size it will fill it using the last bits only.
    // It will not spill data though to neighbouring 'bits'.
    unsigned char value;
    for (value = 0; value < 4; value++)
    {
        printf("Input Value: %d\n", value);
        int bytes_i;
        const int bytes_length = (sizeof(bytes) / sizeof(byte));
        for (bytes_i = 0; bytes_i < bytes_length; bytes_i++)
        {
            if (bytes_i % 2)
            {
                bytes[bytes_i].bit_3 = 0;
            }
            else
            {
                bytes[bytes_i].bit_3 = value;
            }
            printf(" %d  ", bytes[bytes_i].bit_3);
        }
        printf("\n");

        int bytes_bit_field_i;
        const int bytes_bit_field_length = (sizeof(bytes_bit_field) / sizeof(byte_bit_field));
        for (bytes_bit_field_i = 0; bytes_bit_field_i < bytes_bit_field_length; bytes_bit_field_i++)
        {
            if (bytes_bit_field_i % 2)
            {
                bytes_bit_field[bytes_bit_field_i].bit_3 = 0;
            }
            else
            {
                bytes_bit_field[bytes_bit_field_i].bit_3 = value;
            }
            printf("%d%d%d ",
                   bytes_bit_field[bytes_bit_field_i].bit_2,
                   bytes_bit_field[bytes_bit_field_i].bit_3,
                   bytes_bit_field[bytes_bit_field_i].bit_4);
        }
        printf("\n");
    }
    return 0;
}
bytes_bit_fields.c (compressed) (165 downloads)

Execution output

Memory size occupied by 'byte' struct : 8 bytes
Memory size occupied by 'byte_bit_field' struct : 1 bytes
Memory size occupied by 'bytes' array : 64 bytes
Memory size occupied by 'bytes_bit_field' array : 8 bytes
Input Value: 0
 0   0   0   0   0   0   0   0  
000 000 000 000 000 000 000 000 
Input Value: 1
 1   0   1   0   1   0   1   0  
010 000 010 000 010 000 010 000 
Input Value: 2
 2   0   2   0   2   0   2   0  
000 000 000 000 000 000 000 000 
Input Value: 3
 3   0   3   0   3   0   3   0  
010 000 010 000 010 000 010 000

grep: How to match lines using any of multiple patterns

Recently, we needed to filter the results of ps x using two different patterns.
The first pattern was ./ where we needed to match that exact character sequence.
The . period character is treated as a special character in regular expressions (it matches a single character of any value, except for the end of line), so we decided to use the -F parameter to remove this special handling.
Doing this change prevented us from writing a regular expression that uses the OR | operator.

-F (or --fixed-strings) is a matching control option that instructs grep to interpret the patterns as a list of fixed strings (instead of regular expressions), separated by newlines, any of which is to be matched.
We tried assigning the different patterns as different lines to a variable and then using them on the pipe, like in the following example:

patterns="./
banana";
ps x | grep -F $patterns;

..but it failed.

Solution

grep supports a matching control option -e that allows us to define multiple patterns using different strings.

-e PATTERN (or --regexp=PATTERN) uses the value PATTERN as the pattern. If this option is used  multiple times or it is combined with the -f (--file) option, grep will search for all patterns given.

In the end, our command was transformed to the following, which worked just fine!

ps x | grep -F -e "./" -e "banana";