Monthly Archives: February 2017


Online Qualification Round Problem for Google Hash Code 2017

Online Qualification Round Problem for Google Hash Code 2017 - Problem Statement (91 downloads)

Online Qualification Round Problem for Google Hash Code 2017 - Data Sets (79 downloads)

Streaming videos

Problem statement for Online Qualification Round, Hash Code 2017

Introduction

Have you ever wondered what happens behind the scenes when you watch a YouTube video? As more and more people watch online videos (and as the size of these videos increases), it is critical that video-serving infrastructure is optimized to handle requests reliably and quickly.

This typically involves putting in place cache servers, which store copies of popular videos. When a user request for a particular video arrives, it can be handled by a cache server close to the user, rather than by a remote data center thousands of kilometers away.

But how should you decide which videos to put in which cache servers?

Task

Given a description of cache servers, network endpoints and videos, along with predicted requests for individual videos, decide which videos to put in which cache server in order to minimize the average waiting time for all requests.

Problem description

The picture below represents the video serving network.

Videos

Each video has a size given in megabytes (MB). The data center stores ​ all videos​ . Additionally, each video can be put in 0, 1, or more cache servers​. Each cache server has a maximum capacity given in megabytes.

Endpoints

Each endpoint represents a group of users connecting to the Internet in the same geographical area (for example, a neighborhood in a city). Every endpoint is connected to the data center. Additionally, each endpoint may (but doesn’t have to) be connected to 1 or more cache servers​ .

Each endpoint is characterized by the latency of its connection to the data center (how long it takes to serve a video from the data center to a user in this endpoint), and by the latencies to each cache server that the endpoint is connected to (how long it takes to serve a video stored in the given cache server to a user in this endpoint).

Requests

The predicted requests provide data on how many times a particular video is requested from a particular endpoint.

Input data set

The input data is provided as a data set file – a plain text file containing exclusively ASCII characters with a single \n character at the end of each line (UNIX-​ style line endings).

Videos, endpoints and cache servers are referenced by integer IDs. There are V videos numbered from 0 to V − 1 , E endpoints numbered from 0 to E − 1 and C cache servers numbered from 0 to C − 1 .

File format

All numbers mentioned in the specification are natural numbers that fit within the indicated ranges. When multiple numbers appear in a single line, they are separated by a single space.

The first line of the input contains the following numbers:

  • V ( 1 ≤ V ≤ 10000) – the number of videos
  • E ( 1 ≤ E ≤ 1000) – the number of endpoints
  • R ( 1 ≤ R ≤ 1000000) – the number of request descriptions
  • C ( 1 ≤ C ≤ 1000) – the number of cache servers
  • X ( 1 ≤ X ≤ 500000) – the capacity of each cache server in megabytes

The next line contains ​V numbers describing the sizes of individual videos in megabytes: S0, S1, … SV-1. Si is the size of video i​ in megabytes ( 1 ≤ Si ≤ 1000).

The next section describes each of the endpoints one after another, from endpoint 0 to endpoint E − 1 . The description of each endpoint consists of the following lines:

  • a line containing two numbers:
    • LD ( 2 ≤ LD ≤ 4000) – the latency of serving a video request from the data center to this endpoint, in milliseconds
    • K ( 0 ≤ K ≤ C ) – the number of cache servers that this endpoint is connected to
  • K lines describing the connections from the endpoint to each of the K connected cache servers.
    Each line contains the following numbers:

    • c ( 0 ≤ c < C ) – the ID of the cache server
    • Lc ( 1 ≤ Lc ≤ 500) – the latency of serving a video request from this cache server to this endpoint, in milliseconds. You can assume that latency from the cache is strictly lower than latency from the data center ( 1 ≤ Lc < LD

Finally, the last section contains R request descriptions in separate lines. Each line contains the following numbers:

  • Rv ( 0 ≤ Rv < V ) – the ID of the requested video
  • Re ( 0 ≤ Re < E ) – the ID of the endpoint from which the requests are coming from
  • Rn ( 0 < Rn ≤ 10000) – the number of requests

Example

5 2 4 3 100
50 50 80 30 110
1000 3
0 100
2 200
1 300
500 0
3 0 1500
0 1 1000
4 0 500
1 0 1000

Example input file explanation.

5 videos, 2 endpoints, 4 request descriptions, 3 caches 100MB each.
Videos 0, 1, 2, 3, 4 have sizes 50MB, 50MB, 80MB, 30MB, 110MB.
Endpoint 0 has 1000ms datacenter latency and is connected to 3 caches:
The latency (of endpoint 0) to cache 0 is 100ms.
The latency (of endpoint 0) to cache 2 is 200ms.
The latency (of endpoint 0) to cache 1 is 200ms.
Endpoint 1 has 500ms datacenter latency and is not connected to a cache.
1500 requests for video 3 coming from endpoint 0.
1000 requests for video 0 coming from endpoint 1.
500 requests for video 4 coming from endpoint 0.
1000 requests for video 1 coming from endpoint 0.

Connections and latencies between the endpoints and caches of example input.

Submissions

File format

Your submission should start with a line containing a single number N ( 0 ≤ N ≤ C ) – the number of cache server descriptions to follow.

Each of the subsequent N lines should describe the videos cached in a single cache server. It should contain the following numbers:

  • c ( 0 ≤ c < C ) – the ID of the cache server being described,
  • the IDs of the videos stored in this cache server: v0, …, vn ( 0 ≤ vi < V) (at least 0 and at most V numbers), given in any order without repetitions

Each cache server should be described in at most one line. It is not necessary to describe all cache servers: if a cache does not occur in the submission, this cache server will be considered as empty. Cache servers can be described in any order.

Example

3
0 2
1 3 1
2 0 1

Example submission file explanation.

We are using  all 3 cache servers.
Cache server 0 contains only video 2.
Cache server 1 contains videos 3 and 1.
Cache server 2 contains videos 0 and 1.

Validation

The output file is valid if it meets the following criteria:

  • the format matches the description above
  • the total size of videos stored in each cache server does not exceed the maximum cache server capacity

Scoring

The score is the average time saved per request, in microseconds. (Note that the latencies in the input file are given in milliseconds. The score is given in microseconds to provide a better resolution of results.)
For each request description ( Rv, Re, Rn) in the input file, we choose the best way to stream the video Rv to the endpoint Re. We pick the lowest possible latency L = min(LD, L0, … , Lk−1) , where L​D is the latency of serving a video to the endpoint Re from the data center, and L0, … , Lk−1 are latencies of serving a video to the endpoint Re from each cache server that:

  • is connected to the endpoint Re, and
  • contains the video Rv

The time that was saved for each request is LD

As each request description describes Rn requests, the time saved for the entire request description is Rn × ( LD − L ) .

To compute the total score for the data set, we sum the time saved for individual request descriptions in milliseconds, multiply by 1000 and divide it by the total number of requests in all request descriptions, rounding down.

A schematic representation of the example submission file above​ .

In the example​ above, there are three request descriptions for the endpoint 0

  • 1500 requests for video 3, streamed from cache 1 with 300ms of latency, saving 1000ms − 300ms = 700ms per request
  • 500 requests for video 4, streamed from the data center, saving 0ms per request
  • 1000 requests for video 1, streamed from cache 2 with 200ms of latency saving 800ms per request

There is also one request description for the endpoint 1:

  • 1000 requests for video 0, streamed from the data center, saving 0ms per request

The average time saved is:

( 1500x700 + 500x0 + 1000x800 + 1000x0 )/(1500 + 500 + 1000 + 1000)

which equals 462.5ms. Multiplied by 1000, this gives the score of 462 500​.

Note that there are multiple data sets representing separate instances of the problem. The final score for your team will be the sum of your best scores on the individual data sets.

Online Qualification Round Problem for Google Hash Code 2017 - Problem Statement (91 downloads)

Online Qualification Round Problem for Google Hash Code 2017 - Data Sets (79 downloads)


Jenkins: Improve the format of the email

Like many people, we use Jenkins to perform several tasks automatically.
Jenkins is an open source automation server, it provides hundreds of plugins to support building, deploying and automating any project.

One of the features of Jenkins is the option to send an email to the user under certain circumstances, e.g. after a build was complete.
The default layout of those emails is not so pretty, so we did some changes to it.
Note: below we present how to make the changes apply to the whole system, if you do not want to do that, you could apply these changes to individual projects only.

Using an administrative account, click on Manage Jenkins option on the left menu and in the new screen click on Configure System on the right column.
These actions will take you to the configuration page of your Jenkins installation (e.g. http://jenkins:8080/configure).

Scroll down to the category Extended E-mail Notification.

  1. From the drop down menu, change the value of Default Content Type to HTML (text/html).
  2. Update the value of Default Subject to [$BUILD_STATUS] - $PROJECT_NAME - Build # $BUILD_NUMBER ($BUILD_ID). We prefer to have the build status first (like a tag).
  3. In the box named Default Content enter the following block of data:
    $PROJECT_NAME - Build # $BUILD_NUMBER - $BUILD_STATUS.<br/>
    <br/>
    Check console <a href="$BUILD_URL">output</a> to view full results.<br/>
    If you cannot connect to the build server, check the attached logs.<br/>
    <br/>
    --<br/>
    Following is the last 100 lines of the log.<br/>
    <br/>
    --LOG-BEGIN--<br/>
    <pre style='line-height: 22px; display: block; color: #333; font-family: Monaco,Menlo,Consolas,"Courier New",monospace; padding: 10.5px; margin: 0 0 11px; font-size: 13px; word-break: break-all; word-wrap: break-word; white-space: pre-wrap; background-color: #f5f5f5; border: 1px solid #ccc; border: 1px solid rgba(0,0,0,.15); -webkit-border-radius: 4px; -moz-border-radius: 4px; border-radius: 4px;'>
    ${BUILD_LOG, maxLines=100, escapeHtml=true}
    </pre>
    --LOG-END--
    

The email you will receive after a successful execution will be similar to the one below:

Subject: [Successful] - banana - Build # 77 (77)
 Body:

Press the Apply button to save the changes.

You will notice that in the email we mention the following: If you cannot connect to the build server, check the attached logs..
To enable the option to attach the logs, you need to configure your project itself.
Select your project from the main screen and then click on Configure on the left column.

Scroll down to Post-build Actions section.

From the Add post-built action drop down list select Editable Email Notification.

A new block will appear in the page.
Set the value of Attach Build Log drop down to Compress and Attach Build Log and then hit the Apply button.


C: Full example of pthread_cond_timedwait()

The following code has two threads.
The main thread spawns a pthread and then blocks on a condition waiting for a signal from the pthread.
The pthread will perform its task and then signal the main thread.
Once the main thread receives its signal, it will join the pthread and terminate.

Full example of pthread_cond_timedwait with cmake (compressed) (62 downloads)
#include <stdio.h>
#include <sys/types.h>
#include <pthread.h>
#include <asm/errno.h>

#define MAX_WAIT_TIME_IN_SECONDS (6)

struct thread_info_t
{
    // Used to identify a thread.
    pthread_t thread_id;
    // A condition is a synchronization device that allows threads to suspend execution and relinquish the processors until some predicate on shared data is satisfied.
    // The basic operations on conditions are: signal the condition (when the predicate becomes true), and wait for the condition, suspending the thread execution until another thread signals the condition.
    pthread_cond_t condition;
    // A mutex is a MUTual EXclusion device, and is useful for protecting shared data structures from concurrent modifications, and implementing critical sections and monitors.
    // A mutex has two possible states: unlocked (not owned by any thread), and locked (owned by one thread).
    // A mutex can never be owned by two different threads simultaneously.
    // A thread attempting to lock a mutex that is already locked by another thread is suspended until the owning thread unlocks the mutex first.
    pthread_mutex_t mutex;
};

void error_pthread_mutex_unlock(const int unlock_rv)
{
    fprintf(stderr, "Failed to unlock mutex.\n");
    switch (unlock_rv)
    {
        case EINVAL:
            fprintf(stderr, "The value specified by mutex does not refer to an initialized mutex object.\n");
            break;
        case EAGAIN:
            fprintf(stderr, "The mutex could not be acquired because the maximum number of recursive locks for mutex has been exceeded.\n");
            break;
        case EPERM:
            fprintf(stderr, "The current thread does not own the mutex.\n");
            break;
        default:
            break;
    }
}

void error_pthread_mutex_lock(const int lock_rv)
{
    fprintf(stderr, "Failed to lock mutex.\n");
    switch (lock_rv)
    {
        case EINVAL:
            fprintf(stderr, "The value specified by mutex does not refer to an initialized mutex object or the mutex was created with the protocol attribute having the value PTHREAD_PRIO_PROTECT and the calling thread's priority is higher than the mutex's current priority ceiling.\n");
            break;
        case EAGAIN:
            fprintf(stderr, "The mutex could not be acquired because the maximum number of recursive locks for mutex has been exceeded.\n");
            break;
        case EDEADLK:
            fprintf(stderr, "A deadlock condition was detected or the current thread already owns the mutex.\n");
            break;
        default:
            break;
    }
}

void error_pthread_cond_signal(const int signal_rv)
{
    fprintf(stderr, "Could not signal.\n");
    if (signal_rv == EINVAL)
    {
        fprintf(stderr, "The value cond does not refer to an initialised condition variable.\n");
    }
}

void error_pthread_setcanceltype(const int setcanceltype_rv)
{
    fprintf(stderr, "Could not change cancelability type of thread.\n");
    if (setcanceltype_rv == EINVAL)
    {
        fprintf(stderr, "Invalid value for type.\n");
    }
}

void error_pthread_create(const int create_rv)
{
    fprintf(stderr, "Could not create thread.\n");
    switch (create_rv)
    {
        case EAGAIN:
            fprintf(stderr, "Insufficient resources to create another thread or a system-imposed limit on the number of threads was encountered.\n");
            break;
        case EINVAL:
            fprintf(stderr, "Invalid settings in attr.\n");
            break;
        case EPERM:
            fprintf(stderr, "No permission to set the scheduling policy and parameters specified in attr.\n");
            break;
        default:
            break;
    }
}

void error_pthread_cond_timedwait(const int timed_wait_rv)
{
    fprintf(stderr, "Conditional timed wait, failed.\n");
    switch (timed_wait_rv)
    {
        case ETIMEDOUT:
            fprintf(stderr, "The time specified by abstime to pthread_cond_timedwait() has passed.\n");
            break;
        case EINVAL:
            fprintf(stderr, "The value specified by abstime, cond or mutex is invalid.\n");
            break;
        case EPERM:
            fprintf(stderr, "The mutex was not owned by the current thread at the time of the call.\n");
            break;
        default:
            break;
    }
}

void error_pthread_join(const int join_rv)
{

    fprintf(stderr, "Could not join thread.\n");
    switch (join_rv)
    {
        case EINVAL:
            fprintf(stderr, "The implementation has detected that the value specified by thread does not refer to a joinable thread.\n");
            break;
        case ESRCH:
            fprintf(stderr, "No thread could be found corresponding to that specified by the given thread ID.\n");
            break;
        case EDEADLK:
            fprintf(stderr, "A deadlock was detected or the value of thread specifies the calling thread.\n");
            break;
        default:
            break;
    }
}

void error_clock_gettime(const int gettime_rv)
{
    fprintf(stderr, "Could not get time from clock.\n");
    switch (gettime_rv)
    {
        case EFAULT:
            fprintf(stderr, "tp points outside the accessible address space.\n");
            break;
        case EINVAL:
            fprintf(stderr, "The clk_id specified is not supported on this system.\n");
            break;
        case EPERM:
            fprintf(stderr, "clock_settime() does not have permission to set the clock indicated.\n");
            break;
        default:
            break;
    }
}

// This is the thread that will be called by pthread_create() and it will be executed by the new thread.
void *worker_thread(void *data)
{
    // We know that the input data pointer is pointing to a thread_info_t so we are casting it to the right type.
    struct thread_info_t *thread_info = (struct thread_info_t *) data;

    // We block this thread trying to lock the mutex, this way we will make sure that the parent thread had enough time to call pthread_cond_timedwait().
    // When the parent thread calls pthread_cond_timedwait() it will unlock the mutex and this thread will be able to proceed.
    const int lock_rv = pthread_mutex_lock(&(thread_info->mutex));
    if (lock_rv)
    {
        error_pthread_mutex_lock(lock_rv);
    }

    int oldtype;
    // The pthread_setcanceltype() sets the cancelability type of the calling thread to the value given in type.
    // The previous cancelability type of the thread is returned in the buffer pointed to by oldtype.
    // The argument PTHREAD_CANCEL_ASYNCHRONOUS means that the thread can be canceled at any time.
    const int setcanceltype_rv = pthread_setcanceltype(PTHREAD_CANCEL_ASYNCHRONOUS, &oldtype);
    if (setcanceltype_rv)
    {
        error_pthread_setcanceltype(setcanceltype_rv);
    }

    // TODO: This is the place you should implement the functionality that is needed for this thread

    // The pthread_cond_signal() call unblocks at least one of the threads that are blocked on the specified condition variable cond (if any threads are blocked on cond).
    const int signal_rv = pthread_cond_signal(&(thread_info->condition));
    if (signal_rv)
    {
        error_pthread_cond_signal(signal_rv);
    }

    // The pthread_mutex_unlock() function shall release the mutex object referenced by mutex.
    const int unlock_rv = pthread_mutex_unlock(&(thread_info->mutex));
    if (unlock_rv)
    {
        error_pthread_mutex_unlock(unlock_rv);
    }
    return NULL;
}

int main()
{
    struct thread_info_t thread_info;
    pthread_cond_init(&thread_info.condition, NULL);
    pthread_mutex_init(&thread_info.mutex, NULL);

    const int lock_rv = pthread_mutex_lock(&thread_info.mutex);
    if (lock_rv)
    {
        error_pthread_mutex_lock(lock_rv);
    }

    const int create_rv = pthread_create(&(thread_info.thread_id), NULL, &worker_thread, (void *) &thread_info);
    if (create_rv)
    {
        error_pthread_create(create_rv);
        const int unlock_rv = pthread_mutex_unlock(&thread_info.mutex);
        if (unlock_rv)
        {
            error_pthread_mutex_unlock(unlock_rv);
        }
    }
    else
    {
        // timespec is a structure holding an interval broken down into seconds and nanoseconds.
        struct timespec max_wait = {0, 0};

        // The clock_gettime system call has higher precision than its successor the gettimeofday().
        // It has the ability to request specific clocks using the clock id.
        // It fills in a timespec structure with the seconds and nanosecond count of the time since the Epoch (00:00 1 January, 1970 UTC).
        // CLOCK_REALTIME argument represents a system-wide real-time clock. This clock is supported by all implementations and returns the number of seconds and nanoseconds since the Epoch.
        const int gettime_rv = clock_gettime(CLOCK_REALTIME, &max_wait);
        if (gettime_rv)
        {
            error_clock_gettime(gettime_rv);
        }
        max_wait.tv_sec += MAX_WAIT_TIME_IN_SECONDS;

        // The pthread_cond_timedwait() function blocks on a condition variable.
        // It must be called with a mutex locked by the calling thread or undefined behavior results will occur.
        // This function atomically releases the mutex and causes the calling thread to block on the condition variable cond;
        // atomically here means "atomically with respect to access by another thread to the mutex and then the condition variable".
        // That is, if another thread is able to acquire the mutex after the about-to-block thread has released it, then a subsequent call to pthread_cond_broadcast() or pthread_cond_signal() in that thread shall behave as if it were issued after the about-to-block thread has blocked.
        const int timed_wait_rv = pthread_cond_timedwait(&thread_info.condition, &thread_info.mutex, &max_wait);
        if (timed_wait_rv)
        {
            error_pthread_cond_timedwait(timed_wait_rv);
        }

        // The pthread_join() function suspends execution of the calling thread until the target thread terminates, unless the target thread has already terminated.
        const int join_rv = pthread_join(thread_info.thread_id, NULL);
        if (join_rv)
        {
            error_pthread_join(join_rv);
        }
    }
    return 0;
}

Full example of pthread_cond_timedwait with cmake (compressed) (62 downloads)

Back Up Jenkins instance except for workspace and build logs

Our Jenkins setup has a lot of cool features and configuration.
It has ‘project-based security’, it has parametrized projects, multiple source code management blocks per project and fairly extensive tests implemented with several build steps.
Of course, we do not want to lose them, so we make backups often.
The commands we use for the backup are the following.

jenkins_folder="/var/lib/jenkins/";
 backup_folder="$HOME/jenkins/`date +%F`";
 mkdir -p "$backup_folder";
 (cd "$jenkins_folder"/jobs/; find . -mindepth 3 -type d -regex '.*/[0-9]*$' -print) | sed 's|./|jobs/|' | sudo rsync --archive --exclude 'workspace/*' --exclude-from=- "$jenkins_folder" "$backup_folder";

Explanation of commands:

  • In backup_folder="$HOME/jenkins/`date +%F`"; we used the $HOME variable instead of the tilde ~ as this would create a folder in the current directory called ~ instead of creating a new folder called jenkins in the home directory.
  • mkdir -p "$backup_folder"; instructs mkdir to create all parent folders needed to create our destination folder.
  • (cd "$jenkins_folder"/jobs/; find . -mindepth 3 -type d -regex '.*/[0-9]*$' -print) navigates to the directory of jenkins before performing the search, this way the result file names will be relative to the installation location which we need later to pass to rsync.
    Then we search for all folders which their name is numeric and they at least on depth 3. We filter by depth as well to avoid matching folders directly in the jobs folder.
  • sed 's|./|jobs/|' replaces the prefix ./ with jobs/ to match the relative path from where rsync will work from
  • sudo rsync --archive --exclude 'workspace/*' --exclude-from=- "$jenkins_folder" "$backup_folder"; it will copy everything from $jenkins_folder to the folder $backup_folder while excluding the data in workspace and the folders matched from find (the job build folders).
    --exclude-from=- instructs rsync to read from stdin the list of files to exclude.

Fedora 25: install / start / enable ssh server

Install

To install the openssh-server, you need to install the openssh-server package:

sudo dnf install -y openssh-server;

Start

To start the sshd daemon (openssh-server) in the current session:

sudo systemctl start sshd.service;

Stop

To stop the active (if any) sshd daemon in the current session:

sudo systemctl stop sshd.service;

Enable

To configure the sshd daemon to start automatically at boot time:

sudo systemctl enable sshd.service;

You will get an output similar to this:

ln -s '/usr/lib/systemd/system/sshd.service' '/etc/systemd/system/multi-user.target.wants/sshd.service'

Disable

To configure the sshd daemon to stop automatic initialization at boot time:

sudo systemctl disable sshd.service;

Find which Ports are listening on Linux using netstat

netstat prints network connections, routing tables, interface statistics, masquerade connections, and multicast memberships.

Using the parameter -l (or --listening) it will show only listening sockets/ports (which are omitted by default.).
--numeric-ports shows numerical port numbers but does not affect the resolution of host or user names (e.g. instead of showing the name ssh, it will show the value 22).

We used netstat using the following syntax to check which sockets/ports are open on the current machine:

netstat --listening --numeric-ports;

The results appeared as follows:

[[email protected] ~]$ netstat --listening --numeric-ports
 Active Internet connections (only servers)
 Proto Recv-Q Send-Q Local Address           Foreign Address         State      
 tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN     
 tcp        0      0 localhost:25            0.0.0.0:*               LISTEN     
 tcp6       0      0 [::]:44300              [::]:*                  LISTEN     
 tcp6       0      0 [::]:8080               [::]:*                  LISTEN     
 tcp6       0      0 [::]:22                 [::]:*                  LISTEN     
 tcp6       0      0 localhost:25            [::]:*                  LISTEN     
 udp        0      0 0.0.0.0:39925           0.0.0.0:*                          
 udp        0      0 0.0.0.0:24186           0.0.0.0:*                          
 udp        0      0 0.0.0.0:68              0.0.0.0:*                          
 udp        0      0 localhost:323           0.0.0.0:*                          
 udp        0      0 0.0.0.0:5353            0.0.0.0:*                          
 udp6       0      0 localhost:323           [::]:*                             
 udp6       0      0 [::]:33848              [::]:*                             
 udp6       0      0 [::]:61453              [::]:*                             
 raw6       0      0 [::]:58                 [::]:*                  7          
 Active UNIX domain sockets (only servers)
 Proto RefCnt Flags       Type       State         I-Node   Path
 unix  2      [ ACC ]     STREAM     LISTENING     22489    public/showq
 unix  2      [ ACC ]     STREAM     LISTENING     22445    public/pickup
 unix  2      [ ACC ]     STREAM     LISTENING     22449    public/cleanup
 unix  2      [ ACC ]     STREAM     LISTENING     22477    private/proxymap
 unix  2      [ ACC ]     STREAM     LISTENING     22480    private/proxywrite
 unix  2      [ ACC ]     STREAM     LISTENING     15452    /run/systemd/private
 unix  2      [ ACC ]     STREAM     LISTENING     22483    private/smtp
 unix  2      [ ACC ]     STREAM     LISTENING     22486    private/relay
 unix  2      [ ACC ]     STREAM     LISTENING     22492    private/error
 unix  2      [ ACC ]     STREAM     LISTENING     22495    private/retry
 unix  2      [ ACC ]     STREAM     LISTENING     22498    private/discard
 unix  2      [ ACC ]     STREAM     LISTENING     22501    private/local
 unix  2      [ ACC ]     STREAM     LISTENING     22504    private/virtual
 unix  2      [ ACC ]     STREAM     LISTENING     22507    private/lmtp
 unix  2      [ ACC ]     STREAM     LISTENING     22510    private/anvil
 unix  2      [ ACC ]     STREAM     LISTENING     22513    private/scache
 unix  2      [ ACC ]     STREAM     LISTENING     14445    /var/run/NetworkManager/private-dhcp
 unix  2      [ ACC ]     SEQPACKET  LISTENING     15476    /run/udev/control
 unix  2      [ ACC ]     STREAM     LISTENING     1404     /run/systemd/journal/stdout
 unix  2      [ ACC ]     STREAM     LISTENING     22452    public/qmgr
 unix  2      [ ACC ]     STREAM     LISTENING     15498    /run/lvm/lvmpolld.socket
 unix  2      [ ACC ]     STREAM     LISTENING     22474    public/flush
 unix  2      [ ACC ]     STREAM     LISTENING     22471    private/verify
 unix  2      [ ACC ]     STREAM     LISTENING     16034    /var/run/dbus/system_bus_socket
 unix  2      [ ACC ]     STREAM     LISTENING     16037    /var/run/avahi-daemon/socket
 unix  2      [ ACC ]     STREAM     LISTENING     15537    /run/lvm/lvmetad.socket
 unix  2      [ ACC ]     STREAM     LISTENING     22456    private/tlsmgr
 unix  2      [ ACC ]     STREAM     LISTENING     22459    private/rewrite
 unix  2      [ ACC ]     STREAM     LISTENING     22462    private/bounce
 unix  2      [ ACC ]     STREAM     LISTENING     22465    private/defer
 unix  2      [ ACC ]     STREAM     LISTENING     22468    private/trace

Check a specific port if it is open from a remote machine

In case you want to check a specific port if it is open from a remote machine, you can use nmap.
Using nmap to scan specific ports allows you to check if a remote machine appears to have open ports available to you.
nmap is a network exploration tool and security / port scanner.

The following example checks ports 80 and 8080 on 192.168.1.199 if they are open.

[[email protected] ~]$ nmap -vv -p 80,8080 192.168.1.199
 
 Starting Nmap 6.40 ( http://nmap.org ) at 2017-02-22 14:10 EET
 Initiating Ping Scan at 14:10
 Scanning 192.168.1.199 [2 ports]
 Completed Ping Scan at 14:10, 0.00s elapsed (1 total hosts)
 Initiating Parallel DNS resolution of 1 host. at 14:10
 Completed Parallel DNS resolution of 1 host. at 14:10, 0.00s elapsed
 Initiating Connect Scan at 14:10
 Scanning 192.168.1.199 [2 ports]
 Discovered open port 8080/tcp on 192.168.1.199
 Completed Connect Scan at 14:10, 0.00s elapsed (2 total ports)
 Nmap scan report for 192.168.1.199
 Host is up (0.000060s latency).
 Scanned at 2017-02-22 14:10:29 EET for 0s
 PORT     STATE  SERVICE
 80/tcp   closed http
 8080/tcp open   http-proxy
 
 Read data files from: /usr/bin/../share/nmap
 Nmap done: 1 IP address (1 host up) scanned in 0.03 seconds

The -vv parameter for nmap increases the verbosity of the results.
The -p parameter defines the ports to be checked.


About restoring a deleted Git branch

Recently, a branch was deleted from the server without it being merged.
Luckily for us, we had a local copy.

We used used the command git reflog to get access to the reference logs of the branch.
The command returned results similar to the below:

271f0084 [email protected]{0}: pull: Merge made by the 'recursive' strategy.
 0e71c0b [email protected]{1}: commit: Minor change: Cracked NSA systems
 c81624b [email protected]{2}: pull: Fast-forward
 ef1f281 [email protected]{3}: commit (amend): Deployed satelite
 1ed1a5a [email protected]{4}: commit (amend): Deployed satelite
 a8682cb [email protected]{5}: commit (amend): Deployed satelite
 e4560c8 [email protected]{6}: commit (amend): Deployed satelite
 1679a90 [email protected]{7}: commit (amend): Deployed satelite
 d27d2c9 [email protected]{8}: commit: Deployed satelite
 aaf8261 [email protected]{9}: checkout: moving from master to crack-nsa-systems
 2500e11 [email protected]{10}: clone: from ssh://[email protected]:7999/secret/bananas.git

From this information we got the hash value in front of the commit which we wanted to use to restore, which was 271f0084.

Then, we checked out that version using

git checkout 271f0084;

When we tried to push the branch back to the server the Git pre commit hooks blocked the operation.
Based on the rsa key used, we could only submit changes that were committed by the owner of that key.
The option to disable temporarily the pre commit hooks was unfortunately out of the question.
So we had to replace all author names and emails with the name of the one holding the key.

To do so we used the following command before pushing to the server once more:

git filter-branch --commit-filter '
   if [ "$GIT_COMMITTER_NAME" = "Doe, John" ];
   then
     GIT_COMMITTER_NAME="Squarepants, Bob";
     GIT_AUTHOR_NAME="Squarepants, Bob";
     GIT_COMMITTER_EMAIL="[email protected]";
     GIT_AUTHOR_EMAIL="[email protected]";
     git commit-tree "[email protected]";
   else
     git commit-tree "[email protected]";
   fi' HEAD

git filter-branch lets you rewrite Git revision history by rewriting the branches mentioned, in our case it was HEAD, applying custom filters on each revision. Those filters can modify each tree (e.g. removing a file or running a perl rewrite on all files) or information about each commit. Otherwise, all information (including original commit times or merge information) will be preserved.

HEAD is a reference to the currently checked out commit. In normal states, it’s actually a symbolic reference to the branch you have checked out.
Looking at the contents of .git/HEAD you’ll see something similar to ref: refs/heads/master.
The branch itself is a reference to the commit at the tip of the branch.


Rough notes for DJI Phantom 2 Vision +

After you connect to the Wi-Fi hot spot of the device (SSID: Phantom_XXXXXX):

System 1

ssh [email protected]
#Password for root is 19881209
[email protected]'s password: 


BusyBox v1.19.4 (2013-04-22 22:41:57 CST) built-in shell (ash)
Enter 'help' for a list of built-in commands.

 -----------------------------------------------------
 DJI-INNOVATIONS
 -----------------------------------------------------
  * FC200-Vision+        
  * [email protected]
  * 02/20/2014
  * Version 1.01
  * ART / DHCP down / ping
 -----------------------------------------------------



System is based on OpenWrt.

System 2

ssh [email protected]
#Password for root is 19881209
[email protected]'s password: 


BusyBox v1.19.4 (2013-04-22 22:41:57 CST) built-in shell (ash)
Enter 'help' for a list of built-in commands.

 -----------------------------------------------------
 DJI-INNOVATIONS
 -----------------------------------------------------
  * FC200-Vision+ RE   
  * [email protected]
  * 02/21/2014
  * Version 1.01
  * ART / DHCP down / ping
 -----------------------------------------------------

ssh -L 8080:downloads.openwrt.org:80 [email protected]

cat /etc/opkg.conf 
src/gz barrier_breaker http://downloads.openwrt.org/snapshots/trunk/ar71xx/packages
dest root /
dest ram /tmp
lists_dir ext /var/opkg-lists
option overlay_root /overlay

cat /etc/opkg.conf 
src/gz barrier_breaker http://downloads.openwrt.org/barrier_breaker/14.07/ar71xx/generic/packages/base
dest root /
dest ram /tmp
lists_dir ext /var/opkg-lists
option overlay_root /overlay
option http_proxy http://127.0.0.1:8080/

[email protected]:~# opkg update
Downloading http://downloads.openwrt.org/barrier_breaker/14.07/ar71xx/generic/packages/base/Packages.gz.
Updated list of available packages in /var/opkg-lists/barrier_breaker.
[email protected]:~# opkg install tcpdump-mini
Installing tcpdump-mini (4.5.1-4) to root...
Downloading http://downloads.openwrt.org/barrier_breaker/14.07/ar71xx/generic/packages/base/tcpdump-mini_4.5.1-4_ar71xx.ipk.
Installing libpcap (1.5.3-1) to root...
Downloading http://downloads.openwrt.org/barrier_breaker/14.07/ar71xx/generic/packages/base/libpcap_1.5.3-1_ar71xx.ipk.
Configuring libpcap.
Configuring tcpdump-mini.




BAD TRUNC
opkg install tcpdump-mini
Installing tcpdump-mini (4.5.1-4) to root...
Downloading http://downloads.openwrt.org/snapshots/trunk/ar71xx/generic/packages/base/tcpdump-mini_4.5.1-4_ar71xx.ipk.
Installing libpcap (1.7.4-1) to root...
Downloading http://downloads.openwrt.org/snapshots/trunk/ar71xx/generic/packages/base/libpcap_1.7.4-1_ar71xx.ipk.
Configuring libpcap.
//usr/lib/opkg/info/libpcap.postinst: line 4: default_postinst: not found
Configuring tcpdump-mini.
//usr/lib/opkg/info/tcpdump-mini.postinst: line 4: default_postinst: not found
Collected errors:
 * pkg_run_script: package "libpcap" postinst script returned status 127.
 * opkg_configure: libpcap.postinst returned 127.
 * pkg_run_script: package "tcpdump-mini" postinst script returned status 127.
 * opkg_configure: tcpdump-mini.postinst returned 127.


From host machine:
ssh [email protected] tcpdump -i br-lan -s0 -w - port not 22 > dji-wifi-traffic.pcap

ssh [email protected] tcpdump -n -i br-lan -s0 -w - port 2001 > dji.2001.pcap


ip.src==192.168.1.1 || ip.dst==192.168.1.1 ||ip.src==192.168.1.2 || ip.dst==192.168.1.2 || ip.src==192.168.1.10 || ip.dst==192.168.1.10

System is based on OpenWrt.


C Bit Fields: Full example code

The following two examples demonstrate the use of bit fields to reduce memory consumption of certain applications.

In the first example we create a compressed ‘bit’ struct and on the second we create a weird struct representation for bytes to show that the size of that struct is significantly less that the original one.

First example: Using bit fields to create a ‘bit’ structure

bits_bit_fields.c (compressed) (30 downloads)
#include <stdio.h>

// Weird structure to represent a 'bit'
typedef struct {
    unsigned char value;
} bit;

// 'bit' structure using bit fields
typedef struct {
    unsigned char value : 1;
} bit_bit_field;

int main( ) {

    printf( "Memory size occupied by 'bit' struct : %zu bytes\n", sizeof(bit));
    printf( "Memory size occupied by 'bit_bit_field' struct : %zu bytes\n", sizeof(bit_bit_field));

    bit bits[8];
    bit_bit_field bits_bit_field[8];

    printf( "Memory size occupied by 'bits' array : %zu bytes\n", sizeof(bits));
    printf( "Memory size occupied by 'bits_bit_field' array : %zu bytes\n", sizeof(bits_bit_field));

    // Setting the value of the first 'bit' and then printing it.
    // We will use various values for this test to show that when you set a value to a bit field
    // that is greater that the allowed size it will fill it using the last bits only.
    // It will not spill data though to neighbouring 'bits'.
    unsigned char value;
    for (value = 0; value < 4; value++)
    {
        printf("Input Value: %d\n", value);
        int bits_i;
        const int bits_length = (sizeof(bits) / sizeof(bit));
        for (bits_i = 0; bits_i < bits_length; bits_i++)
        {
            if (bits_i % 2)
            {
                bits[bits_i].value = 0;
            }
            else
            {
                bits[bits_i].value = value;
            }
            printf("%d", bits[bits_i].value);
        }
        printf("\n");

        int bits_bit_field_i;
        const int bits_bit_field_length = (sizeof(bits_bit_field) / sizeof(bit_bit_field));
        for (bits_bit_field_i = 0; bits_bit_field_i < bits_bit_field_length; bits_bit_field_i++)
        {
            if (bits_bit_field_i % 2)
            {
                bits_bit_field[bits_bit_field_i].value = 0;
            }
            else
            {
                bits_bit_field[bits_bit_field_i].value = value;
            }
            printf("%d", bits_bit_field[bits_bit_field_i].value);
        }
        printf("\n");
    }
    return 0;
}
bits_bit_fields.c (compressed) (30 downloads)

Execution output

Memory size occupied by 'bit' struct : 1 bytes
Memory size occupied by 'bit_bit_field' struct : 1 bytes
Memory size occupied by 'bits' array : 8 bytes
Memory size occupied by 'bits_bit_field' array : 8 bytes
Input Value: 0
00000000
00000000
Input Value: 1
10101010
10101010
Input Value: 2
20202020
00000000
Input Value: 3
30303030
10101010

Second example: Using bit fields to create a ‘byte’ structure where each ‘bit’ is another named member

bytes_bit_fields.c (compressed) (33 downloads)
#include <stdio.h>

// Weird structure to represent a 'byte'
typedef struct {
    unsigned char bit_0;
    unsigned char bit_1;
    unsigned char bit_2;
    unsigned char bit_3;
    unsigned char bit_4;
    unsigned char bit_5;
    unsigned char bit_6;
    unsigned char bit_7;
} byte;

// 'byte' structure using bit fields
// Unfortunately we cannot declare an array where the values are bit fields,
// so we have to declare each member separately.
// We instruct the compiler to use only one bit per element.
typedef struct {
    unsigned char bit_0 : 1;
    unsigned char bit_1 : 1;
    unsigned char bit_2 : 1;
    unsigned char bit_3 : 1;
    unsigned char bit_4 : 1;
    unsigned char bit_5 : 1;
    unsigned char bit_6 : 1;
    unsigned char bit_7 : 1;
} byte_bit_field;

int main( ) {

    printf( "Memory size occupied by 'byte' struct : %zu bytes\n", sizeof(byte));
    printf( "Memory size occupied by 'byte_bit_field' struct : %zu bytes\n", sizeof(byte_bit_field));

    byte bytes[8];
    byte_bit_field bytes_bit_field[8];

    printf( "Memory size occupied by 'bytes' array : %zu bytes\n", sizeof(bytes));
    printf( "Memory size occupied by 'bytes_bit_field' array : %zu bytes\n", sizeof(bytes_bit_field));

    // Setting the value of the first 'bit' and then printing it.
    // We will use various values for this test to show that when you set a value to a bit field
    // that is greater that the allowed size it will fill it using the last bits only.
    // It will not spill data though to neighbouring 'bits'.
    unsigned char value;
    for (value = 0; value < 4; value++)
    {
        printf("Input Value: %d\n", value);
        int bytes_i;
        const int bytes_length = (sizeof(bytes) / sizeof(byte));
        for (bytes_i = 0; bytes_i < bytes_length; bytes_i++)
        {
            if (bytes_i % 2)
            {
                bytes[bytes_i].bit_3 = 0;
            }
            else
            {
                bytes[bytes_i].bit_3 = value;
            }
            printf(" %d  ", bytes[bytes_i].bit_3);
        }
        printf("\n");

        int bytes_bit_field_i;
        const int bytes_bit_field_length = (sizeof(bytes_bit_field) / sizeof(byte_bit_field));
        for (bytes_bit_field_i = 0; bytes_bit_field_i < bytes_bit_field_length; bytes_bit_field_i++)
        {
            if (bytes_bit_field_i % 2)
            {
                bytes_bit_field[bytes_bit_field_i].bit_3 = 0;
            }
            else
            {
                bytes_bit_field[bytes_bit_field_i].bit_3 = value;
            }
            printf("%d%d%d ",
                   bytes_bit_field[bytes_bit_field_i].bit_2,
                   bytes_bit_field[bytes_bit_field_i].bit_3,
                   bytes_bit_field[bytes_bit_field_i].bit_4);
        }
        printf("\n");
    }
    return 0;
}
bytes_bit_fields.c (compressed) (33 downloads)

Execution output

Memory size occupied by 'byte' struct : 8 bytes
Memory size occupied by 'byte_bit_field' struct : 1 bytes
Memory size occupied by 'bytes' array : 64 bytes
Memory size occupied by 'bytes_bit_field' array : 8 bytes
Input Value: 0
 0   0   0   0   0   0   0   0  
000 000 000 000 000 000 000 000 
Input Value: 1
 1   0   1   0   1   0   1   0  
010 000 010 000 010 000 010 000 
Input Value: 2
 2   0   2   0   2   0   2   0  
000 000 000 000 000 000 000 000 
Input Value: 3
 3   0   3   0   3   0   3   0  
010 000 010 000 010 000 010 000

grep: How to match lines using any of multiple patterns

Recently, we needed to filter the results of ps x using two different patterns.
The first pattern was ./ where we needed to match that exact character sequence.
The . period character is treated as a special character in regular expressions (it matches a single character of any value, except for the end of line), so we decided to use the -F parameter to remove this special handling.
Doing this change prevented us from writing a regular expression that uses the OR | operator.

-F (or --fixed-strings) is a matching control option that instructs grep to interpret the patterns as a list of fixed strings (instead of regular expressions), separated by newlines, any of which is to be matched.
We tried assigning the different patterns as different lines to a variable and then using them on the pipe, like in the following example:

patterns="./
banana";
ps x | grep -F $patterns;

..but it failed.

Solution

grep supports a matching control option -e that allows us to define multiple patterns using different strings.

-e PATTERN (or --regexp=PATTERN) uses the value PATTERN as the pattern. If this option is used  multiple times or it is combined with the -f (--file) option, grep will search for all patterns given.

In the end, our command was transformed to the following, which worked just fine!

ps x | grep -F -e "./" -e "banana";