Daily Archives: 22 February 2017


Jenkins: Improve the format of the email 4

Like many people, we use Jenkins to perform several tasks automatically.
Jenkins is an open source automation server, it provides hundreds of plugins to support building, deploying and automating any project.

One of the features of Jenkins is the option to send an email to the user under certain circumstances, e.g. after a build was complete.
The default layout of those emails is not so pretty, so we did some changes to it.
Note: below we present how to make the changes apply to the whole system, if you do not want to do that, you could apply these changes to individual projects only.

Using an administrative account, click on Manage Jenkins option on the left menu and in the new screen click on Configure System on the right column.
These actions will take you to the configuration page of your Jenkins installation (e.g. http://jenkins:8080/configure).

Scroll down to the category Extended E-mail Notification.

  1. From the drop down menu, change the value of Default Content Type to HTML (text/html).
  2. Update the value of Default Subject to [$BUILD_STATUS] - $PROJECT_NAME - Build # $BUILD_NUMBER ($BUILD_ID). We prefer to have the build status first (like a tag).
  3. In the box named Default Content enter the following block of data:
    $PROJECT_NAME - Build # $BUILD_NUMBER - $BUILD_STATUS.<br/>
    <br/>
    Check console <a href="$BUILD_URL">output</a> to view full results.<br/>
    If you cannot connect to the build server, check the attached logs.<br/>
    <br/>
    --<br/>
    Following is the last 100 lines of the log.<br/>
    <br/>
    --LOG-BEGIN--<br/>
    <pre style='line-height: 22px; display: block; color: #333; font-family: Monaco,Menlo,Consolas,"Courier New",monospace; padding: 10.5px; margin: 0 0 11px; font-size: 13px; word-break: break-all; word-wrap: break-word; white-space: pre-wrap; background-color: #f5f5f5; border: 1px solid #ccc; border: 1px solid rgba(0,0,0,.15); -webkit-border-radius: 4px; -moz-border-radius: 4px; border-radius: 4px;'>
    ${BUILD_LOG, maxLines=100, escapeHtml=true}
    </pre>
    --LOG-END--
    

The email you will receive after a successful execution will be similar to the one below:

Subject: [Successful] - banana - Build # 77 (77)
 Body:

Press the Apply button to save the changes.

You will notice that in the email we mention the following: If you cannot connect to the build server, check the attached logs..
To enable the option to attach the logs, you need to configure your project itself.
Select your project from the main screen and then click on Configure on the left column.

Scroll down to Post-build Actions section.

From the Add post-built action drop down list select Editable Email Notification.

A new block will appear in the page.
Set the value of Attach Build Log drop down to Compress and Attach Build Log and then hit the Apply button.


C: Full example of pthread_cond_timedwait() 2

The following code has two threads.
The main thread spawns a pthread and then blocks on a condition waiting for a signal from the pthread.
The pthread will perform its task and then signal the main thread.
Once the main thread receives its signal, it will join the pthread and terminate.

[download id=”2713″]


#include <stdio.h>
#include <sys/types.h>
#include <pthread.h>
#include <asm/errno.h>

#define MAX_WAIT_TIME_IN_SECONDS (6)

struct thread_info_t
{
    // Used to identify a thread.
    pthread_t thread_id;
    // A condition is a synchronization device that allows threads to suspend execution and relinquish the processors until some predicate on shared data is satisfied.
    // The basic operations on conditions are: signal the condition (when the predicate becomes true), and wait for the condition, suspending the thread execution until another thread signals the condition.
    pthread_cond_t condition;
    // A mutex is a MUTual EXclusion device, and is useful for protecting shared data structures from concurrent modifications, and implementing critical sections and monitors.
    // A mutex has two possible states: unlocked (not owned by any thread), and locked (owned by one thread).
    // A mutex can never be owned by two different threads simultaneously.
    // A thread attempting to lock a mutex that is already locked by another thread is suspended until the owning thread unlocks the mutex first.
    pthread_mutex_t mutex;
};

void error_pthread_mutex_unlock(const int unlock_rv)
{
    fprintf(stderr, "Failed to unlock mutex.\n");
    switch (unlock_rv)
    {
        case EINVAL:
            fprintf(stderr, "The value specified by mutex does not refer to an initialized mutex object.\n");
            break;
        case EAGAIN:
            fprintf(stderr, "The mutex could not be acquired because the maximum number of recursive locks for mutex has been exceeded.\n");
            break;
        case EPERM:
            fprintf(stderr, "The current thread does not own the mutex.\n");
            break;
        default:
            break;
    }
}

void error_pthread_mutex_lock(const int lock_rv)
{
    fprintf(stderr, "Failed to lock mutex.\n");
    switch (lock_rv)
    {
        case EINVAL:
            fprintf(stderr, "The value specified by mutex does not refer to an initialized mutex object or the mutex was created with the protocol attribute having the value PTHREAD_PRIO_PROTECT and the calling thread's priority is higher than the mutex's current priority ceiling.\n");
            break;
        case EAGAIN:
            fprintf(stderr, "The mutex could not be acquired because the maximum number of recursive locks for mutex has been exceeded.\n");
            break;
        case EDEADLK:
            fprintf(stderr, "A deadlock condition was detected or the current thread already owns the mutex.\n");
            break;
        default:
            break;
    }
}

void error_pthread_cond_signal(const int signal_rv)
{
    fprintf(stderr, "Could not signal.\n");
    if (signal_rv == EINVAL)
    {
        fprintf(stderr, "The value cond does not refer to an initialised condition variable.\n");
    }
}

void error_pthread_setcanceltype(const int setcanceltype_rv)
{
    fprintf(stderr, "Could not change cancelability type of thread.\n");
    if (setcanceltype_rv == EINVAL)
    {
        fprintf(stderr, "Invalid value for type.\n");
    }
}

void error_pthread_create(const int create_rv)
{
    fprintf(stderr, "Could not create thread.\n");
    switch (create_rv)
    {
        case EAGAIN:
            fprintf(stderr, "Insufficient resources to create another thread or a system-imposed limit on the number of threads was encountered.\n");
            break;
        case EINVAL:
            fprintf(stderr, "Invalid settings in attr.\n");
            break;
        case EPERM:
            fprintf(stderr, "No permission to set the scheduling policy and parameters specified in attr.\n");
            break;
        default:
            break;
    }
}

void error_pthread_cond_timedwait(const int timed_wait_rv)
{
    fprintf(stderr, "Conditional timed wait, failed.\n");
    switch (timed_wait_rv)
    {
        case ETIMEDOUT:
            fprintf(stderr, "The time specified by abstime to pthread_cond_timedwait() has passed.\n");
            break;
        case EINVAL:
            fprintf(stderr, "The value specified by abstime, cond or mutex is invalid.\n");
            break;
        case EPERM:
            fprintf(stderr, "The mutex was not owned by the current thread at the time of the call.\n");
            break;
        default:
            break;
    }
}

void error_pthread_join(const int join_rv)
{

    fprintf(stderr, "Could not join thread.\n");
    switch (join_rv)
    {
        case EINVAL:
            fprintf(stderr, "The implementation has detected that the value specified by thread does not refer to a joinable thread.\n");
            break;
        case ESRCH:
            fprintf(stderr, "No thread could be found corresponding to that specified by the given thread ID.\n");
            break;
        case EDEADLK:
            fprintf(stderr, "A deadlock was detected or the value of thread specifies the calling thread.\n");
            break;
        default:
            break;
    }
}

void error_clock_gettime(const int gettime_rv)
{
    fprintf(stderr, "Could not get time from clock.\n");
    switch (gettime_rv)
    {
        case EFAULT:
            fprintf(stderr, "tp points outside the accessible address space.\n");
            break;
        case EINVAL:
            fprintf(stderr, "The clk_id specified is not supported on this system.\n");
            break;
        case EPERM:
            fprintf(stderr, "clock_settime() does not have permission to set the clock indicated.\n");
            break;
        default:
            break;
    }
}

// This is the thread that will be called by pthread_create() and it will be executed by the new thread.
void *worker_thread(void *data)
{
    // We know that the input data pointer is pointing to a thread_info_t so we are casting it to the right type.
    struct thread_info_t *thread_info = (struct thread_info_t *) data;

    // We block this thread trying to lock the mutex, this way we will make sure that the parent thread had enough time to call pthread_cond_timedwait().
    // When the parent thread calls pthread_cond_timedwait() it will unlock the mutex and this thread will be able to proceed.
    const int lock_rv = pthread_mutex_lock(&(thread_info->mutex));
    if (lock_rv)
    {
        error_pthread_mutex_lock(lock_rv);
    }

    int oldtype;
    // The pthread_setcanceltype() sets the cancelability type of the calling thread to the value given in type.
    // The previous cancelability type of the thread is returned in the buffer pointed to by oldtype.
    // The argument PTHREAD_CANCEL_ASYNCHRONOUS means that the thread can be canceled at any time.
    const int setcanceltype_rv = pthread_setcanceltype(PTHREAD_CANCEL_ASYNCHRONOUS, &oldtype);
    if (setcanceltype_rv)
    {
        error_pthread_setcanceltype(setcanceltype_rv);
    }

    // TODO: This is the place you should implement the functionality that is needed for this thread

    // The pthread_cond_signal() call unblocks at least one of the threads that are blocked on the specified condition variable cond (if any threads are blocked on cond).
    const int signal_rv = pthread_cond_signal(&(thread_info->condition));
    if (signal_rv)
    {
        error_pthread_cond_signal(signal_rv);
    }

    // The pthread_mutex_unlock() function shall release the mutex object referenced by mutex.
    const int unlock_rv = pthread_mutex_unlock(&(thread_info->mutex));
    if (unlock_rv)
    {
        error_pthread_mutex_unlock(unlock_rv);
    }
    return NULL;
}

int main()
{
    struct thread_info_t thread_info;
    pthread_cond_init(&thread_info.condition, NULL);
    pthread_mutex_init(&thread_info.mutex, NULL);

    const int lock_rv = pthread_mutex_lock(&thread_info.mutex);
    if (lock_rv)
    {
        error_pthread_mutex_lock(lock_rv);
    }

    const int create_rv = pthread_create(&(thread_info.thread_id), NULL, &worker_thread, (void *) &thread_info);
    if (create_rv)
    {
        error_pthread_create(create_rv);
        const int unlock_rv = pthread_mutex_unlock(&thread_info.mutex);
        if (unlock_rv)
        {
            error_pthread_mutex_unlock(unlock_rv);
        }
    }
    else
    {
        // timespec is a structure holding an interval broken down into seconds and nanoseconds.
        struct timespec max_wait = {0, 0};

        // The clock_gettime system call has higher precision than its successor the gettimeofday().
        // It has the ability to request specific clocks using the clock id.
        // It fills in a timespec structure with the seconds and nanosecond count of the time since the Epoch (00:00 1 January, 1970 UTC).
        // CLOCK_REALTIME argument represents a system-wide real-time clock. This clock is supported by all implementations and returns the number of seconds and nanoseconds since the Epoch.
        const int gettime_rv = clock_gettime(CLOCK_REALTIME, &max_wait);
        if (gettime_rv)
        {
            error_clock_gettime(gettime_rv);
        }
        max_wait.tv_sec += MAX_WAIT_TIME_IN_SECONDS;

        // The pthread_cond_timedwait() function blocks on a condition variable.
        // It must be called with a mutex locked by the calling thread or undefined behavior results will occur.
        // This function atomically releases the mutex and causes the calling thread to block on the condition variable cond;
        // atomically here means "atomically with respect to access by another thread to the mutex and then the condition variable".
        // That is, if another thread is able to acquire the mutex after the about-to-block thread has released it, then a subsequent call to pthread_cond_broadcast() or pthread_cond_signal() in that thread shall behave as if it were issued after the about-to-block thread has blocked.
        const int timed_wait_rv = pthread_cond_timedwait(&thread_info.condition, &thread_info.mutex, &max_wait);
        if (timed_wait_rv)
        {
            error_pthread_cond_timedwait(timed_wait_rv);
        }

        // The pthread_join() function suspends execution of the calling thread until the target thread terminates, unless the target thread has already terminated.
        const int join_rv = pthread_join(thread_info.thread_id, NULL);
        if (join_rv)
        {
            error_pthread_join(join_rv);
        }
    }
    return 0;
}

[download id=”2713″]


Back Up Jenkins instance except for workspace and build logs

Our Jenkins setup has a lot of cool features and configuration.
It has ‘project-based security’, it has parametrized projects, multiple source code management blocks per project and fairly extensive tests implemented with several build steps.
Of course, we do not want to lose them, so we make backups often.
The commands we use for the backup are the following.


jenkins_folder="/var/lib/jenkins/";
 backup_folder="$HOME/jenkins/`date +%F`";
 mkdir -p "$backup_folder";
 (cd "$jenkins_folder"/jobs/; find . -mindepth 3 -type d -regex '.*/[0-9]*$' -print) | sed 's|./|jobs/|' | sudo rsync --archive --exclude 'workspace/*' --exclude-from=- "$jenkins_folder" "$backup_folder";

Explanation of commands:

  • In backup_folder="$HOME/jenkins/`date +%F`"; we used the $HOME variable instead of the tilde ~ as this would create a folder in the current directory called ~ instead of creating a new folder called jenkins in the home directory.
  • mkdir -p "$backup_folder"; instructs mkdir to create all parent folders needed to create our destination folder.
  • (cd "$jenkins_folder"/jobs/; find . -mindepth 3 -type d -regex '.*/[0-9]*$' -print) navigates to the directory of jenkins before performing the search, this way the result file names will be relative to the installation location which we need later to pass to rsync.
    Then we search for all folders which their name is numeric and they at least on depth 3. We filter by depth as well to avoid matching folders directly in the jobs folder.
  • sed 's|./|jobs/|' replaces the prefix ./ with jobs/ to match the relative path from where rsync will work from
  • sudo rsync --archive --exclude 'workspace/*' --exclude-from=- "$jenkins_folder" "$backup_folder"; it will copy everything from $jenkins_folder to the folder $backup_folder while excluding the data in workspace and the folders matched from find (the job build folders).
    --exclude-from=- instructs rsync to read from stdin the list of files to exclude.

Fedora 25: install / start / enable ssh server

Install

To install the openssh-server, you need to install the openssh-server package:

sudo dnf install -y openssh-server;

Start

To start the sshd daemon (openssh-server) in the current session:

sudo systemctl start sshd.service;

Stop

To stop the active (if any) sshd daemon in the current session:

sudo systemctl stop sshd.service;

Enable

To configure the sshd daemon to start automatically at boot time:

sudo systemctl enable sshd.service;

You will get an output similar to this:

ln -s '/usr/lib/systemd/system/sshd.service' '/etc/systemd/system/multi-user.target.wants/sshd.service'

Disable

To configure the sshd daemon to stop automatic initialization at boot time:

sudo systemctl disable sshd.service;