exclude

Back Up Jenkins instance except for workspace and build logs

 in Applications  tagged depth / exclude / find / jenkins / mkdir / rsync by

Our Jenkins setup has a lot of cool features and configuration.
It has ‘project-based security’, it has parametrized projects, multiple source code management blocks per project and fairly extensive tests implemented with several build steps.
Of course, we do not want to lose them, so we make backups often.
The commands we use for the backup are the following.


jenkins_folder="/var/lib/jenkins/";
 backup_folder="$HOME/jenkins/`date +%F`";
 mkdir -p "$backup_folder";
 (cd "$jenkins_folder"/jobs/; find . -mindepth 3 -type d -regex '.*/[0-9]*$' -print) | sed 's|./|jobs/|' | sudo rsync --archive --exclude 'workspace/*' --exclude-from=- "$jenkins_folder" "$backup_folder";

Explanation of commands:

  • In backup_folder="$HOME/jenkins/`date +%F`"; we used the $HOME variable instead of the tilde ~ as this would create a folder in the current directory called ~ instead of creating a new folder called jenkins in the home directory.
  • mkdir -p "$backup_folder"; instructs mkdir to create all parent folders needed to create our destination folder.
  • (cd "$jenkins_folder"/jobs/; find . -mindepth 3 -type d -regex '.*/[0-9]*$' -print) navigates to the directory of jenkins before performing the search, this way the result file names will be relative to the installation location which we need later to pass to rsync.
    Then we search for all folders which their name is numeric and they at least on depth 3. We filter by depth as well to avoid matching folders directly in the jobs folder.
  • sed 's|./|jobs/|' replaces the prefix ./ with jobs/ to match the relative path from where rsync will work from
  • sudo rsync --archive --exclude 'workspace/*' --exclude-from=- "$jenkins_folder" "$backup_folder"; it will copy everything from $jenkins_folder to the folder $backup_folder while excluding the data in workspace and the folders matched from find (the job build folders).
    --exclude-from=- instructs rsync to read from stdin the list of files to exclude.

How to suppress binary files from matching results

 in GNU/Linux  tagged binary / exclude / find / grep by

When you try to find all files that contain a certain string value, it can be very costly to check binary files that you might not want to check.
To automatically prevent your search from testing if the binary files contain the needle you can add the parameter -I (capital i) to prevent grep from testing them.
Using grep, -I will process a binary file as if it did not contain matching data, this is equivalent to the --binary-files=without-match option.

Example

find . -type f -exec grep 'string' '{}' -s -l -I \;

The above command breaks down as follows:

  • find . -type f Find all files in current directory.
  • -exec For each match execute the following.
  • grep 'string' '{}' Search the matched file '{}' if it contains the value ‘string’.
  • -s Suppress error messages about nonexistent or unreadable files.
  • -l (lambda lower case) or --files-with-matches Suppress normal output, instead print the name of each input file from which output would normally have been printed. The scanning will stop on the first match.
  • -I (i capital) or --binary-files=without-match Process a binary file as if it did not contain matching data.