rsync


A trick into using rsync with a custom port

As rsync will not accept to be given a custom port on the destination address, a way around it is to initiate an ssh connection that will handle it for you:

rsync -e 'ssh -p 2222' $SOURCE_DIR $DEST_DIR;


How we sync files between two drives

The rsync command is a powerful tool for file synchronization and transfer in Linux and Unix-like operating systems. It provides a robust and efficient way to copy, backup, and mirror files and directories both locally and remotely. In this article, we will explore the technical details of the following rsync command:

rsync -avh --delete --progress "path/to/source/" "path/to/destination/";

We will explain each option and argument used in the command and their respective functionality.

Options and Arguments

The rsync command is a versatile tool with many options and arguments to customize its behavior. The options used in the above command are:

  • “-a”: This option enables the archive mode, which is a shorthand for several options such as -rlptgoD, -l, -p, -t, -g, -o, and -D. It ensures that rsync preserves file permissions, ownership, timestamps, and symbolic links during the synchronization process.
  • “-v”: This option enables verbose mode, which displays detailed output of the rsync operation, including the transferred files and their sizes.
  • “-h”: This option enables human-readable mode, which displays file sizes in a more readable format, such as “1K” for 1 kilobyte, “1M” for 1 megabyte, etc.
  • “–delete”: This option tells rsync to delete any files at the destination that do not exist at the source. This ensures that the destination is an exact copy of the source.
  • “–progress”: This option displays real-time progress information during the rsync operation, including the percentage of completion and the estimated time remaining.

The arguments used in the command are:

  • “path/to/source/”: This argument specifies the source directory or file that we want to sync. It can be a local path or a remote path using the ssh protocol.
  • “path/to/destination/”: This argument specifies the destination directory or file where we want to copy the source files. It can also be a local or remote path using the ssh protocol.

Explanation

The rsync command is a powerful tool that synchronizes the source and destination directories or files. The -a option enables the archive mode, which ensures that the file metadata is preserved during the transfer, including permissions, ownership, timestamps, and symbolic links. The -v option enables the verbose mode, which provides detailed output of the rsync operation, including the transferred files and their sizes.

The -h option enables the human-readable mode, which displays file sizes in a more readable format. The –delete option tells rsync to delete any files at the destination that do not exist at the source, which ensures that the destination is an exact copy of the source. Finally, the –progress option displays real-time progress information during the rsync operation, including the percentage of completion and the estimated time remaining.

Conclusion

The rsync command is a powerful tool for file synchronization and transfer in Linux and Unix-like operating systems. The command discussed in this article synchronizes the source and destination directories or files, preserves file metadata, displays detailed output, deletes any files that do not exist at the destination, and displays real-time progress information. With the proper use of rsync options and arguments, file synchronization and transfer can be made easy, efficient, and reliable.

Other

We have two external hard disks that we use to keep backups of our data.
The way we do that is by using the command rsync that makes our life easy.

Specifically, we use the following command to synchronize the first hard disk with the second one: rsync -avh –delete –progress “path/to/source/” “path/to/destination/”;

rsync is a fast, versatile, remote (and local) file-copying tool, it is available in almost every system (GNU/Linux, Unix (MacOS as well) and Windows).

The parameters we use are the following:

  • -a, --archive enables archive mode which is equal to -rlptgoD (no -H,-A,-X)
    In more detail it enables all of the following options
    -r, --recursive recurse into directories
    -l, --links copy symlinks as symlinks
    -p, --perms preserve permissions
    -t, --times preserve modification times
    -g, --group preserve group
    -o, --owner preserve owner (super-user only)
    -D same as --devices --specials
    --devices preserve device files (super-user only)
    --specials preserve special files
  • -v, --verbose it increases verbosity of the output
  • -h, --human-readable outputs numbers in a human-readable format
  • --delete deletes extraneous files from destination directories
  • --progress shows progress during transfer

Back Up Jenkins instance except for workspace and build logs

Our Jenkins setup has a lot of cool features and configuration.
It has ‘project-based security’, it has parametrized projects, multiple source code management blocks per project and fairly extensive tests implemented with several build steps.
Of course, we do not want to lose them, so we make backups often.
The commands we use for the backup are the following.


jenkins_folder="/var/lib/jenkins/";
 backup_folder="$HOME/jenkins/`date +%F`";
 mkdir -p "$backup_folder";
 (cd "$jenkins_folder"/jobs/; find . -mindepth 3 -type d -regex '.*/[0-9]*$' -print) | sed 's|./|jobs/|' | sudo rsync --archive --exclude 'workspace/*' --exclude-from=- "$jenkins_folder" "$backup_folder";

Explanation of commands:

  • In backup_folder="$HOME/jenkins/`date +%F`"; we used the $HOME variable instead of the tilde ~ as this would create a folder in the current directory called ~ instead of creating a new folder called jenkins in the home directory.
  • mkdir -p "$backup_folder"; instructs mkdir to create all parent folders needed to create our destination folder.
  • (cd "$jenkins_folder"/jobs/; find . -mindepth 3 -type d -regex '.*/[0-9]*$' -print) navigates to the directory of jenkins before performing the search, this way the result file names will be relative to the installation location which we need later to pass to rsync.
    Then we search for all folders which their name is numeric and they at least on depth 3. We filter by depth as well to avoid matching folders directly in the jobs folder.
  • sed 's|./|jobs/|' replaces the prefix ./ with jobs/ to match the relative path from where rsync will work from
  • sudo rsync --archive --exclude 'workspace/*' --exclude-from=- "$jenkins_folder" "$backup_folder"; it will copy everything from $jenkins_folder to the folder $backup_folder while excluding the data in workspace and the folders matched from find (the job build folders).
    --exclude-from=- instructs rsync to read from stdin the list of files to exclude.