Monthly Archives: November 2016


Reclaim empty space from VirtualBox VDI disk images by shrinking

Recently, we did some cleanup in certain GNU/Linux virtual machines, where we hoped that VirtualBox would release the disk space that is not used and shrink the size of the VDI files.
Unfortunately, that did not happen even after freeing more than 100GB of space from the guest machine.

We did manage though to reclaim the empty space manually, using the zerofree and VBoxManage utilities.

Following is the guide we followed to do so:

Part 1: Clean-up the guest machine using zerofree

We needed to find the unallocated, blocks with non-zero value content in the ext2, ext3 or ext4 filesystem (e.g. /dev/sda1) and fill them with zeroes.
Since the filesystem has to be unmounted or mounted as read-only for zerofree to work, we decided to use a Live CD to complete this task as it would be the simplest solution to follow.

Step 1: Download SystemRescueCd Live Image

We downloaded the SystemRescueCd Live Image and inserted it to the guest machine as a Live CD.

Step 2: Identify partitions and disks using fdisk -l

We booted the virtual machine from the Live CD, when the terminal appeared, we typed fdisk -l to see all available disks and partitions.

Depending on your configuration you will see the following

If you performed your installation using standard partition

Disk /dev/sda: 200 GiB, 214748364800 bytes, 419430400 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x64af3527

Device     Boot     Start       End   Sectors  Size Id Type
/dev/sda1  *         2048 411043839 411041792  196G 83 Linux
/dev/sda2       411045886 419428351   8382466    4G  5 Extended
/dev/sda5       411045888 419428351   8382464    4G 82 Linux swap / Solaris

or if you performed your installation using LVM

Disk /dev/sda: 214.7 GB, 214748364800 bytes, 419430400 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk label type: dos
Disk identifier: 0x000523ce

   Device Boot      Start         End      Blocks   Id  System
/dev/sda1   *        2048     1026047      512000   83  Linux
/dev/sda2         1026048   419430399   209202176   8e  Linux LVM

Disk /dev/mapper/centos-root: 53.7 GB, 53687091200 bytes, 104857600 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes


Disk /dev/mapper/centos-swap: 4160 MB, 4160749568 bytes, 8126464 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes


Disk /dev/mapper/centos-home: 156.3 GB, 156304932864 bytes, 305283072 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes

From the above results, we keep the paths to the real partitions and the paths to the LVM partitions, e.g.
/dev/sda1 which is 196GB, /dev/mapper/centos-home and /dev/mapper/centos-root

Step 3: Perform the cleanup

To perform the clean up, execute for each of the above paths the command zerofree, e.g.

zerofree /dev/sda1;
zerofree /dev/mapper/centos-home;
zerofree /dev/mapper/centos-root;

Give it some time to complete the task, the larger the partition, the more the time it will take.

Part 2: Shrink the guest machine disk images using VBoxManage

We used VBoxManage with the parameter --compact, which it is used to compact disk images, i.e. remove blocks that only contains zeroes. It shrinks dynamically allocated images by reducing the physical size of the image without affecting the logical size of the virtual disk. Compaction works both for base images and for diff images created as part of a snapshot. For this operation to be effective, it is required that free space in the guest system first be zeroed out and that is why we had to perform Step 1 using zerofree before.
Please note that compacting is currently only available for VDI images.

To use, just issue the command "C:\Program Files\Oracle\VirtualBox\VBoxManage.exe" modifyhd --compact <DISK_PATH> pointing to the disk you just cleaned up using zerofree
Please note that the virtual machine should be stopped before starting this operation.

Press the key combination Win + R to pop up the Run prompt.
Type cmd in the input box and hit the Enter key.

"C:\Program Files\Oracle\VirtualBox\VBoxManage.exe" modifyhd --compact "F:\Ubuntu\Ubuntu.vdi"
"C:\Program Files\Oracle\VirtualBox\VBoxManage.exe" modifyhd --compact "C:\Users\bytefreaks\VirtualBox VMs\CentOS\CentOS.vdi"

Advertisements

Downgrade GNU patch on CentOS 7.0 (64bit) to version 2.6.1

Recently we had to download GNU patch from version 2.7.1 to any version less than version 2.7 series.
We used patch version 2.6.1 which is the latest in the version 2.6 series.

We were trying to compile Linux Kernel 3.0.35 source code with some custom patches.
While applying the patches we got the following errors:

File firmware/imx/sdma/sdma-imx25-to1.bin: git binary diffs are not supported.
File firmware/imx/sdma/sdma-imx31-to1.bin: git binary diffs are not supported.
File firmware/imx/sdma/sdma-imx31-to2.bin: git binary diffs are not supported.
File firmware/imx/sdma/sdma-imx35-to1.bin: git binary diffs are not supported.
File firmware/imx/sdma/sdma-imx35-to2.bin: git binary diffs are not supported.
File firmware/imx/sdma/sdma-imx50-to1.bin: git binary diffs are not supported.
File firmware/imx/sdma/sdma-imx51-to3.bin: git binary diffs are not supported.
File firmware/imx/sdma/sdma-imx53-to1.bin: git binary diffs are not supported.

Apparently, version 2.7 does not support binary diffs.
We can verify this claim from the release announcement.

Support for most features of the “diff –git” format, including renames and copies, permission changes, and symlink diffs. Binary diffs are not supported yet; patch will complain and skip them.

Methodology

#Making sure we are not missing any 32bit libraries since we are on a 64bit machine
yum install glibc.i686 ncurses-libs.i686;
#Download the source code
wget ftp://ftp.gnu.org/gnu/patch/patch-2.6.1.tar.gz;
#Extract the files
tar -zxf patch-2.6.1.tar.gz;
#Navigate to the folder
cd patch-2.6.1;
#Configure the installation and make all necessary checks
./configure;
#Build
make;
#Remove existing version
sudo yum remove patch -y;
#Install
sudo make install;


How to suppress binary files from matching results

When you try to find all files that contain a certain string value, it can be very costly to check binary files that you might not want to check.
To automatically prevent your search from testing if the binary files contain the needle you can add the parameter -I (capital i) to prevent grep from testing them.
Using grep, -I will process a binary file as if it did not contain matching data, this is equivalent to the --binary-files=without-match option.

Example

find . -type f -exec grep 'string' '{}' -s -l -I \;

The above command breaks down as follows:

  • find . -type f Find all files in current directory.
  • -exec For each match execute the following.
  • grep 'string' '{}' Search the matched file '{}' if it contains the value ‘string’.
  • -s Suppress error messages about nonexistent or unreadable files.
  • -l (lambda lower case) or --files-with-matches Suppress normal output, instead print the name of each input file from which output would normally have been printed. The scanning will stop on the first match.
  • -I (i capital) or --binary-files=without-match Process a binary file as if it did not contain matching data.

Extract audio from online video

Using the youtube-dl command line application you can download videos and directly extract the audio of the video in various formats.

youtube-dl supports a large variety of online video hosts, including:

  • youtube.com
  • 9gag.com
  • crunchyroll.com
  • dailymotion.com
  • southparkstudios.com

Note: Please be sure that you are allowed to download a video before you do that, many of these hosts do not expect that you will be downloading their videos as they do not allow that.

The following command, will download a video, convert it to an mp3 and delete the original video:

youtube-dl --extract-audio --audio-format mp3 https://www.youtube.com/watch?v=cwI-n3sI8ec

If you want to keep the original video, you just add the parameter -k or --keep-video.

The --audio-format parameter accepts other types of audio format outputs, specifically it supports

youtube-dl is a very powerful tool, advice the documentation for some of the great features it supports.

 


How to add automatically all empty folders in git repository 4

Since you are searching for this issue, you must have realised that git does not support storing empty folders/directories.

Currently the design of the Git index (staging area) only permits files to be listed, and nobody competent enough to make the change to allow empty directories has cared enough about this situation to remedy it.

Directories are added automatically when adding files inside them. That is, directories never have to be added to the repository, and are not tracked on their own.
— From https://git.wiki.kernel.org/index.php/Git_FAQ#Can_I_add_empty_directories.3F

All the content is stored as tree and blob objects, with trees corresponding to UNIX directory entries and blobs corresponding more or less to inodes or file contents. A single tree object contains one or more tree entries, each of which contains a SHA-1 pointer to a blob or subtree with its associated mode, type, and filename.
— From https://git-scm.com/book/en/v2/Git-Internals-Git-Objects

Below we propose two solutions, depending on how you want to use those empty folders.

Solution A – The folders will always be empty

There are scenarios where the empty folders should always remain empty on git no matter what the local copy has inside them.
Such a scenario would be, wanting to add on git folders where you will build your objects and/or put temporary/cached data.
In such scenarios it is important to have the structure available but never to add those files in git.

To achieve that, we add a .gitignore file in every empty folder containing the following:

# git does not allow empty directories.
# Yet, we need to add this empty directory on git.
# To achieve that, we created this .gitignore file, so that the directory will not be empty thus enabling us to commit it.
# Since we want all generated files/folders in this directory to be ignored by git, we add a rule for this.
*
# And then add an exception for this specifc file (so that we can commit it).
!.gitignore
.gitignore (compressed) (399 downloads)

The above .gitignore file, instructs git to add this file on the repository, and thus add the folder itself while ignoring ALL other files.
You can always update your .gitignore file to allow additional files to be added on the repository at any time.
This way you will never get prompted for temporary files that they were modified/created as part of the status of your repository.

Automation

A way to achieve this automatically, and place a copy of the .gitignore file in every empty folder would be to use the following command to copy an existing .gitignore file in all empty folders.

find $PATH_TO_REPOSITORY -type d ! -path "*.git*" -empty -exec cp .gitignore '{}'/ \;

The above command assumes that there is a .gitignore file in the folder that we are executing from, which it will copy in every empty directory inside the folder that the variable $PATH_TO_REPOSITORY is pointing to.

.gitignore (compressed) (399 downloads)

Solution B – Files will be added eventually to the folders

There are scenarios where the empty folders will be filled at a later stage and we want allow those files on git.
Such a scenario would be, wanting to add on git folders where right now are empty but in some time we will add new source code or resources there.

To achieve that, we add an empty .gitkeep file in every empty folder.

The above .gitkeep file is nothing more than a placeholder.
It is not documented, because it’s not a feature of Git.
It’s a dummy file, so git will not process the empty directory, since git tracks only files.
Once you add other files to the folder, you can safely delete it.
This way you will always get prompted for files that they were modified/created as part of the status of your repository.

Automation

A way to achieve this automatically, and place a copy of the .gitkeep file in every empty folder would be to use the following command to create an empty .gitkeep file in all empty folders.

find $PATH_TO_REPOSITORY -type d ! -path "*.git*" -empty -exec touch '{}'/.gitkeep \;

The above command will create in every empty directory inside the folder that the variable $PATH_TO_REPOSITORY is pointing to a new .gitkeep file.

Finally, push the changes to the git repository

After you create/copy the files, navigate to the repository, add all the new files to the commit, commit them and push them to the repository.

cd $PATH_TO_REPOSITORY;
# Create a new branch
git checkout -b empty_folders;
# Add all modified files to the next commit.
git add .;
git commit -m "Minor change: Adding all empty folders to the repository.";
git push -u origin empty_folders;