dd

A solution to running out of memory while executing mysqldump

 in Applications  tagged /etc/fstab / /proc/meminfo / /proc/swaps / dd / fstab / mysql / mysqldump / swapon by

Are you trying to perform a mysqldump on a large table and running out of memory every time? This can be a frustrating experience. Even if you try to use the –quick parameter, you may still run out of memory. In this blog post, we will discuss a solution to this problem.

One option is to create a swap file to add more swap space. A swap file differs from a swap partition but can be accessible and dynamic. In the following steps, we will show you how to create a swap file.

First, create an empty file. This file will contain virtual memory contents, so make sure to create a file big enough for your needs. The following command will create a 1GiB file, which means +1GiB swap space for your system:

dd if=/dev/zero of=/media/tux/bigdisk/swapfile.img bs=1024 count=1M;

If you want to create a 3GiB file, change the count value to count=3M. Refer to the man dd for more information.

Next, make a “swap filesystem” inside your new swap file using the following command:

mkswap /media/tux/bigdisk/swapfile.img;
chmod 600 /media/tux/bigdisk/swapfile.img;
chown root:root /media/tux/bigdisk/swapfile.img;

To ensure that your new swap space is activated while booting up your computer, add it to the filesystem configuration file /etc/fstab. Add the following line to the end of the file:

/media/tux/bigdisk/swapfile.img swap swap sw 0 0

This is recommended because other filesystems (at least one that contains a swap file) must be mounted in read-write mode before we can access any files.

Finally, you can either reboot your computer or activate the new swap file manually with the following command:

swapon /media/tux/bigdisk/swapfile.img;

If everything goes well, you should see that more swap space is available for use. You can use the following commands to check your new swap and confirm that it is active:

cat /proc/swaps;

This should display something like:

Filename                           Type       Size    Used    Priority
/swapfile                          file       16777212 1048796    -2
/media/tux/bigdisk/swapfile.img    file       67108860 0          -3

You can also use the following command to check your swap usage:

grep 'Swap' /proc/meminfo;

This should display something like:

SwapCached:         132456 kB
SwapTotal:        83886072 kB
SwapFree:         82837276 kB

Creating a swap file can be an effective solution to running out of memory while performing a mysqldump on a large table. It is a simple, dynamic solution that can be implemented easily on most Linux systems. Following the steps outlined in this post, you should be able to create a swap file and add more swap space to your system.

Status of an executing dd 1

 in Bash  tagged dd / kill / pgrep / status / watch by

Recently, we were cloning a large hard disk on another using dd.
This operation took a really long time, at some point we got curious on what the status of the execution was.
Due to the minimal output dd offers, there was no indication for us whether the system was still copying and if it had a long way to go or not.

Fortunately, the developers of dd added a feature where sending a USR1 signal to a running dd process makes it print I/O statistics to standard error and then resume copying.

To achieve that we used a second terminal and followed these steps:

  1. We used pgrep to look up the running process based on its name and get the dd running process ID (PID): pgrep ^dd$ .
  2. We passed that PID to kill -USR1 which triggered the printing of the statistics on the terminal where dd was executing: kill -USR1 $(pgrep ^dd$).

Solution

kill -USR1 $(pgrep ^dd$);

Bonus

Additionally, we wanted to have dd statistics printed automatically every minute.
To achieve that, we used watch.  watch executes a program periodically, showing it’s output in full-screen.
We defined the interval in seconds using the parameter -n. (Please note that, the command will not allow less than 0.1 second interval.)

In the end, our command became as follows:

watch -n 60 kill -USR1 $(pgrep ^dd$)

The above command was sending a USR1 signal to dd via the kill application every minute (60 seconds) forcing it to print on standard output the I/O statistics.

Example

On terminal 1, we executed the command dd if=/dev/sda of=/dev/sdb;, which will copy disk sda over sdb.

On terminal 2, we executed the command kill -USR1 $(pgrep ^dd$);, which forced dd to print I/O statistics back on terminal 1.

0+49728 records in
7218+0 records out
3695616 bytes (3.7 MB) copied, 2.85812 s, 1.3 MB/s
0+78673 records in
11443+0 records out
5858816 bytes (5.9 MB) copied, 4.49477 s, 1.3 MB/s
0+99003 records in
14386+0 records out
7365632 bytes (7.4 MB) copied, 5.75575 s, 1.3 MB/s
^C0+172104 records in
24918+0 records out
12758016 bytes (13 MB) copied, 10.197 s, 1.3 MB/s

Linux: Bash: Create a empty file of an specific size

 in Bash / GNU/Linux  tagged dd by 

dd if=/dev/zero of=1MBfile bs=1024 count=$((1 * 1024))

The above command will create an empty file named “1MBfile” which is 1 MegaByte large.

Name is defined at the “of” parameter. Size is defined by the number of blocks (variable “count”) times the size of each block (defined by “bs”).

In this example, we have 1024 blocks of 1024 bytes size each.