DD + Lzo = superfast

I recently purchased a new laptop, (Hp Envy 14) that came pre-installed with windows. After leaving it on the hard drive for some time i finally needed the space. I shrank the partition as small as it could go but alas i have a SSD and every GB is expensive and needed.

I thought to myself what’s the best way to back this up, dd + nc was the way to go i thought. So i fired up

dd if=/dev/sda1 | nc 192.168.1.1

and went away. After 2 minutes i came back and did a iotop only to find it running at 22MB per second. Interesting I thought, why is it so slow running over gigabyte Ethernet to another machine.

Arrrr the other machine has a spinning metal disk storage system. So lets try this again,

dd if=/dev/sda1 | pv | gzip -c | nc 192.168.1.1 

I added the PV so i could see what was going on ( read more about pv here ). Ok now its running at 25MB per second, WTF its eating all my CPU and that’s now the limiting factor.

Ok after some time I thought what about lzo, OK it’s not amazing at compressing things, but its relatively CPU inexpensive. I only really knew about this after working with it on the Hadoop cluster at work. After compiling lzop i gave it a shot.

dd if=/dev/sda1 | pv | lzop -c |nc 192.168.1.1 8000 

After watching it for 1 minutes the normal throughput was abut 52MB per second. Giving me a massive speed increase, doubling the throughput. The limiting factor now is the disk at the other end. I could start playing with LZO compression options as my CPU is only running at 50% but its faster enough for me now.