philipp's weblog header image

Copy files with dd and netcat

October 5th, 2009 · No Comments · other stuff

Today I had to copy several LVM partitions to another machine. The first tools I thought of were dd and netcat (nc), the swiss army knife for every sysadmin. But the following two lines didn’t make me happy:

source$ ssh root@192.168.66.2 -- nc -l 3333 \| dd of=/dev/VolGroupData/data1 &
source$ dd if=/dev/VolGroupData/data1 | nc 192.168.66.2 3333

dstat on the target host shows:

----total-cpu-usage---- -dsk/total- -net/total- ---paging-- ---system--
usr sys idl wai hiq siq| read  writ| recv  send|  in   out | int   csw
[... removed first few lines until memory caches are full ...]
 0   1  88  10   0   0|  15M   15M|  16M  493k|   0     0 |  15k 9177
 0   1  88  10   0   0|  16M   17M|  17M  538k|   0     0 |  16k 9597
 0   1  89  10   0   0|  16M   16M|  17M  547k|   0     0 |  16k 9767
 0   1  88  10   0   0|  16M   15M|  17M  541k|   0     0 |  16k 9413

Only 17 MB/s over Gigabit ethernet? And where are the disk reads coming from? Well, after some more experimenting, the answer was not that hard: block size. dd uses a default block size of 512 Bytes, increasing it to 4 KB shows an entirely different picture:

source$ ssh root@192.168.66.2 -- nc -l 3333 \| dd obs=4K of=/dev/VolGroupData/data1 &
source$ dd ibs=4K if=/dev/VolGroupData/data1 | nc 192.168.66.2 3333
----total-cpu-usage---- -dsk/total- -net/total- ---paging-- ---system--
usr sys idl wai hiq siq| read  writ| recv  send|  in   out | int   csw
[... removed first few lines until memory caches are full ...]
1   6  92   0   0   0|   0    74M|  77M 1821k|   0     0 |  57k   85k
1   7  92   0   0   0|1638B   73M|  77M 1819k|   0     0 |  59k   88k
1   8  91   0   0   0|   0    70M|  74M 1724k|   0     0 |  49k   72k
1   7  92   0   0   0|   0    70M|  74M 1737k|   0     0 |  56k   82k
1   6  92   0   0   0|   0    73M|  76M 1795k|   0     0 |  57k   86k
1   6  93   0   0   0|   0    72M|  75M 1779k|   0     0 |  56k   85k

Now, how about that? Average was in the end around 65 MB/s, which is much closer to what I expected.

Tags:

0 responses so far ↓

  • There are no comments yet...Kick things off by filling out the form below.

Leave a Comment