Benchmarking FreeNAS WritePerformance

Jonas left me a comment here about strange write performance of his FreeNAS 0.7. 

Jonas said...

Im haveing issues with performance on raidz for Freenas. Right now i have four 500GB disks in a raidz1 pool.

Trying to dd 1GB file to my pool is only giving a write performance of 47MB/s. If i test one single disk with diskinfo -tv i get write 48-82MB.

After some private mail and some testing, we've found out that this has something to do with the dd options Jonas used.

freenas:/mnt/Media# dd if=/dev/zero of=mytestfile.out bs=1000 count=1000000 
1000000+0 records in 
1000000+0 records out 
1000000000 bytes transferred in 21.231106 secs (47100702 bytes/sec)

Around 47 MB/s for a very performant system is not enough (AMD Athlon x2 64 Bit 4200+, 2200 MHz, 2 GByte RAM, 4x 500 GB Harddisks configured as RAIDZ1)

Speed of a single disk was measured with diskinfo

freenas:/mnt/Media# diskinfo -tv ad4 
ad4 
512 # sectorsize 
500107862016 # mediasize in bytes (466G) 
976773168 # mediasize in sectors 
969021 # Cylinders according to firmware. 
16 # Heads according to firmware. 
63 # Sectors according to firmware. 
ad:S13TJ1MQ702083 # Disk ident. 
 
Seek times: 
Full stroke: 250 iter in 5.368466 sec = 21.474 msec 
Half stroke: 250 iter in 3.997566 sec = 15.990 msec 
Quarter stroke: 500 iter in 6.592644 sec = 13.185 msec 
Short forward: 400 iter in 2.437786 sec = 6.094 msec 
Short backward: 400 iter in 1.262446 sec = 3.156 msec 
Seq outer: 2048 iter in 0.251187 sec = 0.123 msec 
Seq inner: 2048 iter in 0.243935 sec = 0.119 msec 
Transfer rates: 
outside: 102400 kbytes in 1.245477 sec = 82217 kbytes/sec 
middle: 102400 kbytes in 1.417240 sec = 72253 kbytes/sec 
inside: 102400 kbytes in 2.351795 sec = 43541 kbytes/sec

This looks OK. So whats the problem?

It is the blocksize (bs) dd used to write the testfile! It's just 1000 Bytes!

Use higher numbers and a 'binary prefix' (see http://en.wikipedia.org/wiki/Binary_prefix). I mean something like 512k, 1024k, 2048k, 4096k, 8192k.

Jonas repeated the test with the following results:

freenas:/mnt/Media# dd if=/dev/zero of=testfile bs=8192k count=100
100+0 records in
100+0 records out
838860800 bytes transferred in 5.715633 secs (146766032 bytes/sec)

freenas:/mnt/Media# dd if=/dev/zero of=testfile bs=4096k count=200
200+0 records in
200+0 records out
838860800 bytes transferred in 5.663804 secs (148109079 bytes/sec)

freenas:/mnt/Media# dd if=/dev/zero of=testfile bs=2048k count=200
200+0 records in
200+0 records out
419430400 bytes transferred in 3.429654 secs (122295248 bytes/sec)

freenas:/mnt/Media# dd if=/dev/zero of=testfile bs=1024k count=200
200+0 records in
200+0 records out
209715200 bytes transferred in 0.357848 secs (586045588 bytes/sec)  << To little data gives strange result. Probably because of HD cache.

freenas:/mnt/Media# dd if=/dev/zero of=testfile bs=512k count=1000
1000+0 records in
1000+0 records out
524288000 bytes transferred in 2.576585 secs (203481736 bytes/sec)

freenas:/mnt/Media# dd if=/dev/zero of=testfile bs=256k count=10000
10000+0 records in
10000+0 records out
2621440000 bytes transferred in 16.343732 secs (160394212 bytes/sec)

freenas:/mnt/Media# dd if=/dev/zero of=testfile bs=25k count=100000
100000+0 records in
100000+0 records out
2560000000 bytes transferred in 20.552446 secs (124559383 bytes/sec)

So, it depends how you measure the performance ;-)