Benchmarking FreeNAS WritePerformance

Jonas left me a comment here about strange write performance of his FreeNAS 0.7. 

Jonas said...

Im haveing issues with performance on raidz for Freenas. Right now i have four 500GB disks in a raidz1 pool.

Trying to dd 1GB file to my pool is only giving a write performance of 47MB/s. If i test one single disk with diskinfo -tv i get write 48-82MB.

After some private mail and some testing, we've found out that this has something to do with the dd options Jonas used.

freenas:/mnt/Media# dd if=/dev/zero of=mytestfile.out bs=1000 count=1000000 
1000000+0 records in 
1000000+0 records out 
1000000000 bytes transferred in 21.231106 secs (47100702 bytes/sec)

Around 47 MB/s for a very performant system is not enough (AMD Athlon x2 64 Bit 4200+, 2200 MHz, 2 GByte RAM, 4x 500 GB Harddisks configured as RAIDZ1)

Speed of a single disk was measured with diskinfo

freenas:/mnt/Media# diskinfo -tv ad4 
ad4 
512 # sectorsize 
500107862016 # mediasize in bytes (466G) 
976773168 # mediasize in sectors 
969021 # Cylinders according to firmware. 
16 # Heads according to firmware. 
63 # Sectors according to firmware. 
ad:S13TJ1MQ702083 # Disk ident. 
 
Seek times: 
Full stroke: 250 iter in 5.368466 sec = 21.474 msec 
Half stroke: 250 iter in 3.997566 sec = 15.990 msec 
Quarter stroke: 500 iter in 6.592644 sec = 13.185 msec 
Short forward: 400 iter in 2.437786 sec = 6.094 msec 
Short backward: 400 iter in 1.262446 sec = 3.156 msec 
Seq outer: 2048 iter in 0.251187 sec = 0.123 msec 
Seq inner: 2048 iter in 0.243935 sec = 0.119 msec 
Transfer rates: 
outside: 102400 kbytes in 1.245477 sec = 82217 kbytes/sec 
middle: 102400 kbytes in 1.417240 sec = 72253 kbytes/sec 
inside: 102400 kbytes in 2.351795 sec = 43541 kbytes/sec

This looks OK. So whats the problem?

It is the blocksize (bs) dd used to write the testfile! It's just 1000 Bytes!

Use higher numbers and a 'binary prefix' (see http://en.wikipedia.org/wiki/Binary_prefix). I mean something like 512k, 1024k, 2048k, 4096k, 8192k.

Jonas repeated the test with the following results:

freenas:/mnt/Media# dd if=/dev/zero of=testfile bs=8192k count=100
100+0 records in
100+0 records out
838860800 bytes transferred in 5.715633 secs (146766032 bytes/sec)

freenas:/mnt/Media# dd if=/dev/zero of=testfile bs=4096k count=200
200+0 records in
200+0 records out
838860800 bytes transferred in 5.663804 secs (148109079 bytes/sec)

freenas:/mnt/Media# dd if=/dev/zero of=testfile bs=2048k count=200
200+0 records in
200+0 records out
419430400 bytes transferred in 3.429654 secs (122295248 bytes/sec)

freenas:/mnt/Media# dd if=/dev/zero of=testfile bs=1024k count=200
200+0 records in
200+0 records out
209715200 bytes transferred in 0.357848 secs (586045588 bytes/sec)  << To little data gives strange result. Probably because of HD cache.

freenas:/mnt/Media# dd if=/dev/zero of=testfile bs=512k count=1000
1000+0 records in
1000+0 records out
524288000 bytes transferred in 2.576585 secs (203481736 bytes/sec)

freenas:/mnt/Media# dd if=/dev/zero of=testfile bs=256k count=10000
10000+0 records in
10000+0 records out
2621440000 bytes transferred in 16.343732 secs (160394212 bytes/sec)

freenas:/mnt/Media# dd if=/dev/zero of=testfile bs=25k count=100000
100000+0 records in
100000+0 records out
2560000000 bytes transferred in 20.552446 secs (124559383 bytes/sec)

So, it depends how you measure the performance ;-)

BackupPC on FreeNAS - Part 3 (Issues with Samba)

I've showed you in two posts how to install BackupPC on FreeNAS (BackupPC on FreeNAS & BackupPC on FreeNAS - Part 2)

It looks like that there is a problem with samba if you install it as I've described this in my previous posts. The FreeNAS binaries are compiled with different path-settings as the original pkg's from FreeBSD.

So if you do not need samba, don't install the packages! Skip the following steps:

freenas:~# pkg_add -r net/samba-libsmbclient-3.0.28

freenas:~# pkg_add -r net/samba-nmblookup-3.0.28.tbz

freenas:~# pkg_add -r net/samba-3.0.28,1

If you need samba, please be aware that you might see some problems!

One issue I've seen is that the /var/etc/private/smbpasswd file was not updated during the boot. My workaround is to add the following in /etc/rc.d/smbpasswd:

(/bin/echo "${_password}"; /bin/echo "${_password}") | ${command} -c /var/etc/smb.conf -s -a "${_username}" > /dev/null

Can anyone confirm similar problems?

Tuning FreeNAS & ZFS

I am running FreeNAS 0.7 (rev. 3514) since several weeks as my 'production' server at home on a Mini-ITX Atom based system (see FreeNAS 0.7 on a Intel D945GCLF).

I've done some tuning of FreeBSD and ZFS with good experiences (good performance, no panics, etc.).

Here is what I've done...

First of all it is important to 'tune' ZFS. I've seen some panics of my systems without using this parameters. It is necessary to use lots of RAM for ZFS. I have 2GB in my little server...

I've followed the FreeBSD ZFSTuning Guide and added the following lines to the /boot/loader.conf file

vm.kmem_size_max="1073741824"

vm.kmem_size="1073741824"

vfs.zfs.prefetch_disable=1

I am not sure about the vfs.zfs.prefetch_disable=1, but as I said, my experiences are good with this settings.

Editing of the /boot/loader.conf file is easy :-) Use your Web-GUI, go to -> Advanced -> Edit File -> add /boot/loader.conf to the File path and hit Load. Add the lines to that file and hit the Save Button.

Please be aware that these changes are not saved to the 'XML system configuration' (-> System -> Backup/Restore -> Download configuration).

FreeBSD tuning... I've simply used the 'Tuning' of FreeNAS. You can find that also on your Web-GUI.

Go to -> System -> Advanced -> mark the Tuning Box and hit Save.

And last but not least there are some nice tuning variables to be found on the TCP Tuning Guide for FreeBSD (here http://acs.lbl.gov/TCP-tuning/FreeBSD.html or here http://fasterdata.es.net/TCP-tuning/FreeBSD.html)

Add this variables via -> System -> Advanced -> sysctl.conf

Here are the variables in detail:

# ups spinup time for drive recognition

hw.ata.to=15

# System tuning - Original -> 2097152

kern.ipc.maxsockbuf=16777216

# System tuning

kern.ipc.nmbclusters=32768

# System tuning

kern.ipc.somaxconn=8192

# System tuning

kern.maxfiles=65536

# System tuning

kern.maxfilesperproc=32768

# System tuning

net.inet.tcp.delayed_ack=0

# System tuning

net.inet.tcp.inflight.enable=0

# System tuning

net.inet.tcp.path_mtu_discovery=0

# http://acs.lbl.gov/TCP-tuning/FreeBSD.html

net.inet.tcp.recvbuf_auto=1

# http://acs.lbl.gov/TCP-tuning/FreeBSD.html

net.inet.tcp.recvbuf_inc=16384

# http://acs.lbl.gov/TCP-tuning/FreeBSD.html

net.inet.tcp.recvbuf_max=16777216

# System tuning

net.inet.tcp.recvspace=65536

# http://acs.lbl.gov/TCP-tuning/FreeBSD.html

net.inet.tcp.rfc1323=1

# http://acs.lbl.gov/TCP-tuning/FreeBSD.html

net.inet.tcp.sendbuf_auto=1

# http://acs.lbl.gov/TCP-tuning/FreeBSD.html

net.inet.tcp.sendbuf_inc =8192

# System tuning

net.inet.tcp.sendspace=65536

# System tuning

net.inet.udp.maxdgram=57344

# System tuning

net.inet.udp.recvspace=65536

# System tuning

net.local.stream.recvspace=65536

# System tuning

net.local.stream.sendspace=65536

# http://acs.lbl.gov/TCP-tuning/FreeBSD.html

net.inet.tcp.sendbuf_max=16777216

I would really appreciate any comments about this tuning variables!