FreeNAS vs Nexentastor

I've tested the performance of FreeNAS 0.7.1 stable (Build 5127) and Nexentastor Community Edition 3.0.2. I will not dig into the different features or the details of both WebGUIs.

For the tests I've used the following hardware

Server

CPU - Intel Core2Duo E6300 - 1.8 GHz - 2 MB Cache

RAM - 2 GByte

Mainboard - Intel DG965WH

Harddisks - 3x WesternDigital WD3200 (320 GByte connected to the onboard SATA2)

Client

My good old MacBook Pro (late 2008), 4 GByte RAM, Mac OS X 10.6.3

Software
iSCSI initiator - Global SAN iSCSI initiator (4.0.0.204)
Benchmark Software - XBench

Benchmark Software - Helios LAN Test

Benchmark Software - iozone (MacPorts)

Protocols

CIFS
NFS
AFP (currently only available for FreeNAS)
iSCSI

Benchmarks
Xbench - The best out of 5 results
Helios LAN Test - 10 tests, two times, best result
iozone - options -e -i0 -i1 -i2 -+n -r 256k -s2g -t4 -c -x

Tuning

No tuning on the Nexentastor. For FreeNAS I've enabled in the WebGUI System | Advanced | Tuning (enable tuning of some kernel variables)

CIFS
Keep in mind, I've tested CIFS with the built-in smb client of OSX. A native Windows OS might provide different results.

Tuning for FreeNAS (via WebGUI)

Services | CIFS/SMB | Settings

-> Send and Receive Buffer Size to 65535

-> Enable large read/write

-> Auxiliary parameters : max xmit = 65535

-> Auxiliary parameters : socket options = TCP_NODELAY SO_SNDBUF=65535 SO_RCVBUF=65535

If you compare the CIFS results you will see that NexentaStor is slightly faster than FreeNAS

NFS
I wasn't able to get XBench to run on a FreeNAS NFS mountpoint. So you wont see any results on the XBench charts.




With the missing XBench results for FreeNAS I would say NexentaStor is here the winner. It provides slightly higher throughput rates.

AFP
NexentaStor doesn't provide this protocol. As I am a Mac user it is very useful to have a system serving data via AFP. Especially the support of FreeNAS for time machine is IMHO excellent (see my blogposts about backup and restore using FreeNAS and time machine)




FreeNAS is in this disciplin the clear winner.

iSCSI
There was also an issue to get iozone to run on a FreeNAS iSCSI target.





The iSCSI protocol shows several advantages compared to the other. It shows high throughput rates with a low latency. But it is not usable to share the LUN to several systems at the same time (it is possible with special Filesystems, I know, but I think this is not very common). With the lack of the iozone results for FreeNAS I would also say, Nexentastor is here the winner.

Overall comparisson











Conclusion
FreeNAS and Nexenta is on the performance side very much comparable. NexentaStor is slitghly faster in most of the disciplines, but the lack of AFP make them both overall equal.
FreeNAS is easy to setup and has a very clear and structured WebGUI. Nexenta has more features (e.g. built in snapshots, deduplication, etc.)

At the moment I definitely will stay with FreeNAS as the performance is comparable and I really like the integration of Time Machine via AFP.

P.S.: If you are interested in a detailed Excel-Sheet with more details, please drop me a mail harryd71 at users dot sourceforge dot net

Benchmark of FreeNAS 0.7 and a single SSD

As you can read in my previous blog post, I have a SuperTalent Ultra Drive ME MLC 64GB. I am interested about the performance of this little drives. So here is what I've found out...

First of all I've installed this SSD into a server for doing tests.

Server hardware specs

• CPU Intel Core2Quad Q6000 (2.4 GHz)
• 8 GB RAM
• Mainboard Intel DG965WH
• Network Card Intel Pro/1000 PT

FreeNAS 0.7 RC2 (AMD64, rev. 4744)

I've tuned the system a little bit :-)

-> Network|Interfaces|LAN -> Advanced Configuration -> MTU 9000

-> System|Advanced ->Tuning -> Enable tuning of some kernel variables

And -> System|Advanced|sysctl.conf

hw.ata.to = 15
# ATA disk timeout vis-a-vis power-saving

kern.coredump = 0
# Disable core dump

kern.ipc.maxsockbuf = 16777216
# System tuning - Original -> 2097152

kern.ipc.nmbclusters = 32768
# System tuning

kern.ipc.somaxconn = 8192
# System tuning

kern.maxfiles = 65536
# System tuning

kern.maxfilesperproc = 32768
# System tuning

net.inet.tcp.delayed_ack = 0
# System tuning

net.inet.tcp.inflight.enable = 0
# System tuning

net.inet.tcp.path_mtu_discovery = 0
# System tuning

net.inet.tcp.recvbuf_auto = 1
# http://acs.lbl.gov/TCP-tuning/FreeBSD.html

net.inet.tcp.recvbuf_inc = 524288
# http://fasterdata.es.net/TCP-tuning/FreeBSD.html

net.inet.tcp.recvbuf_max = 16777216
# http://acs.lbl.gov/TCP-tuning/FreeBSD.html

net.inet.tcp.recvspace = 65536
# System tuning

net.inet.tcp.rfc1323 = 1
# http://acs.lbl.gov/TCP-tuning/FreeBSD.html

net.inet.tcp.sendbuf_auto = 1
# http://acs.lbl.gov/TCP-tuning/FreeBSD.html

net.inet.tcp.sendbuf_inc = 16384
# http://fasterdata.es.net/TCP-tuning/FreeBSD.html

net.inet.tcp.sendbuf_max = 16777216
# http://acs.lbl.gov/TCP-tuning/FreeBSD.html

net.inet.tcp.sendspace = 65536
# System tuning

net.inet.udp.maxdgram = 57344
# System tuning

net.inet.udp.recvspace = 65536
# System tuning

net.local.stream.recvspace = 65536
# System tuning

net.local.stream.sendspace = 65536
# System tuning

net.inet.tcp.hostcache.expire = 1
# http://fasterdata.es.net/TCP-tuning/FreeBSD.html

Protocol, Client and benchmark programm

I've used AFP (Apple File Protocol), my new MacBook Pro (15", Core2Due 2.4 GHz, 4GB RAM, Unibody late 2008, I've lost my old one) and IOZONE.

IOZONE (Version 3.323 from macports)

Iozone is a w

onderful tool to benchmark filesystems, but it is also a feature monster. After some investigation I've found some useful options for my tests.

iozone -e -i0 -i1 -i2 -+n -r 64k -s8g -t2 -c -x

• -e -- Include flush (fsync,fflush) in the timing calculations
• -i0 -- Test write/rewrite
• -i1 -- Test read/reread
• -i2 -- Test random read/write
• -+n -- Disable retests
• -r 64k -- Record or block size
• -s8g -- Size of the file to test (2x RAM of the client)
• -t2 -- number of threads
• -c -- Include close() in timing calculations
• -x -- Turn of stone-walling (I've received slightly better results with turned this off, see iozone manpage for details)

First impression

The first tests didn't showed me the results I've expected! Diskinfo showed good resutls (140 MB/s read, low latency) but over the GBE-Network I was not able to get mor than 60 MB/s read and 80 MB/s write!

Troubleshooting

I've identified that the network switch I have here (Longshine LCS-GS8208A) was not able to deliver more speed. This is the reason why I've connected the MBP direct to the FreeNAS Server.

Screenshots

Write Performance - Traffic graph

A peak traffic of 890 Mbps isn't too bad for a single HDD

Read performance - Traffic graph

The read performance shows nearly constant bandwidth of more that 910 Mbps

IOZONE results

iozone -e -i0 -i1 -i2 -+n -r 64k -s8g -t2 -c -x

Iozone: Performance Test of File I/O
Version $Revision: 3.323 $
Compiled for 32 bit mode.
Build: macosx

Contributors:William Norcott, Don Capps, Isom Crawford, Kirby Collins
Al Slater, Scott Rhine, Mike Wisner, Ken Goss
Steve Landherr, Brad Smith, Mark Kelly, Dr. Alain CYR,
Randy Dunlap, Mark Montague, Dan Million, Gavin Brebner,
Jean-Marc Zucconi, Jeff Blomberg, Benny Halevy,
Erik Habbinga, Kris Strecker, Walter Wong, Joshua Root.

Run began: Tue Jun 23 15:47:17 2009

Include fsync in write timing
No retest option selected
Record Size 64 KB
File size set to 8388608 KB
Include close in write timing
Stonewall disabled
Command line used: iozone -e -i0 -i1 -i2 -+n -r 64k -s8g -t2 -c -x
Output is in Kbytes/sec
Time Resolution = 0.000001 seconds.
Processor cache size set to 1024 Kbytes.
Processor cache line size set to 32 bytes.
File stride size set to 17 * record size.
Throughput test with 2 processes
Each process writes a 8388608 Kbyte file in 64 Kbyte records

Children see throughput for 2 initial writers = 79800.84 KB/sec
Parent sees throughput for 2 initial writers = 78177.66 KB/sec
Min throughput per process = 39088.91 KB/sec
Max throughput per process = 40711.93 KB/sec
Avg throughput per process = 39900.42 KB/sec
Min xfer = 8388608.00 KB

Children see throughput for 2 readers = 107705.98 KB/sec
Parent sees throughput for 2 readers = 107705.48 KB/sec
Min throughput per process = 53852.83 KB/sec
Max throughput per process = 53853.15 KB/sec
Avg throughput per process = 53852.99 KB/sec
Min xfer = 8388608.00 KB

Children see throughput for 2 random readers = 56735.84 KB/sec
Parent sees throughput for 2 random readers = 56735.20 KB/sec
Min throughput per process = 28367.65 KB/sec
Max throughput per process = 28368.19 KB/sec
Avg throughput per process = 28367.92 KB/sec
Min xfer = 8388608.00 KB

Children see throughput for 2 random writers = 24004.56 KB/sec
Parent sees throughput for 2 random writers = 23958.08 KB/sec
Min throughput per process = 11979.05 KB/sec
Max throughput per process = 12025.51 KB/sec Avg throughput per process = 12002.28 KB/sec
Min xfer = 8388608.00 KB

iozone test complete.

Conclusion

You can see in the results a constant bandwidth of 77 MByte/s write and 105 MByte/s read performance which is IMHO quite good for a single HDD. I am not sure about the random read write performance. I need to digg a bit deeper into this :-)

P.S.
Maybe you want to know 'Why AFP?' First of all, I use Apple Computers at home and it showed me the best throughput compared to NFS or Samba

FreeNAS 0.7 - diskinfo of a SuperTalent UltraDrive ME MLC 64GB

I've bought a SuperTalent UltraDrive ME MLC 64GB for playing around with SSD's. So I will provide here my experiences

For the FreeNAS 0.7 ZFS tests I've installed this SSD into a PC with a Intel Core2Quad Q6000 (2.4 GHz), 8 GB RAM. Mainboard Intel DG965WH.

My first test is a quick diskinfo -ct

Here we go...

freenas:~# diskinfo -ct /dev/ad6

/dev/ad6

512 # sectorsize

64023257088 # mediasize in bytes (60G)

125045424 # mediasize in sectors

124053 # Cylinders according to firmware.

16 # Heads according to firmware.

63 # Sectors according to firmware.

ad:P565045-BDIX-6029019 # Disk ident.

I/O command overhead:

time to read 10MB block 0.068941 sec = 0.003 msec/sector

time to read 20480 sectors 1.169508 sec = 0.057 msec/sector

calculated command overhead = 0.054 msec/sector

Seek times:

Full stroke: 250 iter in 0.024970 sec = 0.100 msec

Half stroke: 250 iter in 0.025027 sec = 0.100 msec

Quarter stroke: 500 iter in 0.049870 sec = 0.100 msec

Short forward: 400 iter in 0.039449 sec = 0.099 msec

Short backward: 400 iter in 0.039230 sec = 0.098 msec

Seq outer: 2048 iter in 0.114818 sec = 0.056 msec

Seq inner: 2048 iter in 0.123779 sec = 0.060 msec

Transfer rates:

outside: 102400 kbytes in 0.696010 sec = 147124 kbytes/sec

middle: 102400 kbytes in 0.618375 sec = 165595 kbytes/sec

inside: 102400 kbytes in 0.623304 sec = 164286 kbytes/sec

As comparison... here is the same command with a WD20EADS 2TB

quake:~# diskinfo -ct /dev/ad4

/dev/ad4

512 # sectorsize

2000398934016 # mediasize in bytes (1.8T)

3907029168 # mediasize in sectors

3876021 # Cylinders according to firmware.

16 # Heads according to firmware.

63 # Sectors according to firmware.

I/O command overhead:

time to read 10MB block 0.072554 sec = 0.004 msec/sector

time to read 20480 sectors 2.515795 sec = 0.123 msec/sector

calculated command overhead = 0.119 msec/sector

Seek times:

Full stroke: 250 iter in 6.150143 sec = 24.601 msec

Half stroke: 250 iter in 4.960021 sec = 19.840 msec

Quarter stroke: 500 iter in 8.214024 sec = 16.428 msec

Short forward: 400 iter in 2.523203 sec = 6.308 msec

Short backward: 400 iter in 2.008208 sec = 5.021 msec

Seq outer: 2048 iter in 0.298850 sec = 0.146 msec

Seq inner: 2048 iter in 0.379041 sec = 0.185 msec

Transfer rates:

outside: 102400 kbytes in 1.077248 sec = 95057 kbytes/sec

middle: 102400 kbytes in 1.378671 sec = 74274 kbytes/sec

inside: 102400 kbytes in 2.267264 sec = 45165 kbytes/sec

As you can see... the seek times of the SSD are impressive!

Benchmarking FreeNAS WritePerformance

Jonas left me a comment here about strange write performance of his FreeNAS 0.7. 

Jonas said...

Im haveing issues with performance on raidz for Freenas. Right now i have four 500GB disks in a raidz1 pool.

Trying to dd 1GB file to my pool is only giving a write performance of 47MB/s. If i test one single disk with diskinfo -tv i get write 48-82MB.

After some private mail and some testing, we've found out that this has something to do with the dd options Jonas used.

freenas:/mnt/Media# dd if=/dev/zero of=mytestfile.out bs=1000 count=1000000 
1000000+0 records in 
1000000+0 records out 
1000000000 bytes transferred in 21.231106 secs (47100702 bytes/sec)

Around 47 MB/s for a very performant system is not enough (AMD Athlon x2 64 Bit 4200+, 2200 MHz, 2 GByte RAM, 4x 500 GB Harddisks configured as RAIDZ1)

Speed of a single disk was measured with diskinfo

freenas:/mnt/Media# diskinfo -tv ad4 
ad4 
512 # sectorsize 
500107862016 # mediasize in bytes (466G) 
976773168 # mediasize in sectors 
969021 # Cylinders according to firmware. 
16 # Heads according to firmware. 
63 # Sectors according to firmware. 
ad:S13TJ1MQ702083 # Disk ident. 
 
Seek times: 
Full stroke: 250 iter in 5.368466 sec = 21.474 msec 
Half stroke: 250 iter in 3.997566 sec = 15.990 msec 
Quarter stroke: 500 iter in 6.592644 sec = 13.185 msec 
Short forward: 400 iter in 2.437786 sec = 6.094 msec 
Short backward: 400 iter in 1.262446 sec = 3.156 msec 
Seq outer: 2048 iter in 0.251187 sec = 0.123 msec 
Seq inner: 2048 iter in 0.243935 sec = 0.119 msec 
Transfer rates: 
outside: 102400 kbytes in 1.245477 sec = 82217 kbytes/sec 
middle: 102400 kbytes in 1.417240 sec = 72253 kbytes/sec 
inside: 102400 kbytes in 2.351795 sec = 43541 kbytes/sec

This looks OK. So whats the problem?

It is the blocksize (bs) dd used to write the testfile! It's just 1000 Bytes!

Use higher numbers and a 'binary prefix' (see http://en.wikipedia.org/wiki/Binary_prefix). I mean something like 512k, 1024k, 2048k, 4096k, 8192k.

Jonas repeated the test with the following results:

freenas:/mnt/Media# dd if=/dev/zero of=testfile bs=8192k count=100
100+0 records in
100+0 records out
838860800 bytes transferred in 5.715633 secs (146766032 bytes/sec)

freenas:/mnt/Media# dd if=/dev/zero of=testfile bs=4096k count=200
200+0 records in
200+0 records out
838860800 bytes transferred in 5.663804 secs (148109079 bytes/sec)

freenas:/mnt/Media# dd if=/dev/zero of=testfile bs=2048k count=200
200+0 records in
200+0 records out
419430400 bytes transferred in 3.429654 secs (122295248 bytes/sec)

freenas:/mnt/Media# dd if=/dev/zero of=testfile bs=1024k count=200
200+0 records in
200+0 records out
209715200 bytes transferred in 0.357848 secs (586045588 bytes/sec)  << To little data gives strange result. Probably because of HD cache.

freenas:/mnt/Media# dd if=/dev/zero of=testfile bs=512k count=1000
1000+0 records in
1000+0 records out
524288000 bytes transferred in 2.576585 secs (203481736 bytes/sec)

freenas:/mnt/Media# dd if=/dev/zero of=testfile bs=256k count=10000
10000+0 records in
10000+0 records out
2621440000 bytes transferred in 16.343732 secs (160394212 bytes/sec)

freenas:/mnt/Media# dd if=/dev/zero of=testfile bs=25k count=100000
100000+0 records in
100000+0 records out
2560000000 bytes transferred in 20.552446 secs (124559383 bytes/sec)

So, it depends how you measure the performance ;-)

FreeNAS 0.7 on a Intel D945GCLF

The Intel D945GCLF will be my new Homeserver. Currently I am using a QNAP TS-209 but it is not optimal for my requirements. As I am using often big files (videos from my HD Camera) and I have lots of photos, the QNAP is a littlebit too slow. 

I've installed FreeNAS 0.7 AMD64 (revision 3514) on that Motherboard.

CPU: Intel Atom processor 230 @ 1.6GHz (standard)
Mem: 2 GB RAM (Kingston KVR667DSN5/2G)
Harddisk: 1x SAMSUNG HD103UJ (configured as a ZPOOL)

Network: Intel Pro/1000 GT

The HELIOS LAN Test showed me up to 45-55 MByte/s Read and Write...

FreeNAS 0.7 on an Intel DG965WH

I've installed FreeNAS 0.7 AMD64 (revision 3514) on an Intel DG965WH Mainboard.

CPU: Core2Duo 6300 @ 1.86GHz (standard)

Mem: 2 GB RAM (Kingston HyperX - KHX6400D2K2/2G)

Harddisks: 4x SAMSUNG HD501LJ (configured as RAIDZ)

The HELIOS LAN Test showed me up to 80 MByte/s Read and Write...

Folding at Home Benchmark

I've benchmarked my FreeNAS testserver with the Folding at Home Benchmark Live CD from notfred.

98.3721 PPDs on a AMD Sempron 2600+ (Socket 754).

FreeNAS 0.7 on my testserver

I am running a FreeNAS 0.7 on my testserver (Sempron 2600+, 1.5 GB RAM, HDD Samsung 1x SP2514N 3x SP2014N).

The OS is running from the 250 GB HDD, the rest of this disk is used for 'permanent' data. The three 200 GB Disks are used in an zpool for tests (sometimes assembled as an raidz, sometimes only a stripe).

I've implemented some additional FreeBSD Tuning variables and I get nice preformance out of this box...