As you can read here How to resize ZFS, it is possible to resize a zpool if you replace each disk of the pool with a bigger one. In my first post I used a virtual machine to test this. And... I've replaced one disk (eg. da0) with a different device (eg. da4).
In the real world I would like to replace one device with the same (but bigger one). I was able to get some very old SCSI Disks in an external SCSI enclosure. These disks are perfect for testing this...
Here we go...
My testsystem is a Sempron 2600+ with 1 GB RAM and several SCSI Disks connected via a DAWICONTROL DC-2976 UW controller. The FreeNAS Version I've used for this test is 0.7 Sardaukar (revision 4390)
I've created a testpool with three 2.1 GB SCSI-Disks (yes, it's true, just 2.1 GB ;-) ). And I've also created two filesystems (testfs1, compressed & testfs2), and some testfiles.
crusher:~# zpool list testpool0
NAME SIZE USED AVAIL CAP HEALTH ALTROOT
testpool0 5.91G 2.81G 3.09G 47% ONLINE -
As you can see, this pool has a size of 5.91 GB (remember this)
crusher:~# zfs list
NAME USED AVAIL REFER MOUNTPOINT
bigpool 100M 364G 25.3K /mnt/bigpool
bigpool/temp 100M 364G 100M /mnt/bigpool/temp
testpool0 1.87G 2.00G 28.0K /mnt/testpool0
testpool0/testfs1 640M 2.00G 640M /mnt/testpool0/testfs1
testpool0/testfs2 1.25G 2.00G 1.25G /mnt/testpool0/testfs2
crusher:~# zpool status testpool0
pool: testpool0
state: ONLINE
scrub: none requested
config:
NAME STATE READ WRITE CKSUM
testpool0 ONLINE 0 0 0
raidz1 ONLINE 0 0 0
da0 ONLINE 0 0 0
da1 ONLINE 0 0 0
da2 ONLINE 0 0 0
errors: No known data errors
First, I'd like to start with a scrub to be sure that the data is safe on these disks...
crusher:~# zpool scrub testpool
This will take a while. Check the status with:
crusher:~# zpool status testpool0
pool: testpool0
state: ONLINE
scrub: scrub in progress, 93.34% done, 0h0m to go
config:
NAME STATE READ WRITE CKSUM
testpool0 ONLINE 0 0 0
raidz1 ONLINE 0 0 0
da0 ONLINE 0 0 0
da1 ONLINE 0 0 0
da2 ONLINE 0 0 0
errors: No known data errors
crusher:~# zpool status testpool0
pool: testpool0
state: ONLINE
scrub: scrub completed with 0 errors on Fri Feb 27 21:40:16 2009
config:
NAME STATE READ WRITE CKSUM
testpool0 ONLINE 0 0 0
raidz1 ONLINE 0 0 0
da0 ONLINE 0 0 0
da1 ONLINE 0 0 0
da2 ONLINE 0 0 0
errors: No known data errors
Everything looks perfect... so lets start with the replace. This SCSI enclosure supports Hot-Swapping of the disks (more or less).
Ok, lets replace the first disk (da0)
Offline the first disk
crusher:~# zpool offline testpool0 da0
Bringing device da0 offline
crusher:~# zpool status testpool0
pool: testpool0
state: DEGRADED
status: One or more devices has been taken offline by the administrator.
Sufficient replicas exist for the pool to continue functioning in a
degraded state.
action: Online the device using 'zpool online' or replace the device with
'zpool replace'.
scrub: scrub completed with 0 errors on Fri Feb 27 21:40:16 2009
config:
NAME STATE READ WRITE CKSUM
testpool0 DEGRADED 0 0 0
raidz1 DEGRADED 0 0 0
da0 OFFLINE 0 0 0
da1 ONLINE 0 0 0
da2 ONLINE 0 0 0
errors: No known data errors
Replace the first disk (da0)
First physicaly ;-)
And then do the replace command
crusher:~# zpool replace testpool0 da0
crusher:~# zpool status testpool0
pool: testpool0
state: DEGRADED
status: One or more devices is currently being resilvered. The pool will
continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
scrub: resilver in progress, 14.79% done, 0h2m to go
config:
NAME STATE READ WRITE CKSUM
testpool0 DEGRADED 0 0 0
raidz1 DEGRADED 0 0 0
replacing DEGRADED 0 0 0
da0/old OFFLINE 0 0 0
da0 ONLINE 0 0 0
da1 ONLINE 0 0 0
da2 ONLINE 0 0 0
errors: No known data errors
After a while, the resivering has finished
crusher:~# zpool status testpool
pool: testpool
state: ONLINE
scrub: resilver completed with 0 errors on Fri Feb 27 20:05:52 2009
config:
NAME STATE READ WRITE CKSUM
testpool ONLINE 0 0 0
raidz1 ONLINE 0 0 0
da0 ONLINE 0 0 0
da1 ONLINE 0 0 0
da2 ONLINE 0 0 0
errors: No known data errors
Lets replace the second and third disk...
crusher:~# zpool status testpool0
pool: testpool0
state: ONLINE
scrub: scrub completed with 0 errors on Fri Feb 27 22:57:19 2009
config:
NAME STATE READ WRITE CKSUM
testpool0 ONLINE 0 0 0
raidz1 ONLINE 0 0 0
da0 ONLINE 0 0 0
da1 ONLINE 0 0 0
da2 ONLINE 0 0 0
errors: No known data errors
crusher:~# zpool list
NAME SIZE USED AVAIL CAP HEALTH ALTROOT
bigpool 556G 150M 556G 0% ONLINE -
testpool0 5.91G 2.82G 3.09G 47% ONLINE -
crusher:~# zfs list
NAME USED AVAIL REFER MOUNTPOINT
bigpool 100M 364G 25.3K /mnt/bigpool
bigpool/temp 100M 364G 100M /mnt/bigpool/temp
testpool0 1.87G 2.00G 28.0K /mnt/testpool0
testpool0/testfs1 640M 2.00G 640M /mnt/testpool0/testfs1
testpool0/testfs2 1.25G 2.00G 1.25G /mnt/testpool0/testfs2
As you can see the size of the testpool hasn't changed. After exporting and importing the pool, the additional space is available!
crusher:~# zpool export testpool0
crusher:~# zpool import testpool0
crusher:~# zpool list
NAME SIZE USED AVAIL CAP HEALTH ALTROOT
bigpool 556G 150M 556G 0% ONLINE -
testpool0 12.0G 2.82G 9.15G 23% ONLINE -
The size has been changed to 12.0 GB...
14 comments:
4390 seems to be pretty good (thx to fbsd7 and zfs ;-).
If I make a test server with this build, What's the deal about updates with next 0.7's (alpha) builds ?
Thx
If you look into the 'nightly build' download section, you will see a new release. So, there is much progress...
Thanks for your job.
I'm a 0.69 user and I know how to update freenas in "standard cases".
Can the 0.7 be updated as easily while in alpha ?
I know 0.7 is not stable but I need ZFS...
@Julien: so far it was straight forward to do updates for 0.7. If something doesn't work, you don't have to fear about your data (if you are using ZFS ;-), just do a 'zpool import [-f] yourpool )
I found your blog via LearnfreeNas.com. I also wrote a few Freenas Posts on my blog http://frugaltech.blogspot.com
You have some very helpful posts about freeNAS. Thanks for such a helpful blog.
Nice articles, congratulations and thank you!
I have one question, it's regarding another method of expanding the total storage space available.
I have a zpool with a vdev of 3 disks in RAIDZ. Can I add another vdev of 3 RAIDZ disks to the same zpool? I don't see this possible through the GUI, is it possible at the shell? In theory this should expand the total zpool space by the space provided by the second vdev. Did you try something similar?
This method was what I was looking forward to in ZFS. Easy-as-pie storage expanding without having to rebuild raidz, pools, devices or whatever, just add a device to the pool and voila, more space. :)
@Stra: It is possible to expand a zpool with a vdev without any problems. But to be honest, I haven't done this with FreeNAS. I will going to test this soon.
Nice post! Very useful to see ZFS in action on FreeNAS.
Got a question: would the procedure be the same if the controller didn't support hotswap?
@misterredman If the controller doesn't support hotswap, you should better shutdown your server between each step. Otherwise it could be possible that you damage your hardware.
Alright thanks. So basically after putting the disk offline I should shut the computer down and replace the hard drive. The reboot and use the replace command, right?
@misterredman correct!
First off: Great articles! Thanks a lot!
May I ask if you have tried to resize a ZFS pool by adding an additional disk instead of replacing small disks with bigger-ones? I would like to verify if FreeNAS is capable of growing an existing ZFS pool by just adding disks when there is need for more storage after some time..
Cheers!
Igor
THX :-D It is possible to add disks to a pool. But if you add a single disk to a redundant pool (eg RAIDZ or mirrored) you will loose redundancy. You can combine several redundant configurations to one pool. See my first post... and a good introduction to zfs can be found here -> http://www.opensolaris.org/os/community/zfs
Great post.
I've been playing around with ZFS and FreeNAS in a virtual machine.
My main concern is, if I do a ZFS in raidz with 4x 1TB discs, will I later be able to migrate to 4x 2TB by replacing it one at a time and resilvering?
From my tests with freeNas in the VM, I wasn't able to accomplish disk. (But maybe i just did it wrong - i simulated the fail of 1 of the 4 hard drives by removing it in the "disk management", and now everything seems to be acting weird. Even going to the command line, I can't get the ZFS to detach/attach/scrub)
Post a Comment