ZFS or something else?

[Madtice]

Hi guys,
I have read a lot of threads about zfs vs ext4 and zfs in general. I’m still looking for some answers. Can you guys help me get more insigt into creating a good setup for my homeserver? Currently I use an HP elitedesk with 3 ext4 disks, 1 hdd for root, 1 ssd for VM’s, 1 hdd for data.

my plan is to do the following in my ‘new’ used supermicro 1u :
2x 120gb ssd zfs raid 1 for root
2x 240gb ssd zfs raid 1 for VM’s
I’m still looking for the best way to have local storage for data. My initial thought was to have 2x 2tb (2.5”) in zfs raid 1. But now I’m thinking of 1x 2tb. Mostly because my supermicro mobo only has 6sata ports and for restoring a failing zfs pool you should restore to a new disk before removing the failing disk. Is this correct?

If the single disk is the way to go: is it better to use ext 4? Or zfs? (Also keeping in mind that I’ll have to migrate my current qcow VM’s and data disk to the new zfs pool)

As you can see: lots of questions... to wich I cannot really find the answers. I’m happy with any tips, guidance or recources you can provide me.

Cheerz!
Thijs

 

[aaron]

In general, if the data cannot be restored quickly, never go with a single disk, always use some level of redundancy.

for restoring a failing zfs pool you should restore to a new disk before removing the failing disk. Is this correct?

If the disk has failed, you can only restore from the second disk in the mirror. Thus is makes no difference if the disk is still present.

If the single disk is the way to go: is it better to use ext 4? Or zfs? (Also keeping in mind that I’ll have to migrate my current qcow VM’s and data disk to the new zfs pool)

Even if you decide to use a single disk, ZFS can tell you if there was a problem with the disk, even though it might not be recoverable.

To migrate the current VMs you can create backups and restore them or move the qcow files to another (external) disk temporarily and then use the qm importdisk command to import them to the newly created VMs. It will take care of any conversion between storage types.

 

[Madtice]

Aaron, dude! You have answered so much of my questions! Thanks man! I’ll go ahead and populate all 6 sata ports, good to know that I doensn’t matter if the failing disk is present.

One follow up question: I read a lot about zfs raid5 sort of solutions, is it a dumb idea to do 3x zfs raid1 on a single pve node? It has a single Xeon E31230 and 32gb unbuffered ecc memory (10600e). Will it struggle to much with 3 zfs pools? Not expecting you to pull out your crystal ball but I’ve read that zfs can be a memory hog sometimes.

 

[aaron]

It depends on what you would like to use the raidz for. If you intend to run VMs on it, don't. In short, you will end up with more data used than expected because with ZVOL datasets (which are used for VM disks) parity data will eat it away. I wrote a new section for our documentation covering a few things to consider when planning a ZFS pool. It hasn't been published yet so I can only link to the patch on the mailing list [0]

Should you want to use it purely for local data storage (for a network share) a raidz is okay.

ZFS will use at maximum 50% of RAM. should you run into the situation that you might not be able to start a VM because ZFS is not fast enough to release the hold memory, you can limit it further.

Ideally you have enough RAM available so that close to 100% of all read request during regular operation can be satisfied from RAM.
When you have some kind of monitoring set up, check the zfs arc_hit_ratio.

How much RAM is needed in your setup is something that we cannot say for sure. You will have to test it out to see how well it works.

The following screenshot is taken from the monitoring system of a 2 node cluster that I maintain personally.


You can see that the arc hit ratio is close to 100% most of the time, 99.9x% usually. There are drops, most times when installing updates in VMs or doing other things that are more unusual. You can also see how the arc size fluctuates with time.


[0] https://lists.proxmox.com/pipermail/pve-devel/2020-July/044457.html

 

[Madtice]

Aaron! You’re the man! Thanks for all the info! I’ve read your article (wich to me is a clear read all though I don’t fully understand all the terminology yet) and I’m going to test the 3x zfs raid1 instead of raidz. I will mostly use this thing for VM’s, the only need for bigstorage is my nextcloud instance which doesn’t need to be super mega fast but is is a lot of data.

Thanks a lot for your help!

 

[LnxBil]

Why don't you just go with one pool? This pool will be (assuming you're using stripped mirrors) 3 times faster than one disk and you have all the space in one pool. I don't see any benefit in doing three pools.

 

[aaron]

Why don't you just go with one pool? This pool will be (assuming you're using stripped mirrors) 3 times faster than one disk and you have all the space in one pool. I don't see any benefit in doing three pools.

Since the 3 mirrored disks are all in different sizes (120GB, 240GB, 2TB) the usage of them in one big pool will not be equally distributed. Unless I missed something?

 

[Madtice]

Since the 3 mirrored disks are all in different sizes (120GB, 240GB, 2TB) the usage of them in one big pool will not be equally distributed. Unless I missed something?

Yes, that is correct. I bought a couple of drives before I knew the possibilities of ZFS. Now I’ll probably roll with this because of that.

Why don't you just go with one pool? This pool will be (assuming you're using stripped mirrors) 3 times faster than one disk and you have all the space in one pool. I don't see any benefit in doing three pools.

My primary goal is redundancy. So mirroring instead of striping... It cuts the NET disk space in half but it’ll give me peace of mind (hopefully)

 

[LnxBil]

My primary goal is redundancy. So mirroring instead of striping... It cuts the NET disk space in half but it’ll give me peace of mind (hopefully)

Therefore I suggested a stripped mirror, this is ZFS für RAID10.

Since the 3 mirrored disks are all in different sizes (120GB, 240GB, 2TB) the usage of them in one big pool will not be equally distributed. Unless I missed something?

Oh year, that is a problem. The 120GB and 240GB should work not so bad, but the 2 TB will not work here. You're right. Must have slipped my mind.

 

[apoc]

I would not waste 120gb for a PVE root.
this is way too much storage you never going to use. My PVE server runs from an 8GB data ssd which is plenty of space.
I'd rather use that as a secondary VM pool so you can distribute IO or not even placing them inside the system. You could think of using them as a ZIL or LARC cache for the spinning media as well...

I would always prefer multiple pools over one big pool. All eggs in one basket is dangerous (lost my pool in January this year. That was painful.)