ZFS Advice

[Phil Shaw]

Hi all,

I'm finally getting around to rebuilding my Windows PC with Proxmox. Would like some advice around ZFS

1.Memory. I have 32GB of non-ECC RAM. Is this enough to support 40TB of raw storage and run a Windows VM on top of Proxmox? Also has anyone had bad experience of data failures from using non-ECC RAM? Could a scrub with a faulting RAM SIMM corrupt a pool?

2.I have 4 x 10TB drives off a HBA and an SSD off the motherboard. I've been told booting via HBA is not a great idea so plan to use the SSD for OS. Is this right? also should I buy another SSD and mirror the OS drives?

3.What VDEV config should I use with the 4 x 10TB drives. I've been told that raidz1 is a very poor choice with large drives as a resilver after single drive failure would put a heavy read load on the remaining 3 and a failed resilver would kill the entire pool. That would leave these options 2 VDEVs with a mirror on each or a single VDEV with raidz2. Am I right that a 2 x mirror would have some read advantage but a raidz2 across 4 drives would be more resilient as any 2 drives could fail.

4.I'd like to use native ZFS encryption. Has anyone had a bad experience with this? Is this a good way to push the key to ZFS on boot: https://github.com/chungy/zfs-boottime-encryption

5.I assume LZ4 compression is on by default for ZFS. Is this a good option?

6.ZIL drive. Is this worth adding via an SSD? Does this need to be mirrored? e.g. a ZIL drive failure on boot will stop the zpool from mounting

7.L2ARC drive. Is this worth adding via an SSD? Is it possible to add a ZIL and L2ARC on a single SSD?

8.Am I right that a zpool will only work if all the VDEVS are working and that you can't remove a VDEV once added to a pool? e.g. if I foolishly added a single drive VDEV to the pool and it failed, you essentially destroy the entire pool?

Thanks
Phil

 

[wolfgang]

Hi,

1.Memory. I have 32GB of non-ECC RAM. Is this enough to support 40TB of raw storage and run a Windows VM on top of Proxmox?

This dependence on how large your Window (memory consumption) is.
ZFS need about 1GB for 1TB used data. Used means without parity.
For the ARC I would recommend 4 - 8 GB.
So OS 2 GB + 8 GB ARC + 16 GB ZFS metadata = 26 GB if the pool is fully used.

also should I buy another SSD and mirror the OS drives?

It is always better to separate the OS from the Guest data.

3.What VDEV config should I use with the 4 x 10TB drives.

Raid10.
RaidZx will give you a bad write performance and high CPU
utilization.

Could a scrub with a faulting RAM SIMM corrupt a pool?

No the pool will not be corrupt.
The worse thing that can happen is you loos data.

4.I'd like to use native ZFS encryption. Has anyone had a bad experience with this? Is this a good way to push the key to ZFS on boot:

I would unmount the crypt dev after the datasets are unlocked.
So the ZFS keys are only at boot time available.

5.I assume LZ4 compression is on by default for ZFS. Is this a good option?

Yes

6.ZIL drive. Is this worth adding via an SSD? Does this need to be mirrored? e.g. a ZIL drive failure on boot will stop the zpool from mounting

If you have much small (< 64K) blocks what will be sync write then the answer is yes.
else No.

7.L2ARC drive. Is this worth adding via an SSD? Is it possible to add a ZIL and L2ARC on a single SSD?

No, this has not much benefit in virtualization.

8.Am I right that a zpool will only work if all the VDEVS are working and that you can't remove a VDEV once added to a pool? e.g. if I foolishly added a single drive VDEV to the pool and it failed, you essentially destroy the entire pool?

Yes correct.
but a mirror are one VDEV. The second drive from a mirror you can remove.