Proxmox VE on ZFS USB Drive mirror with ZIL on SSD

[Roman Dissauer]

Hi,

I'm running a 5 node pve Cluster with HP DL360 gen8 Servers since about 3 years. The integrated HP P420i RAID Controller does not allow booting from disks when in HBA mode, so I ran 6 of the SFF disks as separate raid0 each for ceph osds and 2 of the SFF disks as raid1 for pve boot disks. Basically this is working but monitoring disks and changing failed hdds is a pain in the ass.

Going from P420i Hardware RAID to HBA mode - boot from usb drive(s):
Now I'm thinking of installing pve on ZFS mirrored usb drives (2 drive mirror with 1 spare) with a SSD partition as ZIL device. I know that these USB drives are not known as reliable root filesystem storage due to bad write performance but with the ZIL all writes are going to ssd and are synced back (as combined writes) to the usb drives periodically (time range configurable). Reads are then coming from the ARC (RAM) or optionally L2ARC (SSD).

What do you think about this setup for a pve production environment?

Things to know about HP DL360 gen8 Servers:
again... boot from single disk on P420i in HBA mode is not working...
A separate PCIe NVME SSD as boot device is no option because it is not working either...

 

[Roman Dissauer]

anybody with goot ZFS knowledge out there?

 

[guletz]

Now I'm thinking of installing pve on ZFS mirrored usb drives (2 drive mirror with 1 spare) with a SSD partition as ZIL device. I know that these USB drives are not known as reliable root filesystem storage due to bad write performance but with the ZIL all writes are going to ssd and are synced back (as combined writes) to the usb drives periodically (time range configurable). Reads are then coming from the ARC (RAM) or optionally L2ARC (SSD).

A better ideea is to buy a cheap sata controller(even PCI) and then to add 2 HDDs. With usb HDD after 5 sec(default) even ZIL data will be flush'ed to disks as the ARC data, so in the end you will have some big bursts of data on the usb-hdd. Another problem, is the fact that the HDD-usb suspend could also create problems!