Cluster on zfs?

[totalimpact]

I came across some quanta servers too cheap to turn down and want to make a test lab to try some ZFS clustering on low end disks... looking for some commentary- what will give best performance with least risk, and least complexity, most functionality.

I have been using PVE since v1.4 and am well versed at basic setups, to clustering over NFS or drbd and have deployed quite a few systems.

2x compute nodes, dual e5-2650 , 32gb RAM, 10gb LAN, 1 small ssd each node.
1x storage node, dual xeon 32gb RAM, 10gb LAN, 8x 3.5 bays, 10 SATA/SAS ports, room to maybe slip in 2.5 SSD.
1x 10 GB Switch.

I have the basics down with ZFS, I know I probably need some more RAM. I like the option of pve-zsync to some remote zfs nas.

I have 6~8 500gb drives laying around, and thinking a PCIe nmve for L2ARC, and/or small 2.5 SSD for ZIL like older Intel SLC, LZ4 enabled.

Most of my deployments are single node, 8-10 SSD in a Dell R730xd with H710 raid w\ LVM storage and get 1.5gb/s drive benchmarks. I would like to see how well ZFS can do with cheap spinners/ssd cache over 10gb.

Should the main storage node run PVE, or should I use something like Freenas to run the ZFS? Should I share the pool as NFS, or is there a better method to remote mount to 2 clients?

Suggestions on ZFS raid level or separate vdevs?

What features would I have or miss in this scenario, ie live migration??

 

[LnxBil]

Should the main storage node run PVE, or should I use something like Freenas to run the ZFS? Should I share the pool as NFS, or is there a better method to remote mount to 2 clients?

@mir is currently working on a ZFS-over-FreeNAS Plugin which integrates a ZFS storage in a clustered PVE setup.

Suggestions on ZFS raid level or separate vdevs?

Best performance with mirrored vdevs (so basically a RAID10). If you need the space, go with zraid1.

 

[totalimpact]

That is what I was thinking, with the mirrored vdevs, I think that allows me to add new mirrors down the road to more easily expand it.

 

[mir]

That is what I was thinking, with the mirrored vdevs, I think that allows me to add new mirrors down the road to more easily expand it.

If you add a mirrored vdev to another mirrored vdev you effectively creates a RAID10;-)

 

[sienar]

If you add a mirrored vdev to another mirrored vdev you effectively creates a RAID10;-)

Sort of. ZFS attempts to fill all VDEVs equally. When you add a new VDEV, it will be empty and ZFS will send pretty much all writes to that new VDEV. Only once all the VDEVs (old + a new one) are equally full (by percentage, I believe), will you get the performance you're describing.