Need ZFS or general config advice on mixed HPE DL360 cluster setup to replace vmware.

[swits]

I'm looking to create a cluster with zfs replication on at least 3 HPE servers to keep internal VM services up. They could survive some downtime and data loss from migration times.

I'm curious what peoples thoughts are on a setup with the hardware I'm dealing with. I started with my main new server. I have proxmox installed on 2(of the 8) SSDs with ext4 on hardware mirror. I planned to add the other 6 dual mode drives with RAIDz2. I've read concerns about VMs wearing the SSDs, poor performance with RAIDz2 and needing raid10 mirrors. I’m worried about the mixed disk types on the HP servers, but I’m unsure if I’m overcomplicating things. How would you configure this cluster?

Also I have a QNAP and Synology with 10Gbe NFS for backups.


Main Server

• Model: HPE DL360 Gen10
  Processor: 2 x 24 core Xeon Platinum 8168
  Ram: 256GB
  Network: 2 x 10 Gbe + 4 x 1 Gbe
  Storage: 8 x 1TB Sata SSDs
  Hardware Raid: P308i-a SR Gen10

• Model: HPE DL360 Gen10
  Processor: 12 core Xeon Silver 4116
  Ram: 32GB
  Network: 1 x 10Gb + 4 x 1 Gbe
  Storage: 8 x 1.2TB SAS 10k
  Hardware Raid: P408i a SR Gen10

• Model: HPE DL360 Gen9
  Processor: 2 x 12 core Xeon E5-2670
  Ram: 128GB
  Network: 2 x 10Gbe + 4 x 1Gbe
  Storage: 8 x 2.4TB SAS 10k
  Hardware Raid: P440ar

• Model: HPE DL360p Gen8
  Processor: 2 x 6 core Xeon E5-2640
  Ram: 64GB
  Network: 4 x 1Gbe *2 x 10Gbe for $128
  Storage: 8 x 300GB SAS 15k
  Hardware Raid: P420

• Model: HPE DL360p Gen8
  Processor: 2 x 6 core Xeon E5-2640
  Ram: 80GB
  Network: 4 x 1Gbe *2 x 10Gbe for $128
  Storage: 8 x 1.8TB SAS 10k
  Hardware Raid: P420i

Current amount of VMs are 12 Linux and 2 Windows

 

[UdoB]

I'm looking to create a cluster with zfs replication on at least 3 HPE servers

You say "3" but you list five.

If you can actually run more than three then I would evaluate Ceph. With only three I would use ZFS with replication.

Most of the drives are HDDs, right? In that case I would highly recommend multiple ZFS mirrors (aka Raid10) with a supporting "Special Device" consisting of two (mirrored) fast (but small) SSD/NVMe.

That's my quick answer; of course ymmv...