Need ZFS or general config advice on mixed HPE DL360 cluster setup to replace vmware.

[swits]

I'm looking to create a cluster with zfs replication on at least 3 HPE servers to keep internal VM services up. They could survive some downtime and data loss from migration times.

I'm curious what peoples thoughts are on a setup with the hardware I'm dealing with. I started with my main new server. I have proxmox installed on 2(of the 8) SSDs with ext4 on hardware mirror. I planned to add the other 6 dual mode drives with RAIDz2. I've read concerns about VMs wearing the SSDs, poor performance with RAIDz2 and needing raid10 mirrors. I’m worried about the mixed disk types on the HP servers, but I’m unsure if I’m overcomplicating things. How would you configure this cluster?

Also I have a QNAP and Synology with 10Gbe NFS for backups.


Main Server

• Model: HPE DL360 Gen10
  Processor: 2 x 24 core Xeon Platinum 8168
  Ram: 256GB
  Network: 2 x 10 Gbe + 4 x 1 Gbe
  Storage: 8 x 1TB Sata SSDs
  Hardware Raid: P308i-a SR Gen10

• Model: HPE DL360 Gen10
  Processor: 12 core Xeon Silver 4116
  Ram: 32GB
  Network: 1 x 10Gb + 4 x 1 Gbe
  Storage: 8 x 1.2TB SAS 10k
  Hardware Raid: P408i a SR Gen10

• Model: HPE DL360 Gen9
  Processor: 2 x 12 core Xeon E5-2670
  Ram: 128GB
  Network: 2 x 10Gbe + 4 x 1Gbe
  Storage: 8 x 2.4TB SAS 10k
  Hardware Raid: P440ar

• Model: HPE DL360p Gen8
  Processor: 2 x 6 core Xeon E5-2640
  Ram: 64GB
  Network: 4 x 1Gbe *2 x 10Gbe for $128
  Storage: 8 x 300GB SAS 15k
  Hardware Raid: P420

• Model: HPE DL360p Gen8
  Processor: 2 x 6 core Xeon E5-2640
  Ram: 80GB
  Network: 4 x 1Gbe *2 x 10Gbe for $128
  Storage: 8 x 1.8TB SAS 10k
  Hardware Raid: P420i

Current amount of VMs are 12 Linux and 2 Windows

 

[UdoB]

I'm looking to create a cluster with zfs replication on at least 3 HPE servers

You say "3" but you list five.

If you can actually run more than three then I would evaluate Ceph. With only three I would use ZFS with replication.

Most of the drives are HDDs, right? In that case I would highly recommend multiple ZFS mirrors (aka Raid10) with a supporting "Special Device" consisting of two (mirrored) fast (but small) SSD/NVMe.

That's my quick answer; of course ymmv...

 

[floh8]

which storage devices inside the synology?
Support your synology virtualization?

 

[swits]

You say "3" but you list five.

If you can actually run more than three then I would evaluate Ceph. With only three I would use ZFS with replication.

Most of the drives are HDDs, right? In that case I would highly recommend multiple ZFS mirrors (aka Raid10) with a supporting "Special Device" consisting of two (mirrored) fast (but small) SSD/NVMe.

That's my quick answer; of course ymmv...

Thanks for the response! I was looking at ZFS over Ceph mostly to keep it less complicated. Getting NVMe drives going on the DL360s might take a bit. Sounds like for the gen10 you can buy a 867982-B21 riser kit though. If I didn't have the nvme support would it be too slow to work for most VMs?

 

[UdoB]

If I didn't have the nvme support would it be too slow to work for most VMs?

ZFS with HDD without Special Device?

(( The difference between Sata SSD and PCIe NVMe is theoretically large. Practically it is not - in this context. The important aspect is the access time compared to the rotating spindles. Data access will take five milliseconds and it is nearly irrelevant if access to metadata is 100 times faster or a 10000 times. Oversimplified: with SATA you get 5.05 ms and with NVMe you get 5.0005 ms. See the difference? ))

It really depends on your actual workload and expectations. While a "normal" (whatever that means!) VM may run fine, a large database may just be unusable slow.

The good news is: you can just test it. (As long as only Mirrors are involved that SD can be added and removed on-the-fly.) Run without a Special Device and learn if it works okay for your use case. Then add that Special Device and shuffle around all data at least once to actually make use of it. I would run this test before going into production though...

Good luck