CEPH & ZFS Using a PERC H755n

[Taledo]

Bonjour all,

I'll be retiring 3 Dell 650XS from their current jobs soon™, and I would like to put them in a CEPH HA NVMe backed cluster.
Currently, those servers are running a SATA/SAS backplane with a PERC H755 running the show.
My plan would be to replace that with a NVMe backplane and a H755n controller. (and NVMe drives whenever the prices drop to acceptable levels...)

Does anyone here have feedback on using this card for ZFS/CEPH?

Best Regards,

Taledo

 

[jdancer]

As you know, ZFS & Ceph really don't play nice with RAID controllers like Dell PERCs.

While it's true you change the PERC from RAID mode to IT-mode, I've had issues in the past with the megaraid_sas driver. That's why I switch out Dell PERCs for Dell HBAs, like the Dell HBA355i. It uses the much simpler mpt3sas driver.

To get the lowest latency and highest throughput from NVMe, you'll want to connect the NVMes directly to PCIe lanes.

If the 650XS has rear drives, you'll want to ZFS RAID-1 Proxmox itself. Considered best practice to separate the OS from the data/VMs.

Been migrating VMware clusters to Dell 13th/14th-gen Dells at work to Proxmox Ceph using Dell HBA330s. No issues besides the typical drive & memory going bad bad and needing replacing.