NVMe support/experiences

[brad_mssw]

We're currently building out a new 3-node datacenter that uses shared Ceph + Proxmox nodes. We've done this in the past using a classic LSI SAS3 RAID controller attached to a SAS3 backplane extender using all Intel DC S3700 SSDs (in JBOD mode) and have had good success.

Now, with the new datacenter, when pricing it out, for about 15% more we can go with an NVMe based solution (however, we would continue to use a SAS3 RAID1 mirror for the boot disks, NVMe would only be used as Ceph OSDs). We plan on using either Dual 10GbE SFP+, or 40GbE QSFP+ for node interconnects.

What we're looking for is first ... is NVMe supported at all, and if so, is it considered stable? Next would be any experiences anyone has had. Finally would be, would we see any actual performance benefit from NVMe vs SATA/SAS connected SSDs when using Ceph, or would Ceph/Network overhead get rid of any performance benefits?

For the SSDs, we'd go with 7-8 Intel P3600 2.5 NVMe disks ... or for a non-NVMe solution, the Intel DC S3610's.

The NVMe solution would be based on the SuperMicro 2028U-TN24R4T:
https://www.supermicro.com/products/system/2U/2028/SYS-2028U-TN24R4T_.cfm

Compared to a more classic approach like the SuperMicro 2028R-E1CR24H:
https://www.supermicro.com/products/system/2U/2028/SSG-2028R-E1CR24H.cfm

Thanks for any insight anyone can provide!

 

[fabian]

in general NVMe support is already included out of the box on PVE systems.

there seem to be two exceptions:

• some mainboards don't allow booting from NVMe devices (including as log/cache in zpools)
• multipath and some NVMe devices don't play well together

the latter is currently being investigated, but should not affect your setup since you don't use multipath.

 

[brad_mssw]

I don't suppose anyone has any experiences, or know if the additional investment in NVMe is worthwhile when used with Ceph? Clearly NVMe is faster, but Ceph has non-negligible overhead which may make this moot (but I don't have any benchmarks to know), and since I've never used NVMe, its "experimental" in my mind at this point.

I was planning on sticking with LSI SAS3 RAID1 for my boot disks, so I won't be using NVMe for booting.

Thanks!

 

[czechsys]

I have real experience with Ceph+Proxmox 4.3+NVMe over 10Gbps, we are testing new storage setup for Proxmox. You are lucky with full NVMe Supermicro Too bad, CEO dont accepted full NVMe nodes
So, our config for tests:
2x SM 1028-TNRTP+, every node: 2x NVMe P3600 1.2TB, 32GB RAM, 2x E5-2623 @ 2.6GHz, 2x 10G SFTP+ DAC vs switch in LACP
Proxmox 4.3 with openvswitch
Ceph Jewel, xfs OSDs, 9k MTU, replica=2
hyperconverged setup

I am novice in Ceph, so my setup wasn't top performance, anyway, data from fio inside VM (Debian Jessie), all tests was against one OSD even with more VMs running. Fio 4kB blocks, 4GB file, read/write/(read/write 75/25) in kIOPS:
1xVM: 110/11.3/(31.4/10.5)
4xVM (per VM): 46/6.5/(9/3)

Compare with NFSv3: 1xVM: 161/153/(64/21)
Pure ext4 on SM: 302/291/(167/55)

rbd bench (64 PGs) -b 4G 60 write -t 32:
avg IOPS 261, max bw 1480MBps, bw 1045 MBps, avg latency 0.1s, sttdev latency 0.09

 

[brad_mssw]

The raw read iops are impressive ... that said, I see higher iops for write on my current cluster, but its also spread across more OSDs. I need to re-evaluate that I guess. Its possible we wouldn't see any real improvement due to Ceph overhead.

 

[fabian]

The raw read iops are impressive ... that said, I see higher iops for write on my current cluster, but its also spread across more OSDs. I need to re-evaluate that I guess. Its possible we wouldn't see any real improvement due to Ceph overhead.

I would guess so too, although the overhead might get smaller with Jewel (and likely with Kraken if Bluestore becomes a viable option then!).