Storage / cluster recommendations

[beckvo]

Hi,

first of all thanks to all the active forum members and staff for the great input on the forum. Been using PVE at work for quite some time and it's running well.

I have bought a used 4-node dell C6100 for cheap and am currenty figuring out what the best storage and storage network configuration would be.

Specs per node (four identical nodes):
2x Intel Xeon E55xx ~2GHz
48GB DDR3 RAM
Dual port 10Gbit SFP+ NIC
Dual port 1Gbit NIC onboard
Three LFF SATA (unfortunately no SAS) ports

I plan on using ceph storage. The idea is to balance the cluster between price and performance. Which disks would you recommend for this hardware configuraiton (4 nodes with 3 disks per node)?

1. Get 12x 2TB 7200rpm spinning disks, which would be cheap but not fast. Is this much worse performance-wise compared to a standard standalone hardware raid array?
2. Get 12x enterprise ssd (pm863 or similar intel), which would be fast but expensive. I've read the Ceph benchmark paper and understood that you should avoid consumer grade SSD. Although, using cheap SSDs would be better than only using spinning disks.
3. Get a mixture of both (for example two spinners and one ssd per node) and use the ssd for journal. This is the option I would go with.
4. Something completely different

The next part would be the (storage-) network configuration. I initially wanted to do a full mesh network so that I wouldn't have to buy a 10g switch. To my understanding this won't be possible because one needs n-1 NICs for the number of cluster nodes. Is there any way around buying a 10 Gigabit switch?

Looking forward to your answers!

Thanks
Volker

 

[ph0x]

ad 1. Ceph is very latency sensitive. Running it on HDDs will be not much fun.
ad 2. Cheap SSDs will be burned by VM images a lot faster, plus they usually do not hold higher IOPS over time, so the fun is also pretty limited. Buying enterprise SSDs will only make you cry once.
ad 3. This is probably the best balance

For a mesh network you need a direct connection to every other member, so three on your case. With two NICs you can only build a mesh for three nodes.
Did you already buy the SFP+ NICs? Those switches are indeed not cheap, 8-port switches RJ45 are already available for decent prices, though.

 

[beckvo]

Thanks for your help with making this decision, I appreciate it.

I did find some Samsung PM871 with 256gb capacity for cheap and will chose option no. 3:
Fortunately, the SSD can fit inside the chassis, so there is room for 3x HDD per node:
1x 256GB SSD for pve root + logical volumes for 3x Ceph OSD databases
3x 1TB SATA HDD (have found some laying around, otherwise I'd choose 2TB models)

For the network, I've already bought the 10G SFP+ nics since the server came with one pre-installed. Will try to find cheap SFP+ DAC cables and maybe use an existing 10g switch if possible.

So if there are 2x 10g and 2x 1g NICs, what would be the best network configuration? Should I separate the Ceph cluster network to an own 10g link? How does this look:
1x 10g Ceph public + cluster together
1x 10g Traffic for VMs and CTs
1x 1g Corosync ring 0
1x 1g Corosync ring 1

also possible, eventually better for performance:
1x 10g Ceph public
1x 10g Ceph cluster
1x 1g Corosync ring 0
1x 1g Corosync ring 1 + Traffic for VMs and CTs

 

[ph0x]

Here's what I did after reading a lot about Ceph network layout:
The two 10G NICs form a 802.3ad bond (named bond10) and together with one of the 1G NICs an active-backup bond (named bond0). On bond0 lives VLAN-aware bridge vmbr0 together with VLAN interfaces for management, Ceph public and Ceph cluster network.
Corosync's primary connection goes through the second 1G NIC with a secondary connection through Ceph cluster network.
This way either one of the switches can fail without interrupting the cluster or Ceph.