New proxmox cluster - config help

[MattiasS]

Hi,

Currently installing a new proxmox cluster to replace VMware and wanted to get some hints/pointers on the configuration.

Setup is as follows:
7 hosts
2x128GB M.2 SSD for OS per host
12x1,6TB SAS SSD per host
4x10Gb network interfaces

If I understand correctly, if you want shared storage and snapshot capability, ceph is pretty much the only way to go.
My plan is to go with a setup like the diagram attached but I am unsure about the network part, what would work best etc. so any
pointers in the right direction is hugely appreciated.

Thank you!

 

[LnxBil]

For decent ceph speed, the network is too slow. VLANs are not the recommended setup, although technically possible. Why did you use 2x dual LACP instead of LACP over all links? Can you add more or better NICs?

 

[MattiasS]

Thanks for reply LnxBil, no reason for the 2x dual LACP other than not knowing better, There is actually two more 10Gbps NIC's in each host, biggest problem right now is switchport availability but everything can be solved
Can you give me a quick recommendation on what would be the best setup / separation using 4x10Gbps NIC's and a scenario with 6x10Gbps NIC's ?

Thanks again.

 

[LnxBil]

Can you give me a quick recommendation on what would be the best setup / separation using 4x10Gbps NIC's and a scenario with 6x10Gbps NIC's ?

The official recommendation says at least 3 dedicated NICs:

• CEPH public
• CEPH private
• COROSYNC

I would add one for the "normal network stuff", as you did, so 4 are the minimum. With more NICs, I would LACP the things that need speed. According to the aforementioned recommendation, this is CEPH private (synchronization, replication, etc.), then CEPH public.

There are also reports, that you can run on LACP and share the links, yet you may run into timeouts, that will break your cluster. using DSCP could help so that you can share the corosync link over another maybe saturated link.

 

[MarkusKo]

make sure corosync has redundancy, at least 2 links, 1Gbit is enough, no lacp, no vlan, no bridge, different subnets