proxmox cluster and ceph network redundancy with 3 nodes

[benoitc]

This is a followup of another post but I simplify the problem, removing iscsi from the balance for now. What I am trying to understand is if this setup could work for CEPH and what to achieve network redundancy...

I have 3 nodes with 2 nic each (2x10GbE) . Each port is connected to a distinct 10G switch (all ports are 10G) following the current schema:



For now each switch is connected to the router and are not interconnected. Each nodes have the following properties

• CPU Intel® Xeon® Processor D-2166NT 12C/24T 2Ghz - 3Ghz
• RAM of 128GB
• 2 SSDs (2 x 480 GB) used for ceph (disk can be changed at some point)
• 2 nvme disks (256 GB) that are using the only possible PCI-E extension used for the system

So just to check again but would such configuration above is enough for CEPH even if it's not the fastest. Or better to try to achieve HA using a spare NAS or just replicate the vm when needed?

Right now the proxmox cluster handled by the switch 1 and CEPH cluster on the switch2. But then I have no redundancy: if one switch crash i am losing either the ceph cluster or the proxmox cluster. This what I am trying to setup.

At first I wanted to setup a bond with active-backup mode. Actually 2 bonds 1 on the vlan handling ceph with a priority on switch 2 and another for the proxmox cluster with a priority on the proxmox cluster. But it's not supported by the UI. Maybe manually? Is this supported by the kernel?

A second option would be setup a mesh network wich would require to replace the NVME PCIe extension by a 2x25G cards . Then I guess I will need to add another sata drive for the system, maybe one that can use these 2 NVMe disks?

What could be the other option? I am quite a newbie in networking, would a MSTP setup work? Is there any configuration I could look for it?

Any hint/ feedback is welcome

 

[Alwin]

So just to check again but would such configuration above is enough for CEPH even if it's not the fastest.

Depends on the workload, but in principle it works. The only downside you won't be able to achieve reliable HA. Since that requires extra NIC ports.
https://pve.proxmox.com/pve-docs/pve-admin-guide.html#chapter_ha_manager

At first I wanted to setup a bond with active-backup mode. Actually 2 bonds 1 on the vlan handling ceph with a priority on switch 2 and another for the proxmox cluster with a priority on the proxmox cluster. But it's not supported by the UI. Maybe manually? Is this supported by the kernel?

Well, manual setup can be done. But it still has the same issue with HA as above.

A second option would be setup a mesh network wich would require to replace the NVME PCIe extension by a 2x25G cards . Then I guess I will need to add another sata drive for the system, maybe one that can use these 2 NVMe disks?

That certainly increases performance. But same HA issue.

What could be the other option? I am quite a newbie in networking, would a MSTP setup work? Is there any configuration I could look for it?

More configuration to take care of. And still doesn't get you around the HA issue. One approach might be to add a quad port NIC and get corosync to run on a separate NIC port. Then you can utilize the other ports for VM/backup/migration and keep the 10 GbE for Ceph.

 

[benoitc]

I see.... I should have focus more on the ha part when i bought the hardware... would a 4 x1G be enough?

I wonder if using the 2nvme disks inside a sata enclosure works well but that would allow me to reuse them...

 

[Alwin]

I see.... I should have focus more on the ha part when i bought the hardware... would a 4 x1G be enough?

The bandwidth isn't so important for corosync, 1 GbE is enough. The factor is latency, corosync needs low and stable latency to provide accurate information about the state of the cluster.

I wonder if using the 2nvme disks inside a sata enclosure works well but that would allow me to reuse them...

Well, adapters are such a case. But it will quite definitely reduce the performance. Why not utilize the OCuLink?

 

[benoitc]

Well, adapters are such a case. But it will quite definitely reduce the performance. Why not utilize the OCuLink?

I didn't thought to this. I need to look how I can do it with this board and the case I have. thanks for the hint!

 

[benoitc]

The bandwidth isn't so important for corosync, 1 GbE is enough. The factor is latency, corosync needs low and stable latency to provide accurate information about the state of the cluster.

Rather than buying another card i am thinking i can reuse the ipmi port from supermicro (sideband) . Did someone try such thing?

 

[ph0x]

This won't work since the IPMI port is connected to the management subsystem and cannot be used for the network stack of the OS.
The port does not show up in your ethernet ports, does it?

 

[benoitc]

This won't work since the IPMI port is connected to the management subsystem and cannot be used for the network stack of the OS.
The port does not show up in your ethernet ports, does it?

true ... i misread the doc. so i guess my only choices are to use a dedicaced card or play with vlan+ qos.

 

[ph0x]

Should have gone for the TLN4F. ;-)

 

[benoitc]

Should have gone for the TLN4F. ;-)

true... but too late well i guess with some qos it should not be that bad...

 

[benoitc]

So I will go for an active-backup bond strategy + vlan as it sonds easier to go. The only thing I am asking myself is if I am adding an ethernet port using an USB adapter to put the main cororsync ring in. Wonder if it's something that could resist ofver the time compared to "just" using a vlan on the active interface with QOS for it. Has anyone had experience with it?