mesh network or not with ceph?

[benoitc]

I have 3 nodes with 2x1GB and 2x10GB ports , 64B GB or RAM , 2x 2To SAS (12G) disks with used for the ceph storage.

I have for now set proxmox using the Full Mesh Network configuration. But I would like to add redudancy to my network. Since I have 2 10G switch I am wondering if it would be OK to run everying over 1 active-passive 10G bond instead of meshing. The traffic ov vms would also go using that link. Or should I buy another dual-port NIC instead?

 

[aaron]

For better redundancy, 2 switches are for sure better. Can you stack them? Meaning, if for node x, link1 is down, can it still talk over link2 to the other nodes using the other switch?

We recommend separating different services into dedicated physical networks as much as possible, just to avoid and problems if one service is using all the bandwidth.
You should also take a look at your Corosync configuration (Proxmox VE cluster communication) and make sure it has its own dedicated physical network (best) or at least multiple links configured for it to be able to switch between them, should one become unusable. This is especially important if you use the Proxmox VE HA stack.

 

[benoitc]

For better redundancy, 2 switches are for sure better. Can you stack them? Meaning, if for node x, link1 is down, can it still talk over link2 to the other nodes using the other switch?

We recommend separating different services into dedicated physical networks as much as possible, just to avoid and problems if one service is using all the bandwidth.
You should also take a look at your Corosync configuration (Proxmox VE cluster communication) and make sure it has its own dedicated physical network (best) or at least multiple links configured for it to be able to switch between them, should one become unusable. This is especially important if you use the Proxmox VE HA stack.

OK, I see. Unfortunately switches are not stackable. Maybe I should use bond? Or can we have a configuration using IPs (L3) ?

I am thinking to do the following configuration:

2x1G : corosync + management VLAN
2x10G : active-backup bond, used for vm discussion. Or would a bridge with STP enough?
2x25G : in mesh used for the data network (ceph and maybe other).

Would it work? The card are 1025G so I have also the possibility to use a switch but ...