Interface Bonding of Storage and Network Traffic NIC's

[nqnz]

In our 3 node cluster we currently have 2 10G ports that are bonded, and then we set separate VLAN's on the bonded NIC for Ceph Storage traffic (say vlan 1) and proxmox VM traffic (say vlan 2). So we have bond0.1 and bond0.2 traffic.

I was under the impression this was not ideal as the bonded interface would introduce some noise to the Ceph Storage network even though its VLAN'd off. Is this actually fine? I feel like separate NICs for the Ceph storage traffic and VM traffic here would be better than the single bonded interface. Just trying to wrap my head around why this would or wouldn't be a good setup.

 

[Dunuin]

Do you got a dedicated Gbit NIC for corosync?

 

[alexskysilk]

it is possible to operate this way, and even relatively reliably if you separate masters for the bonds, eg:

auto bond0
iface bond0 inet static
bond-slaves eth0.1 eth1.1
bond-mode active-backup
...

auto bond1
iface bond1 inet static
bond-slaves eth1.2 eth0.2
bond-mode active-backup
...

 

[nqnz]

Do you got a dedicated Gbit NIC for corosync?

No, we do not. The only NICs on the box are the 2 10G, and they're being bonded. There is actually a third VLAN on the bonded interface as well but its for creating an internal VM network.

 

[nqnz]

it is possible to operate this way, and even relatively reliably if you separate masters for the bonds, eg:

auto bond0
iface bond0 inet static
bond-slaves eth0.1 eth1.1
bond-mode active-backup
...

auto bond1
iface bond1 inet static
bond-slaves eth1.2 eth0.2
bond-mode active-backup
...

This seems pretty clever, thanks for the tip. Also making me look at the bond-mode of the current NICs with is 802.3ad, so again I'd say less than ideal.

 

[Dunuin]

No, we do not. The only NICs on the box are the 2 10G, and they're being bonded. There is actually a third VLAN on the bonded interface as well but its for creating an internal VM network.

From the documentation about corosync network requirements: https://pve.proxmox.com/pve-docs/pve-admin-guide.html#_cluster_network

5.7. Cluster Network​

The cluster network is the core of a cluster. All messages sent over it have to be delivered reliably to all nodes in their respective order. In Proxmox VE this part is done by corosync, an implementation of a high performance, low overhead, high availability development toolkit. It serves our decentralized configuration file system (pmxcfs).

5.7.1. Network Requirements​

This needs a reliable network with latencies under 2 milliseconds (LAN performance) to work properly. The network should not be used heavily by other members; ideally corosync runs on its own network. Do not use a shared network for corosync and storage (except as a potential low-priority fallback in a redundant configuration).

So basically you don't want corosync and CEPH on the same network because CEPH could saturate the network, the latencies will increase and maybe corosync will be too slow, you loose quorum and your nodes will reboot.

 

[nqnz]

From the documentation about corosync network requirements: https://pve.proxmox.com/pve-docs/pve-admin-guide.html#_cluster_network



So basically you don't want corosync and CEPH on the same network because CEPH could saturate the network, the latencies will increase and maybe corosync will be too slow, you loose quorum and your nodes will reboot.

Thanks for the reply, I guess I was just hoping this VLAN setup was enough to separate CEPH from corosync. Seems like this was hopeful. I'll be pushing to get this changed and give them each their own separate, physical, NICs.