In a 3-node Proxmox Cluster, when one node is rebooted, VMs on other nodes experience momentary packet loss.

[EAMUNDSON]

I have a 3-node Proxmox VE cluster running CEPH and hosting VMs. Whenever I reboot one of the Proxmox nodes, VMs on the other nodes experience packet loss momentarily. This occurs even if the node I'm rebooting has no VMs on it.

I am using active-backup bonding, with Linux VLANs on top of the bond interface, and Linux bridges for the VM connectivity. The Bridges do not have STP enabled (as per default). The two bonded NICs are plugged into two HPE/Aruba switches, and have no special bonding configuration (no LACP). They are simple ports with VLANs trunked to them.

I spent this morning rebooting one node a few times and monitoring the network switches, and cannot find any reason in the switches that this would be happening. I'm starting to think this is some Proxmox/Linux network issue (configuration or otherwise). I opened a Proxmox support ticket, but I thought I'd ask the community if anyone has a clue what is happening.

Contents of one of the nodes interfaces file is below:
# network interface settings; autogenerated
# Please do NOT modify this file directly, unless you know what
# you're doing.
#
# If you want to manage parts of the network configuration manually,
# please utilize the 'source' or 'source-directory' directives to do
# so.
# PVE will preserve these directives, but will NOT read its network
# configuration from sourced files, so do not attempt to move any of
# the PVE managed interfaces into external files!

auto lo
iface lo inet loopback

iface ens6f0 inet manual

iface ens6f1 inet manual

auto ens6f2
iface ens6f2 inet manual

auto ens6f3
iface ens6f3 inet manual

auto ens2f0np0
iface ens2f0np0 inet manual

auto ens2f1np1
iface ens2f1np1 inet manual

auto bond0
iface bond0 inet manual
bond-slaves ens6f2 ens6f3
bond-miimon 100
bond-mode active-backup
bond-primary ens6f2
#Management Bond

auto bond1
iface bond1 inet manual
bond-slaves ens2f0np0 ens2f1np1
bond-miimon 100
bond-mode active-backup
bond-primary ens2f0np0
#VM Networks Bond

auto bond1.2
iface bond1.2 inet manual

auto bond1.10
iface bond1.10 inet manual

auto bond1.11
iface bond1.11 inet manual

auto bond1.12
iface bond1.12 inet manual

auto bond1.13
iface bond1.13 inet manual

auto bond1.15
iface bond1.15 inet manual

auto bond1.16
iface bond1.16 inet manual

auto bond1.19
iface bond1.19 inet manual

auto bond1.41
iface bond1.41 inet manual

auto bond1.42
iface bond1.42 inet manual

auto bond1.51
iface bond1.51 inet manual

auto bond1.8
iface bond1.8 inet manual

auto bond1.9
iface bond1.9 inet manual

auto bond1.14
iface bond1.14 inet manual

auto bond1.60
iface bond1.60 inet manual
#BASFRW-HA

auto vmbr0
iface vmbr0 inet static
address 10.107.130.73/24
gateway 10.107.130.1
bridge-ports bond0 ens6f0
bridge-stp off
bridge-fd 0
#Management Bridge

auto vmbr2
iface vmbr2 inet static
address 10.107.128.63/24
bridge-ports bond1.2
bridge-stp off
bridge-fd 0
#BAS Mgmt and CEPH

auto vmbr10
iface vmbr10 inet manual
bridge-ports bond1.10
bridge-stp off
bridge-fd 0
#BAS Systems

auto vmbr12
iface vmbr12 inet manual
bridge-ports bond1.12
bridge-stp off
bridge-fd 0
#Capstone

auto vmbr13
iface vmbr13 inet manual
bridge-ports bond1.13
bridge-stp off
bridge-fd 0
#Fire Systems

auto vmbr15
iface vmbr15 inet manual
bridge-ports bond1.15
bridge-stp off
bridge-fd 0
#Mech Controls

auto vmbr16
iface vmbr16 inet manual
bridge-ports bond1.16
bridge-stp off
bridge-fd 0
#Security Systems

auto vmbr19
iface vmbr19 inet manual
bridge-ports bond1.19
bridge-stp off
bridge-fd 0
#BAS Sensors

auto vmbr41
iface vmbr41 inet manual
bridge-ports bond1.41
bridge-stp off
bridge-fd 0
#Exterior Cameras

auto vmbr42
iface vmbr42 inet manual
bridge-ports bond1.42
bridge-stp off
bridge-fd 0
#Interior Cameras

auto vmbr51
iface vmbr51 inet manual
bridge-ports bond1.51
bridge-stp off
bridge-fd 0
#Controls Transit

auto vmbr8
iface vmbr8 inet manual
bridge-ports bond1.8
bridge-stp off
bridge-fd 0
#BAS Egress

auto vmbr9
iface vmbr9 inet manual
bridge-ports bond1.9
bridge-stp off
bridge-fd 0
#WAN Segment

auto vmbr11
iface vmbr11 inet manual
bridge-ports bond1.11
bridge-stp off
bridge-fd 0
#Electrical Controls

auto vmbr14
iface vmbr14 inet manual
bridge-ports bond1.14
bridge-stp off
bridge-fd 0
#EnSync

auto vmbr1
iface vmbr1 inet manual
bridge-ports bond1
bridge-stp off
bridge-fd 0
#BAS Management

auto vmbr3
iface vmbr3 inet manual
bridge-ports bond1.60
bridge-stp off
bridge-fd 0

 

[rachelwilson]

this is almost certainly the switches, not proxmox. when you reboot a node its ports flap, the arubas send spanning tree topology change notifications, and both switches flush their mac tables. traffic has to relearn for a few seconds which is exactly the momentary loss youre seeing on the other nodes. put the host-facing ports into edge/portfast mode on the aruba side so a host reboot stops triggering topology changes, then do another test reboot and watch the switch log for TCNs.

 

[EAMUNDSON]

this is almost certainly the switches, not proxmox. when you reboot a node its ports flap, the arubas send spanning tree topology change notifications, and both switches flush their mac tables. traffic has to relearn for a few seconds which is exactly the momentary loss youre seeing on the other nodes. put the host-facing ports into edge/portfast mode on the aruba side so a host reboot stops triggering topology changes, then do another test reboot and watch the switch log for TCNs.

I'm not disputing that it could be something in the switching. However - these ports are not running spanning-tree, so I find it troublesome if that is the problem. I do have a change window next week to put the ports into 'portfast' or 'edge-port' mode. They are currently not in 'edge' mode. I'll update ya'll when that's done.