Hello all!
The setup: I’m running a small three-node Proxmox cluster with Ceph. Each node has a single NVMe OSD; Ceph itself is stable and behaving correctly.
Cluster and corosync traffic runs over a dedicated 2.5 GbE network (directly connected between nodes as a mesh - no switch). VM/LXC traffic uses vmbr0, backed by a bond consisting of the onboard 1 GbE NIC plus a USB Ethernet adapter on each node. Unfortunately due to a kernel bug I cannot yet rely on the motherboard NIC and I actually use the USB NIC as the primary link with 'eth0' as the secondary in an active-backup bond (bond0).
The issue is failure detection and HA behavior when the LAN-facing interface fails.
If 'bond0' (the LAN side backing vmbr0) goes down, the node remains fully reachable over the cluster network, corosync stays healthy, and the cluster remains quorate. From Proxmox’s perspective this is correct, but the practical result is that VMs/LXCs lose external connectivity while HA never triggers a failover. I cannot manually initiate migrations remotely because management access depends on the same failed LAN path. The only solution is to do pull the power cord and wait for the HA churn as things are finally moved around.
Adding USB NICs has reduced the likelihood of failure but doesn’t address the underlying design problem: loss of LAN connectivity is not treated as a node failure condition. This is something I didn’t account for in the original plan and I have since learned how it works through trial-and-error! The approaches I’ve considered so far are:
> "Run a routing daemon (FRR) on each node [...] and advertise a default route “upstream available” only from nodes whose bond is up and has gateway reachability."
Any guidance or pointers would be appreciated.
Thanks,
Corey
The setup: I’m running a small three-node Proxmox cluster with Ceph. Each node has a single NVMe OSD; Ceph itself is stable and behaving correctly.
Cluster and corosync traffic runs over a dedicated 2.5 GbE network (directly connected between nodes as a mesh - no switch). VM/LXC traffic uses vmbr0, backed by a bond consisting of the onboard 1 GbE NIC plus a USB Ethernet adapter on each node. Unfortunately due to a kernel bug I cannot yet rely on the motherboard NIC and I actually use the USB NIC as the primary link with 'eth0' as the secondary in an active-backup bond (bond0).
The issue is failure detection and HA behavior when the LAN-facing interface fails.
If 'bond0' (the LAN side backing vmbr0) goes down, the node remains fully reachable over the cluster network, corosync stays healthy, and the cluster remains quorate. From Proxmox’s perspective this is correct, but the practical result is that VMs/LXCs lose external connectivity while HA never triggers a failover. I cannot manually initiate migrations remotely because management access depends on the same failed LAN path. The only solution is to do pull the power cord and wait for the HA churn as things are finally moved around.
Adding USB NICs has reduced the likelihood of failure but doesn’t address the underlying design problem: loss of LAN connectivity is not treated as a node failure condition. This is something I didn’t account for in the original plan and I have since learned how it works through trial-and-error! The approaches I’ve considered so far are:
- A local health check that validates LAN reachability and, on failure, forces the node into maintenance mode or reboots it to trigger HA.
- Routing LAN traffic over the existing cluster/mesh network as a fallback (e.g., node-to-node forwarding into another node’s vmbr0).
- Collapsing cluster/corosync and LAN traffic into a single, highly redundant bond (2×2.5 GbE + onboard 1 GbE + USB), accepting shared fate.
> "Run a routing daemon (FRR) on each node [...] and advertise a default route “upstream available” only from nodes whose bond is up and has gateway reachability."
Any guidance or pointers would be appreciated.
Thanks,
Corey
