Summary
Since installing corosync-qdevice on our 4-node cluster (an installation that involved several hard VM stops/restarts due to a botched netplan config), the Linux bridge vmbr0 on multiple nodes intermittently and persistently fails to forward unicast traffic to specific VM TAP interfaces — even though:
- the destination MAC address in the incoming Ethernet frame is correct
- the bridge's FDB (bridge fdb show) has a permanent entry mapping that MAC to the correct TAP device
- the VM itself is healthy, has the correct MAC configured in QEMU, and is actively sending ARP requests
- arping from the host to the VM IP succeeds (Layer 2 reachable from the host's own stack)
- there is no firewall (Proxmox firewall fully disabled, ebtables/nftables empty)
- proxy_arp is disabled everywhere
- the problem is not VM-specific: it follows whichever VM is "unlucky," moves between nodes after migration, and affects both Windows (virtio) and Linux (e1000) guests
- it affects multiple nodes in the same cluster independently
Environment
- Proxmox VE 9.2.3, kernel 7.0.6-2-pve
- pve-qemu-kvm 11.0.0-4
- 4-node cluster (wecon-cluster), Ceph storage, bond0 (active-backup) → vmbr0 for LAN, separate vmbr1 for Corosync/Ceph
- /etc/network/interfaces excerpt:
auto vmbr0
iface vmbr0 inet static
address 192.168.77.5/24
gateway 192.168.77.1
bridge-ports bond0
bridge-stp off
bridge-fd 0
- No VLANs on vmbr0 itself (vlan-aware not set), no SDN zones in use for this bridge
- No Proxmox firewall (pve-firewall status → disabled/running, /etc/pve/firewall/cluster.fw doesn't exist)
Timeline / Trigger
- Built a small dedicated VM for corosync-qdevice with two NICs (vmbr1 for corosync net, vmbr0 for LAN/updates).
- During Ubuntu Server install, the install process hung after a DHCP failure on the LAN NIC; the VM was hard-stopped (qm stop) and the install repeated from scratch on the same VMID.
- Shortly after, multiple unrelated VMs (Windows 11 guests, Linux guests) on other nodes started showing the Windows "no internet" globe icon / general unreachability over LAN.
- Two days of troubleshooting later, root cause was narrowed down to: the Linux bridge on the affected node(s) stops delivering unicast frames to specific TAP ports, despite a 100% correct, permanent FDB entry.
Diagnostic evidence
1. TAP MAC never matches the VM's configured MAC
# qm config 2406 | grep net0
net0: virtio=BC:24:11:E9
2:24,bridge=vmbr0,mtu=1500# ip link show tap2406i0
57: tap2406i0: ...
link/ether be:b4:f2:c2:d2:d6 brd ff:ff:ff:ff:ff:ff
This appears to be normal/expected Proxmox behavior (confirmed via code review of pve-bridge / PVE::Network::tap_plug — the MAC is never explicitly set on the host-side TAP via ip link set ... address; QEMU is expected to "own" the MAC inside the guest only). Under normal circumstances the bridge's MAC-learning is supposed to make this irrelevant. In our case, it doesn't.
2. FDB has the correct, permanent entry — but the bridge ignores it
# bridge fdb show | grep bc:24:11:e9:d2:24
bc:24:11:e9:d2:24 dev tap2406i0 vlan 1 master vmbr0 permanent
bc:24:11:e9:d2:24 dev tap2406i0 master vmbr0 permanent
3. Frames arrive at the bond/bridge with the correct destination MAC, but never reach the TAP
From pfSense (gateway, 192.168.77.1), pinging the affected VM (192.168.77.106):
# tcpdump -i bond0 -n -e 'icmp and host 192.168.77.106'
a8:b8:e0:02:e8:51 > bc:24:11:e9:d2:24, ethertype IPv4 ...: 192.168.77.1 > 192.168.77.106: ICMP echo request
```//confirmed identical capture on vmbr0 itself
```
# tcpdump -i tap2406i0 -n icmp
(nothing — zero packets, ever)
So: frame enters via bond0 → vmbr0 with the exact MAC that the FDB says belongs to tap2406i0 → frame is simply never forwarded to that port. No drops are logged anywhere (ebtables counters all zero, no nft rules, no dmesg errors).
4. arping from the host succeeds (host's own ARP stack can reach the VM), but the VM never receives unicast replies from the gateway
# arping -I vmbr0 -c3 192.168.77.106
Unicast reply from 192.168.77.106 [BC:24:11:E9
2:24] 0.7ms (x3)But ip neigh show for that IP shows STALE, and the VM's own outgoing ARP requests for the gateway are visible on the TAP, yet replies from the gateway never arrive at the TAP (same symptom as #3).
5. Workaround found: tearing down and rebuilding the bridge (
After ifdown vmbr0 && ifup vmbr0, all FDB entries are cleared. Restarting the affected VMs causes the bridge to relearn the MACs and traffic flows correctly — for a while. The problem reappears later (sometimes within hours, sometimes after a day) for a subset of VMs again, without any further configuration changes.
6. Not a Proxmox firewall / ebtables / nftables / proxy_arp / STP issue
# ebtables -L
Bridge table: filter
Bridge chain: INPUT, entries: 0, policy: ACCEPT
Bridge chain: FORWARD, entries: 0, policy: ACCEPT
Bridge chain: OUTPUT, entries: 0, policy: ACCEPT
# nft list ruleset
(empty)
# sysctl net.ipv4.conf.vmbr0.proxy_arp
net.ipv4.conf.vmbr0.proxy_arp = 0
# cat /sys/class/net/vmbr0/bridge/stp_state
0
7. Cluster-wide, not single-node
The same symptom (host cannot ping its own locally-running VMs/CTs over vmbr0, despite correct FDB) was independently reproduced on a second node in the same cluster, with completely different VMs/MACs.
What we've already tried (no lasting effect)
- Manually correcting TAP MAC via ip link set tapXi0 address <correct-mac> (works for ~minutes, doesn't fix the actual forwarding issue, which is bridge-internal, not MAC-mismatch related)
- A post-start hookscript that sets TAP/fwln/fwpr interface MACs to match the VM's configured MAC immediately after start (helps temporarily, same as above)
- Manually adding static FDB entries (bridge fdb add ... master static) — no effect, bridge still doesn't forward
- Disabling Proxmox firewall entirely on affected VMs and cluster-wide
- ip link set tapXi0 nomaster && ip link set tapXi0 master vmbr0 — no effect
- Full ifdown vmbr0 && ifup vmbr0 — works temporarily, not a real fix, and momentarily drops the node off the network
- apt full-upgrade to latest 9.2.3 / pve-qemu-kvm 11.0.0-4 — no change
- Ruled out: STP, proxy_arp, ebtables, nftables, rp_filter (set to 2/loose), VLAN filtering (off), bridge ageing_time (default 300s), Ceph/Corosync network is unaffected and stable throughout (separate vmbr1, separate physical NIC)
Ruled out (additional)
- bridge kernel module is built-in (modinfo bridge → filename: (builtin)), confirmed present in modules.builtin along with br_netfilter — not a missing-module issue.
Questions for the community / devs
- Is there a known kernel bug (possibly related to bridge FDB / br_fdb_update or netfilter bridge hooks) in kernel 7.0.6-2-pve that can cause the bridge to silently stop forwarding to a port despite a correct, permanent FDB entry?
- Could the qdevice VM's second NIC (also on vmbr0, used only for LAN/internet access for apt) have somehow corrupted the bridge's internal port/FDB state during the botched install (multiple hard qm stop cycles while the TAP was mid-negotiation)? We have since stopped that VM/NIC entirely and the issue persists on completely unrelated VMs/ports, so this seems unlikely to be the direct ongoing cause, but might have been the trigger.
- Anyone else seen a Linux bridge "lose" a working FDB entry's actual forwarding behavior (entry stays in bridge fdb show, but frames to that MAC are silently dropped) under PVE 9.x / kernel 7.0.x?
- Is ethtool -K bond0 tx-scatter-gather-fraglist off (currently shows [requested on] but effectively off [fixed] upstream — seen in our ethtool -k output) or any other offload setting on the bonded NICs (Intel, mixed 10G/1G slaves in active-backup mode) known to interact badly with bridging in this kernel?
Happy to provide full qm config, ethtool -i/-k for both bond members, dmesg, and pveversion -v if useful. This has been ongoing for over a week and is affecting production VMs across the cluster intermittently in a way that is very hard to pin down because the only reliable "fix" (bridge teardown/rebuild) is itself disruptive.