We run a proxmox ve cluster with 12 nodes (pve-manager/9.2.21, Linux 7.0.14-24-pve (2026-10-09T11:32Z)). We also have a proxmox backup server running on a separate network. There are three network switches between the proxmox servers.
Two weeks ago, we shut down 40 of our 114 virtual machines. After about three hours, we started most of them again.
Since then, we have been experiencing network spikes when the VMs are running(only on one network: vmbr2). We were able to stabilize the network by leaving three VMs shut down.
To investigate, we started the VMs one at a time, but the issue persisted. We disabled networking on the VMs, but that did not resolve it. When we removed their network adapters entirely, the issue disappeared. Changing the MAC addresses didn’t help.
One vm, one of the VMs that we believed was causing issues, is running currently with no issues. We do not know why.
We also found that a backup job for one of the three affected vms was still enabled while the VM was shut down. That also seemed to cause network issues, because it appears the backup job starts the VM in the background while running the backup. (See the dmesg logs below in this comment)
I would like to understand what is causing this, but I have not been able to identify the cause. Starting the affected VMs to troubleshoot is hard because it disrupts our production network.
Has anyone any suggestions for what to investigate? I would like to prevent something like this in the feature.
The VM config of one of the affected VMs:
agent: 1
bios: seabios
boot: order=scsi0
cores: 2
cpu: x86-64-v2-AES
ide0: none,media=cdrom
memory: 2048
meta: creation-qemu=9.2.0,ctime=1334383037
name: faulty-server.example.nl
net0: virtio=00:0c:29:22:22:22,bridge=vmbr0,firewall=1
net1: virtio=00:0c:29:11:11:11,bridge=vmbr1,firewall=1
net2: virtio=00:0c:29:33:33:33,bridge=vmbr4,firewall=1
ostype: l26
scsi0: 12s1:vm-599-disk-0,size=340G
scsihw: virtio-scsi-single
smbios1: uuid=xxxx-xxxx-xxxx-xxxx-xxxx
sockets: 2
vmgenid: xxx-xxx-xx-xx-xxxx
Networking on the proxmox with one faulty vm. Rest of the proxmox servers have similar networking; it might have other vmbr interfaces. But the network on vmbr2 is the same on all proxmox machines:
auto vmbr0
iface vmbr0 inet static
address 192.168.178.12/24
gateway 192.168.178.1
bridge-ports eno1
bridge-stp off
bridge-fd 0
#192.168.178
auto vmbr1
iface vmbr1 inet static
address 10.2.23.12/24
bridge-ports eno3
bridge-stp off
bridge-fd 0
#10
auto vmbr2
iface vmbr2 inet manual
bridge-ports eno2
bridge-stp off
bridge-fd 0
#192.168.12
auto vmbr3
iface vmbr3 inet manual
bridge-ports eno4
bridge-stp off
bridge-fd 0
#192.168.33
auto vmbr4
iface vmbr4 inet manual
bridge-ports eno5
bridge-stp off
bridge-fd 0
#10.10.10
The dmesg log of when the backup for the faulty VM was started at midnight:
[23:31:07 2026] tap599i0: entered promiscuous mode
[23:31:07 2026] vmbr0: port 8(fwpr599p0) entered blocking state
[23:31:07 2026] vmbr0: port 8(fwpr599p0) entered disabled state
[23:31:07 2026] fwpr599p0: entered allmulticast mode
[23:31:07 2026] fwpr599p0: entered promiscuous mode
[23:31:07 2026] vmbr0: port 8(fwpr599p0) entered blocking state
[23:31:07 2026] vmbr0: port 8(fwpr599p0) entered forwarding state
[23:31:07 2026] fwbr599i0: port 1(fwln599i0) entered blocking state
[23:31:07 2026] fwbr599i0: port 1(fwln599i0) entered disabled state
[23:31:07 2026] fwln599i0: entered allmulticast mode
[23:31:07 2026] fwln599i0: entered promiscuous mode
[23:31:07 2026] fwbr599i0: port 1(fwln599i0) entered blocking state
[23:31:07 2026] fwbr599i0: port 1(fwln599i0) entered forwarding state
[23:31:07 2026] fwbr599i0: port 2(tap599i0) entered blocking state
[23:31:07 2026] fwbr599i0: port 2(tap599i0) entered disabled state
[23:31:07 2026] tap599i0: entered allmulticast mode
[23:31:07 2026] fwbr599i0: port 2(tap599i0) entered blocking state
[23:31:07 2026] fwbr599i0: port 2(tap599i0) entered forwarding state
[23:31:08 2026] tap599i1: entered promiscuous mode
[23:31:08 2026] vmbr1: port 11(fwpr599p1) entered blocking state
[23:31:08 2026] vmbr1: port 11(fwpr599p1) entered disabled state
[23:31:08 2026] fwpr599p1: entered allmulticast mode
[23:31:08 2026] fwpr599p1: entered promiscuous mode
[23:31:08 2026] vmbr1: port 11(fwpr599p1) entered blocking state
[23:31:08 2026] vmbr1: port 11(fwpr599p1) entered forwarding state
[23:31:08 2026] fwbr599i1: port 1(fwln599i1) entered blocking state
[23:31:08 2026] fwbr599i1: port 1(fwln599i1) entered disabled state
[23:31:08 2026] fwln599i1: entered allmulticast mode
[23:31:08 2026] fwln599i1: entered promiscuous mode
[23:31:08 2026] fwbr599i1: port 1(fwln599i1) entered blocking state
[23:31:08 2026] fwbr599i1: port 1(fwln599i1) entered forwarding state
[23:31:08 2026] fwbr599i1: port 2(tap599i1) entered blocking state
[23:31:08 2026] fwbr599i1: port 2(tap599i1) entered disabled state
[23:31:08 2026] tap599i1: entered allmulticast mode
[23:31:08 2026] fwbr599i1: port 2(tap599i1) entered blocking state
[23:31:08 2026] fwbr599i1: port 2(tap599i1) entered forwarding state
[23:31:09 2026] tap599i2: entered promiscuous mode
[23:31:09 2026] vmbr4: port 3(fwpr599p2) entered blocking state
[23:31:09 2026] vmbr4: port 3(fwpr599p2) entered disabled state
[23:31:09 2026] fwpr599p2: entered allmulticast mode
[23:31:09 2026] fwpr599p2: entered promiscuous mode
[23:31:09 2026] vmbr4: port 3(fwpr599p2) entered blocking state
[23:31:09 2026] vmbr4: port 3(fwpr599p2) entered forwarding state
[23:31:09 2026] fwbr599i2: port 1(fwln599i2) entered blocking state
[23:31:09 2026] fwbr599i2: port 1(fwln599i2) entered disabled state
[23:31:09 2026] fwln599i2: entered allmulticast mode
[23:31:09 2026] fwln599i2: entered promiscuous mode
[23:31:09 2026] fwbr599i2: port 1(fwln599i2) entered blocking state
[23:31:09 2026] fwbr599i2: port 1(fwln599i2) entered forwarding state
[23:31:09 2026] fwbr599i2: port 2(tap599i2) entered blocking state
[23:31:09 2026] fwbr599i2: port 2(tap599i2) entered disabled state
[23:31:09 2026] tap599i2: entered allmulticast mode
[23:31:09 2026] fwbr599i2: port 2(tap599i2) entered blocking state
[23:31:09 2026] fwbr599i2: port 2(tap599i2) entered forwarding state
[00:44:11 2026] tap599i0: left allmulticast mode
[00:44:11 2026] fwbr599i0: port 2(tap599i0) entered disabled state
[00:44:11 2026] fwbr599i0: port 1(fwln599i0) entered disabled state
[00:44:11 2026] vmbr0: port 8(fwpr599p0) entered disabled state
[00:44:11 2026] fwln599i0 (unregistering): left allmulticast mode
[00:44:11 2026] fwln599i0 (unregistering): left promiscuous mode
[00:44:11 2026] fwbr599i0: port 1(fwln599i0) entered disabled state
[00:44:11 2026] fwpr599p0 (unregistering): left allmulticast mode
[00:44:11 2026] fwpr599p0 (unregistering): left promiscuous mode
[00:44:11 2026] vmbr0: port 8(fwpr599p0) entered disabled state
[00:44:11 2026] tap599i1: left allmulticast mode
[00:44:11 2026] fwbr599i1: port 2(tap599i1) entered disabled state
[00:44:11 2026] fwbr599i1: port 1(fwln599i1) entered disabled state
[00:44:11 2026] vmbr1: port 11(fwpr599p1) entered disabled state
[00:44:11 2026] fwln599i1 (unregistering): left allmulticast mode
[00:44:11 2026] fwln599i1 (unregistering): left promiscuous mode
[00:44:11 2026] fwbr599i1: port 1(fwln599i1) entered disabled state
[00:44:11 2026] fwpr599p1 (unregistering): left allmulticast mode
[00:44:11 2026] fwpr599p1 (unregistering): left promiscuous mode
[00:44:11 2026] vmbr1: port 11(fwpr599p1) entered disabled state
[00:44:11 2026] tap599i2: left allmulticast mode
[00:44:11 2026] fwbr599i2: port 2(tap599i2) entered disabled state
[00:44:11 2026] fwbr599i2: port 1(fwln599i2) entered disabled state
[00:44:11 2026] vmbr4: port 3(fwpr599p2) entered disabled state
[00:44:11 2026] fwln599i2 (unregistering): left allmulticast mode
[00:44:11 2026] fwln599i2 (unregistering): left promiscuous mode
[00:44:11 2026] fwbr599i2: port 1(fwln599i2) entered disabled state
[00:44:11 2026] fwpr599p2 (unregistering): left allmulticast mode
[00:44:11 2026] fwpr599p2 (unregistering): left promiscuous mode
[00:44:11 2026] vmbr4: port 3(fwpr599p2) entered disabled state
Two weeks ago, we shut down 40 of our 114 virtual machines. After about three hours, we started most of them again.
Since then, we have been experiencing network spikes when the VMs are running(only on one network: vmbr2). We were able to stabilize the network by leaving three VMs shut down.
To investigate, we started the VMs one at a time, but the issue persisted. We disabled networking on the VMs, but that did not resolve it. When we removed their network adapters entirely, the issue disappeared. Changing the MAC addresses didn’t help.
One vm, one of the VMs that we believed was causing issues, is running currently with no issues. We do not know why.
We also found that a backup job for one of the three affected vms was still enabled while the VM was shut down. That also seemed to cause network issues, because it appears the backup job starts the VM in the background while running the backup. (See the dmesg logs below in this comment)
I would like to understand what is causing this, but I have not been able to identify the cause. Starting the affected VMs to troubleshoot is hard because it disrupts our production network.
Has anyone any suggestions for what to investigate? I would like to prevent something like this in the feature.
The VM config of one of the affected VMs:
agent: 1
bios: seabios
boot: order=scsi0
cores: 2
cpu: x86-64-v2-AES
ide0: none,media=cdrom
memory: 2048
meta: creation-qemu=9.2.0,ctime=1334383037
name: faulty-server.example.nl
net0: virtio=00:0c:29:22:22:22,bridge=vmbr0,firewall=1
net1: virtio=00:0c:29:11:11:11,bridge=vmbr1,firewall=1
net2: virtio=00:0c:29:33:33:33,bridge=vmbr4,firewall=1
ostype: l26
scsi0: 12s1:vm-599-disk-0,size=340G
scsihw: virtio-scsi-single
smbios1: uuid=xxxx-xxxx-xxxx-xxxx-xxxx
sockets: 2
vmgenid: xxx-xxx-xx-xx-xxxx
Networking on the proxmox with one faulty vm. Rest of the proxmox servers have similar networking; it might have other vmbr interfaces. But the network on vmbr2 is the same on all proxmox machines:
auto vmbr0
iface vmbr0 inet static
address 192.168.178.12/24
gateway 192.168.178.1
bridge-ports eno1
bridge-stp off
bridge-fd 0
#192.168.178
auto vmbr1
iface vmbr1 inet static
address 10.2.23.12/24
bridge-ports eno3
bridge-stp off
bridge-fd 0
#10
auto vmbr2
iface vmbr2 inet manual
bridge-ports eno2
bridge-stp off
bridge-fd 0
#192.168.12
auto vmbr3
iface vmbr3 inet manual
bridge-ports eno4
bridge-stp off
bridge-fd 0
#192.168.33
auto vmbr4
iface vmbr4 inet manual
bridge-ports eno5
bridge-stp off
bridge-fd 0
#10.10.10
The dmesg log of when the backup for the faulty VM was started at midnight:
[23:31:07 2026] tap599i0: entered promiscuous mode
[23:31:07 2026] vmbr0: port 8(fwpr599p0) entered blocking state
[23:31:07 2026] vmbr0: port 8(fwpr599p0) entered disabled state
[23:31:07 2026] fwpr599p0: entered allmulticast mode
[23:31:07 2026] fwpr599p0: entered promiscuous mode
[23:31:07 2026] vmbr0: port 8(fwpr599p0) entered blocking state
[23:31:07 2026] vmbr0: port 8(fwpr599p0) entered forwarding state
[23:31:07 2026] fwbr599i0: port 1(fwln599i0) entered blocking state
[23:31:07 2026] fwbr599i0: port 1(fwln599i0) entered disabled state
[23:31:07 2026] fwln599i0: entered allmulticast mode
[23:31:07 2026] fwln599i0: entered promiscuous mode
[23:31:07 2026] fwbr599i0: port 1(fwln599i0) entered blocking state
[23:31:07 2026] fwbr599i0: port 1(fwln599i0) entered forwarding state
[23:31:07 2026] fwbr599i0: port 2(tap599i0) entered blocking state
[23:31:07 2026] fwbr599i0: port 2(tap599i0) entered disabled state
[23:31:07 2026] tap599i0: entered allmulticast mode
[23:31:07 2026] fwbr599i0: port 2(tap599i0) entered blocking state
[23:31:07 2026] fwbr599i0: port 2(tap599i0) entered forwarding state
[23:31:08 2026] tap599i1: entered promiscuous mode
[23:31:08 2026] vmbr1: port 11(fwpr599p1) entered blocking state
[23:31:08 2026] vmbr1: port 11(fwpr599p1) entered disabled state
[23:31:08 2026] fwpr599p1: entered allmulticast mode
[23:31:08 2026] fwpr599p1: entered promiscuous mode
[23:31:08 2026] vmbr1: port 11(fwpr599p1) entered blocking state
[23:31:08 2026] vmbr1: port 11(fwpr599p1) entered forwarding state
[23:31:08 2026] fwbr599i1: port 1(fwln599i1) entered blocking state
[23:31:08 2026] fwbr599i1: port 1(fwln599i1) entered disabled state
[23:31:08 2026] fwln599i1: entered allmulticast mode
[23:31:08 2026] fwln599i1: entered promiscuous mode
[23:31:08 2026] fwbr599i1: port 1(fwln599i1) entered blocking state
[23:31:08 2026] fwbr599i1: port 1(fwln599i1) entered forwarding state
[23:31:08 2026] fwbr599i1: port 2(tap599i1) entered blocking state
[23:31:08 2026] fwbr599i1: port 2(tap599i1) entered disabled state
[23:31:08 2026] tap599i1: entered allmulticast mode
[23:31:08 2026] fwbr599i1: port 2(tap599i1) entered blocking state
[23:31:08 2026] fwbr599i1: port 2(tap599i1) entered forwarding state
[23:31:09 2026] tap599i2: entered promiscuous mode
[23:31:09 2026] vmbr4: port 3(fwpr599p2) entered blocking state
[23:31:09 2026] vmbr4: port 3(fwpr599p2) entered disabled state
[23:31:09 2026] fwpr599p2: entered allmulticast mode
[23:31:09 2026] fwpr599p2: entered promiscuous mode
[23:31:09 2026] vmbr4: port 3(fwpr599p2) entered blocking state
[23:31:09 2026] vmbr4: port 3(fwpr599p2) entered forwarding state
[23:31:09 2026] fwbr599i2: port 1(fwln599i2) entered blocking state
[23:31:09 2026] fwbr599i2: port 1(fwln599i2) entered disabled state
[23:31:09 2026] fwln599i2: entered allmulticast mode
[23:31:09 2026] fwln599i2: entered promiscuous mode
[23:31:09 2026] fwbr599i2: port 1(fwln599i2) entered blocking state
[23:31:09 2026] fwbr599i2: port 1(fwln599i2) entered forwarding state
[23:31:09 2026] fwbr599i2: port 2(tap599i2) entered blocking state
[23:31:09 2026] fwbr599i2: port 2(tap599i2) entered disabled state
[23:31:09 2026] tap599i2: entered allmulticast mode
[23:31:09 2026] fwbr599i2: port 2(tap599i2) entered blocking state
[23:31:09 2026] fwbr599i2: port 2(tap599i2) entered forwarding state
[00:44:11 2026] tap599i0: left allmulticast mode
[00:44:11 2026] fwbr599i0: port 2(tap599i0) entered disabled state
[00:44:11 2026] fwbr599i0: port 1(fwln599i0) entered disabled state
[00:44:11 2026] vmbr0: port 8(fwpr599p0) entered disabled state
[00:44:11 2026] fwln599i0 (unregistering): left allmulticast mode
[00:44:11 2026] fwln599i0 (unregistering): left promiscuous mode
[00:44:11 2026] fwbr599i0: port 1(fwln599i0) entered disabled state
[00:44:11 2026] fwpr599p0 (unregistering): left allmulticast mode
[00:44:11 2026] fwpr599p0 (unregistering): left promiscuous mode
[00:44:11 2026] vmbr0: port 8(fwpr599p0) entered disabled state
[00:44:11 2026] tap599i1: left allmulticast mode
[00:44:11 2026] fwbr599i1: port 2(tap599i1) entered disabled state
[00:44:11 2026] fwbr599i1: port 1(fwln599i1) entered disabled state
[00:44:11 2026] vmbr1: port 11(fwpr599p1) entered disabled state
[00:44:11 2026] fwln599i1 (unregistering): left allmulticast mode
[00:44:11 2026] fwln599i1 (unregistering): left promiscuous mode
[00:44:11 2026] fwbr599i1: port 1(fwln599i1) entered disabled state
[00:44:11 2026] fwpr599p1 (unregistering): left allmulticast mode
[00:44:11 2026] fwpr599p1 (unregistering): left promiscuous mode
[00:44:11 2026] vmbr1: port 11(fwpr599p1) entered disabled state
[00:44:11 2026] tap599i2: left allmulticast mode
[00:44:11 2026] fwbr599i2: port 2(tap599i2) entered disabled state
[00:44:11 2026] fwbr599i2: port 1(fwln599i2) entered disabled state
[00:44:11 2026] vmbr4: port 3(fwpr599p2) entered disabled state
[00:44:11 2026] fwln599i2 (unregistering): left allmulticast mode
[00:44:11 2026] fwln599i2 (unregistering): left promiscuous mode
[00:44:11 2026] fwbr599i2: port 1(fwln599i2) entered disabled state
[00:44:11 2026] fwpr599p2 (unregistering): left allmulticast mode
[00:44:11 2026] fwpr599p2 (unregistering): left promiscuous mode
[00:44:11 2026] vmbr4: port 3(fwpr599p2) entered disabled state