br_netfilter / PVE-firewall crashes after update to PVE9

[alpeh1974]

Hi there,

first I have to say THANK YOU to the Proxmox People and this nice forum for the great support everyone finds here.
Unfortunately I now ran into an issue that looks like the error described here, but I did not found a solution so far:

After upgrading from latest PVE 8 to 9 we get the following crash on booting (tested with kernel 6.14 and latest 6.8 from PVE 8, no difference...).
Network does not work. Only completely disabling pve-firewall (setting enable:0 in /etc/pve/firewall/cluster.fw) solves this problem = no error message any more and network works without any problem.

Thank you in advance for any hint!

Aug 16 14:39:48 optimus kernel: ------------[ cut here ]------------
Aug 16 14:39:48 optimus kernel: WARNING: CPU: 22 PID: 4160 at net/bridge/br_netfilter_hooks.c:622 br_nf_local_in+0x1b9/0x1e0
Aug 16 14:39:48 optimus kernel: Modules linked in: veth ebt_arp ebtable_filter ebtables ip6table_raw ip6t_REJECT nf_reject_ipv6 ip6table_filter ip6_tables iptable_raw xt_s>
Aug 16 14:39:48 optimus kernel:  btrfs blake2b_generic hid_generic usbkbd usbhid hid raid1 dm_raid raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor r>
Aug 16 14:39:48 optimus kernel: CPU: 22 PID: 4160 Comm: vhost-4127 Tainted: P           O       6.8.12-13-pve #1
Aug 16 14:39:48 optimus kernel: Hardware name: Supermicro Super Server/H12SSL-i, BIOS 3.3 03/28/2025
Aug 16 14:39:48 optimus kernel: RIP: 0010:br_nf_local_in+0x1b9/0x1e0
Aug 16 14:39:48 optimus kernel: Code: df e8 9b 98 da ff 66 83 ab b8 00 00 00 08 eb 92 be 04 00 00 00 48 89 df e8 84 98 da ff 66 83 ab b8 00 00 00 04 e9 78 ff ff ff <0f> 0b>
Aug 16 14:39:48 optimus kernel: RSP: 0018:ffffbc38ff0877a8 EFLAGS: 00010202
Aug 16 14:39:48 optimus kernel: RAX: 0000000000000002 RBX: ffff9d3b937db700 RCX: 0000000000000000
Aug 16 14:39:48 optimus kernel: RDX: ffffbc38ff087818 RSI: ffff9d3b937db700 RDI: 0000000000000000
Aug 16 14:39:48 optimus kernel: RBP: ffffbc38ff0877c8 R08: 0000000000000000 R09: 0000000000000000
Aug 16 14:39:48 optimus kernel: R10: 0000000000000000 R11: 0000000000000000 R12: ffff9d3c2af5a400
Aug 16 14:39:48 optimus kernel: R13: ffffbc38ff087818 R14: 0000000000000001 R15: ffff9d3bf7bd88a0
Aug 16 14:39:48 optimus kernel: FS:  000071e3d2a8f840(0000) GS:ffff9d5a4ed00000(0000) knlGS:0000000000000000
Aug 16 14:39:48 optimus kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Aug 16 14:39:48 optimus kernel: CR2: 00002681e15d70a8 CR3: 0000800172cda000 CR4: 0000000000350ef0
Aug 16 14:39:48 optimus kernel: Call Trace:
Aug 16 14:39:48 optimus kernel:  <TASK>
Aug 16 14:39:48 optimus kernel:  ? show_regs+0x6d/0x80
Aug 16 14:39:48 optimus kernel:  ? __warn+0x89/0x160
Aug 16 14:39:48 optimus kernel:  ? br_nf_local_in+0x1b9/0x1e0
Aug 16 14:39:48 optimus kernel:  ? report_bug+0x17e/0x1b0
Aug 16 14:39:48 optimus kernel:  ? handle_bug+0x6e/0xb0
Aug 16 14:39:48 optimus kernel:  ? exc_invalid_op+0x18/0x80
Aug 16 14:39:48 optimus kernel:  ? asm_exc_invalid_op+0x1b/0x20
Aug 16 14:39:48 optimus kernel:  ? br_nf_local_in+0x1b9/0x1e0
Aug 16 14:39:48 optimus kernel:  nf_hook_slow+0x46/0x120
Aug 16 14:39:48 optimus kernel:  br_pass_frame_up+0x146/0x1d0
Aug 16 14:39:48 optimus kernel:  ? __pfx_br_netif_receive_skb+0x10/0x10
Aug 16 14:39:48 optimus kernel:  br_handle_frame_finish+0x38c/0x670
Aug 16 14:39:48 optimus kernel:  ? srso_return_thunk+0x5/0x5f
Aug 16 14:39:48 optimus kernel:  ? init_conntrack.constprop.0+0x25b/0x4e0 [nf_conntrack]
Aug 16 14:39:48 optimus kernel:  ? __pfx_br_handle_frame_finish+0x10/0x10
Aug 16 14:39:48 optimus kernel:  br_nf_hook_thresh+0x10a/0x120
Aug 16 14:39:48 optimus kernel:  ? __pfx_br_handle_frame_finish+0x10/0x10
Aug 16 14:39:48 optimus kernel:  br_nf_pre_routing_finish+0x191/0x490
Aug 16 14:39:48 optimus kernel:  ? __pfx_br_handle_frame_finish+0x10/0x10
Aug 16 14:39:48 optimus kernel:  ? srso_return_thunk+0x5/0x5f
Aug 16 14:39:48 optimus kernel:  ? ipv4_conntrack_in+0x14/0x20 [nf_conntrack]
Aug 16 14:39:48 optimus kernel:  ? srso_return_thunk+0x5/0x5f
Aug 16 14:39:48 optimus kernel:  br_nf_pre_routing+0x24b/0x5f0
Aug 16 14:39:48 optimus kernel:  ? __pfx_br_nf_pre_routing_finish+0x10/0x10
Aug 16 14:39:48 optimus kernel:  br_handle_frame+0x25d/0x430
Aug 16 14:39:48 optimus kernel:  ? __pfx_br_handle_frame_finish+0x10/0x10
Aug 16 14:39:48 optimus kernel:  __netif_receive_skb_core.constprop.0+0x257/0xf90
Aug 16 14:39:48 optimus kernel:  ? srso_return_thunk+0x5/0x5f
Aug 16 14:39:48 optimus kernel:  __netif_receive_skb_one_core+0x3e/0xa0
Aug 16 14:39:48 optimus kernel:  __netif_receive_skb+0x15/0x60
Aug 16 14:39:48 optimus kernel:  netif_receive_skb+0x12c/0x180
Aug 16 14:39:48 optimus kernel:  tun_sendmsg+0x481/0x820
Aug 16 14:39:48 optimus kernel:  vhost_tx_batch.constprop.0+0x77/0x260 [vhost_net]
Aug 16 14:39:48 optimus kernel:  handle_tx_copy+0x1cd/0x6f0 [vhost_net]
Aug 16 14:39:48 optimus kernel:  handle_tx+0xbc/0xc0 [vhost_net]
Aug 16 14:39:48 optimus kernel:  handle_tx_kick+0x15/0x20 [vhost_net]
Aug 16 14:39:48 optimus kernel:  vhost_run_work_list+0x49/0x80 [vhost]
Aug 16 14:39:48 optimus kernel:  vhost_task_fn+0x5b/0xf0
Aug 16 14:39:48 optimus kernel:  ? __mmdrop+0x125/0x1b0
Aug 16 14:39:48 optimus kernel:  ? srso_return_thunk+0x5/0x5f
Aug 16 14:39:48 optimus kernel:  ? finish_task_switch.isra.0+0x22b/0x310
Aug 16 14:39:48 optimus kernel:  ? srso_return_thunk+0x5/0x5f
Aug 16 14:39:48 optimus kernel:  ? __pfx_vhost_task_fn+0x10/0x10
Aug 16 14:39:48 optimus kernel:  ret_from_fork+0x47/0x70
Aug 16 14:39:48 optimus kernel:  ? __pfx_vhost_task_fn+0x10/0x10
Aug 16 14:39:48 optimus kernel:  ret_from_fork_asm+0x1b/0x30
Aug 16 14:39:48 optimus kernel: RIP: 0033:0x0
Aug 16 14:39:48 optimus kernel: Code: Unable to access opcode bytes at 0xffffffffffffffd6.
Aug 16 14:39:48 optimus kernel: RSP: 002b:0000000000000000 EFLAGS: 00000246 ORIG_RAX: 0000000000000010
Aug 16 14:39:48 optimus kernel: RAX: 0000000000000000 RBX: 000058a1f81816e0 RCX: 000071e3d5d558db
Aug 16 14:39:48 optimus kernel: RDX: 0000000000000000 RSI: 000000000000af01 RDI: 0000000000000015
Aug 16 14:39:48 optimus kernel: RBP: 00007ffdb3793310 R08: 00007ffdb37932b0 R09: 0000000000000001
Aug 16 14:39:48 optimus kernel: R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000014
Aug 16 14:39:48 optimus kernel: R13: 00007ffdb3793308 R14: 000058a1f81816e0 R15: 0000000000000000
Aug 16 14:39:48 optimus kernel:  </TASK>
Aug 16 14:39:48 optimus kernel: ---[ end trace 0000000000000000 ]---

proxmox-ve: 9.0.0 (running kernel: 6.14.8-2-pve)
pve-manager: 9.0.5 (running version: 9.0.5/9c5600b249dbfd2f)
proxmox-kernel-helper: 9.0.3
pve-kernel-6.2: 8.0.5
proxmox-kernel-6.14.8-2-pve-signed: 6.14.8-2
proxmox-kernel-6.14: 6.14.8-2
proxmox-kernel-6.8.12-13-pve-signed: 6.8.12-13
proxmox-kernel-6.8: 6.8.12-13
proxmox-kernel-6.8.12-11-pve-signed: 6.8.12-11
proxmox-kernel-6.8.12-10-pve-signed: 6.8.12-10
proxmox-kernel-6.8.12-9-pve-signed: 6.8.12-9
proxmox-kernel-6.5.13-6-pve-signed: 6.5.13-6
proxmox-kernel-6.5: 6.5.13-6
proxmox-kernel-6.5.13-5-pve-signed: 6.5.13-5
proxmox-kernel-6.5.13-3-pve-signed: 6.5.13-3
proxmox-kernel-6.5.13-1-pve-signed: 6.5.13-1
proxmox-kernel-6.5.11-8-pve-signed: 6.5.11-8
proxmox-kernel-6.5.11-7-pve-signed: 6.5.11-7
proxmox-kernel-6.5.11-6-pve-signed: 6.5.11-6
proxmox-kernel-6.5.11-5-pve-signed: 6.5.11-5
proxmox-kernel-6.2.16-20-pve: 6.2.16-20
proxmox-kernel-6.2: 6.2.16-20
pve-kernel-6.2.16-5-pve: 6.2.16-6
pve-kernel-6.2.16-4-pve: 6.2.16-5
amd64-microcode: 3.20250311.1
ceph-fuse: 19.2.3-pve1
corosync: 3.1.9-pve2
criu: 4.1.1-1
frr-pythontools: 10.3.1-1+pve4
ifupdown: not correctly installed
ifupdown2: 3.3.0-1+pmx9
ksm-control-daemon: 1.5-1
libjs-extjs: 7.0.0-5
libproxmox-acme-perl: 1.7.0
libproxmox-backup-qemu0: 2.0.1
libproxmox-rs-perl: 0.4.1
libpve-access-control: 9.0.3
libpve-apiclient-perl: 3.4.0
libpve-cluster-api-perl: 9.0.6
libpve-cluster-perl: 9.0.6
libpve-common-perl: 9.0.9
libpve-guest-common-perl: 6.0.2
libpve-http-server-perl: 6.0.4
libpve-network-perl: 1.1.6
libpve-rs-perl: 0.10.10
libpve-storage-perl: 9.0.13
libqb0: 1.0.5-1
libspice-server1: 0.15.2-1+b1
lvm2: 2.03.31-2+pmx1
lxc-pve: 6.0.4-2
lxcfs: 6.0.4-pve1
novnc-pve: 1.6.0-3
proxmox-backup-client: 4.0.14-1
proxmox-backup-file-restore: 4.0.14-1
proxmox-backup-restore-image: 1.0.0
proxmox-firewall: 1.1.1
proxmox-kernel-helper: 9.0.3
proxmox-mail-forward: 1.0.2
proxmox-mini-journalreader: 1.6
proxmox-offline-mirror-helper: 0.7.0
proxmox-widget-toolkit: 5.0.5
pve-cluster: 9.0.6
pve-container: 6.0.9
pve-docs: 9.0.8
pve-edk2-firmware: 4.2025.02-4
pve-esxi-import-tools: 1.0.1
pve-firewall: 6.0.3
pve-firmware: 3.16-3
pve-ha-manager: 5.0.4
pve-i18n: 3.5.2
pve-qemu-kvm: 10.0.2-4
pve-xtermjs: 5.5.0-2
qemu-server: 9.0.18
smartmontools: 7.4-pve1
spiceterm: 3.4.0
swtpm: 0.8.0+pve2
vncterm: 1.9.0
zfsutils-linux: 2.3.3-pve1

 

[pep.]

I have noticed this as well on an Asus NUC 14 after updating the BIOS to version 50 (from 49), running on Kernel 6.14.8-2-pve. Network on the node works, though.

 

[alpeh1974]

Maybe this and this is related to our problem? But this is just a guess as I am certainly no kernel code expert. Maybe someone from Proxmox team could sort out this issue?