Good morning, for the past couple of months, I get these firewall crashes and it completely crashes my proxmox instance. It never happened initially and had no crashes for ages. However now its rare to get more than 48 hours of up-time. Any advice on what can be causing this?
I am running 9.1.4, let me know if more details would be helpful in narrowing down the issue.
Jan 08 04:14:38 pve kernel: BUG: Bad page state in process proxmox-firewal pfn:d3237e
Jan 08 04:14:38 pve kernel: page: refcount:-1 mapcount:0 mapping:0000000000000000 index:0x0 pfn:0xd3237e
Jan 08 04:14:38 pve kernel: flags: 0x17ffffc0000000(node=0|zone=2|lastcpupid=0x1fffff)
Jan 08 04:14:38 pve kernel: raw: 0017ffffc0000000 dead000000000100 dead000000000122 0000000000000000
Jan 08 04:14:38 pve kernel: raw: 0000000000000000 0000000000000000 ffffffffffffffff 0000000000000000
Jan 08 04:14:38 pve kernel: page dumped because: nonzero _refcount
Jan 08 04:14:38 pve kernel: Modules linked in: veth cmac nls_utf8 cifs cifs_arc4 nls_ucs2_utils rdma_cm iw_cm ib_cm ib_core cifs_md4 netfs tcp_diag inet_diag ebtable_filter ebtables ip6table_raw ip6t_REJECT nf_reject_ipv6 ip6table_filter ip6_tables iptable_raw ipt_REJECT nf_reject_ipv4 xt_mark xt_set xt_physdev xt_addrtype xt_comment xt_multiport xt_conntrack nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 xt_tcpudp iptable_filter ip_set_hash_net ip_set nf_tables softdog sunrpc binfmt_misc bonding tls nfnetlink_log amd_atl intel_rapl_msr intel_rapl_common snd_hda_codec_alc882 snd_hda_codec_realtek_lib snd_hda_codec_generic snd_hda_codec_nvhdmi iwlmvm snd_hda_codec_hdmi sch_fq_codel edac_mce_amd snd_hda_intel kvm_amd snd_hda_codec mac80211 ccp snd_hda_core libarc4 snd_intel_dspcfg snd_intel_sdw_acpi btusb snd_hwdep btrtl kvm snd_pcm btintel snd_timer iwlwifi polyval_clmulni btbcm ghash_clmulni_intel snd btmtk aesni_intel mxm_wmi wmi_bmof rapl ee1004 pcspkr bluetooth k10temp cfg80211 soundcore joydev mac_hid zfs(PO) spl(O) vhost_net vhost
Jan 08 04:14:38 pve kernel: vhost_iotlb tap vfio_pci vfio_pci_core irqbypass vfio_iommu_type1 vfio iommufd efi_pstore nfnetlink dmi_sysfs ip_tables x_tables autofs4 xfs btrfs blake2b_generic xor raid6_pq hid_generic usbhid hid dm_thin_pool dm_persistent_data dm_bio_prison dm_bufio ucsi_ccg typec_ucsi nvme typec xhci_pci thunderbolt ixgbe i2c_piix4 igb nvme_core atlantic xfrm_algo i2c_smbus ahci mdio i2c_algo_bit nvme_keyring xhci_hcd i2c_nvidia_gpu libahci macsec dca nvme_auth i2c_ccgx_ucsi wmi
Jan 08 04:14:38 pve kernel: CPU: 12 UID: 0 PID: 406972 Comm: proxmox-firewal Tainted: P O 6.17.4-2-pve #1 PREEMPT(voluntary)
Jan 08 04:14:38 pve kernel: Tainted: [P]=PROPRIETARY_MODULE, [O]=OOT_MODULE
Jan 08 04:14:38 pve kernel: Hardware name: Micro-Star International Co., Ltd. MS-7C36/PRESTIGE X570 CREATION (MS-7C36), BIOS 1.M0 10/14/2023
Jan 08 04:14:38 pve kernel: Call Trace:
Jan 08 04:14:38 pve kernel: <TASK>
Jan 08 04:14:38 pve kernel: dump_stack_lvl+0x5f/0x90
Jan 08 04:14:38 pve kernel: dump_stack+0x10/0x18
Jan 08 04:14:38 pve kernel: bad_page.cold+0x77/0x8e
Jan 08 04:14:38 pve kernel: __rmqueue_pcplist+0x187/0x1170
Jan 08 04:14:38 pve kernel: get_page_from_freelist+0x360/0x1d70
Jan 08 04:14:38 pve kernel: ? srso_return_thunk+0x5/0x5f
Jan 08 04:14:38 pve kernel: ? srso_return_thunk+0x5/0x5f
Jan 08 04:14:38 pve kernel: ? terminate_walk+0x66/0x100
Jan 08 04:14:38 pve kernel: ? srso_return_thunk+0x5/0x5f
Jan 08 04:14:38 pve kernel: __alloc_frozen_pages_noprof+0x1bd/0x12a0
Jan 08 04:14:38 pve kernel: ? __fput_deferred+0x10e/0x140
Jan 08 04:14:38 pve kernel: ? srso_return_thunk+0x5/0x5f
Jan 08 04:14:38 pve kernel: ? policy_nodemask+0x111/0x190
Jan 08 04:14:38 pve kernel: alloc_pages_mpol+0x80/0x180
Jan 08 04:14:38 pve kernel: alloc_pages_noprof+0x50/0xa0
Jan 08 04:14:38 pve kernel: pgd_alloc+0x3c/0x270
Jan 08 04:14:38 pve kernel: mm_init+0x1d5/0x550
Jan 08 04:14:38 pve kernel: mm_alloc+0x4d/0x70
Jan 08 04:14:38 pve kernel: alloc_bprm+0xf7/0x250
Jan 08 04:14:38 pve kernel: do_execveat_common.isra.0+0x97/0x1d0
Jan 08 04:14:38 pve kernel: __x64_sys_execve+0x39/0x60
Jan 08 04:14:38 pve kernel: x64_sys_call+0x1f58/0x2330
Jan 08 04:14:38 pve kernel: do_syscall_64+0x80/0xa30
Jan 08 04:14:38 pve kernel: ? do_execveat_common.isra.0+0xf6/0x1d0
Jan 08 04:14:38 pve kernel: ? srso_return_thunk+0x5/0x5f
Jan 08 04:14:38 pve kernel: ? __x64_sys_execve+0x39/0x60
Jan 08 04:14:38 pve kernel: ? srso_return_thunk+0x5/0x5f
Jan 08 04:14:38 pve kernel: ? x64_sys_call+0x1f58/0x2330
Jan 08 04:14:38 pve kernel: ? srso_return_thunk+0x5/0x5f
Jan 08 04:14:38 pve kernel: ? do_syscall_64+0xb8/0xa30
Jan 08 04:14:38 pve kernel: ? srso_return_thunk+0x5/0x5f
Jan 08 04:14:38 pve kernel: ? do_open_execat+0x63/0x120
Jan 08 04:14:38 pve kernel: ? srso_return_thunk+0x5/0x5f
Jan 08 04:14:38 pve kernel: ? putname+0x65/0x90
Jan 08 04:14:38 pve kernel: ? srso_return_thunk+0x5/0x5f
Jan 08 04:14:38 pve kernel: ? do_execveat_common.isra.0+0xf6/0x1d0
Jan 08 04:14:38 pve kernel: ? srso_return_thunk+0x5/0x5f
Jan 08 04:14:38 pve kernel: ? __x64_sys_execve+0x39/0x60
Jan 08 04:14:38 pve kernel: ? srso_return_thunk+0x5/0x5f
Jan 08 04:14:38 pve kernel: ? x64_sys_call+0x1f58/0x2330
Jan 08 04:14:38 pve kernel: ? srso_return_thunk+0x5/0x5f
Jan 08 04:14:38 pve kernel: ? do_syscall_64+0xb8/0xa30
Jan 08 04:14:38 pve kernel: ? srso_return_thunk+0x5/0x5f
Jan 08 04:14:38 pve kernel: ? filp_flush+0x5e/0xb0
Jan 08 04:14:38 pve kernel: ? srso_return_thunk+0x5/0x5f
Jan 08 04:14:38 pve kernel: ? __set_task_blocked+0x29/0x80
Jan 08 04:14:38 pve kernel: ? _copy_from_user+0x27/0x70
Jan 08 04:14:38 pve kernel: ? srso_return_thunk+0x5/0x5f
Jan 08 04:14:38 pve kernel: ? __x64_sys_rt_sigprocmask+0xd8/0x160
Jan 08 04:14:38 pve kernel: ? srso_return_thunk+0x5/0x5f
Jan 08 04:14:38 pve kernel: ? x64_sys_call+0x178d/0x2330
Jan 08 04:14:38 pve kernel: ? srso_return_thunk+0x5/0x5f
Jan 08 04:14:38 pve kernel: ? do_syscall_64+0xb8/0xa30
Jan 08 04:14:38 pve kernel: ? ksys_dup3+0x9d/0x120
Jan 08 04:14:38 pve kernel: ? srso_return_thunk+0x5/0x5f
Jan 08 04:14:38 pve kernel: ? __x64_sys_dup2+0x2e/0xd0
Jan 08 04:14:38 pve kernel: ? srso_return_thunk+0x5/0x5f
Jan 08 04:14:38 pve kernel: ? x64_sys_call+0x1361/0x2330
Jan 08 04:14:38 pve kernel: ? srso_return_thunk+0x5/0x5f
Jan 08 04:14:38 pve kernel: ? do_syscall_64+0xb8/0xa30
Jan 08 04:14:38 pve kernel: ? srso_return_thunk+0x5/0x5f
Jan 08 04:14:38 pve kernel: ? irqentry_exit_to_user_mode+0x2e/0x290
Jan 08 04:14:38 pve kernel: ? srso_return_thunk+0x5/0x5f
Jan 08 04:14:38 pve kernel: ? irqentry_exit+0x43/0x50
Jan 08 04:14:38 pve kernel: ? srso_return_thunk+0x5/0x5f
Jan 08 04:14:38 pve kernel: ? exc_page_fault+0x90/0x1b0
Jan 08 04:14:38 pve kernel: entry_SYSCALL_64_after_hwframe+0x76/0x7e
Jan 08 04:14:38 pve kernel: RIP: 0033:0x7bf01a0e7dd7
Jan 08 04:14:38 pve kernel: Code: af e8 ed ed 03 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 00 48 8b 05 d9 91 10 00 48 8b 10 e9 01 00 00 00 90 b8 3b 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d f1 8f 10 00 f7 d8 64 89 01 48
Jan 08 04:14:38 pve kernel: RSP: 002b:00007bf01a886d98 EFLAGS: 00000202 ORIG_RAX: 000000000000003b
Jan 08 04:14:38 pve kernel: RAX: ffffffffffffffda RBX: 0000604cb0346a80 RCX: 00007bf01a0e7dd7
Jan 08 04:14:38 pve kernel: RDX: 00007ffeff57a530 RSI: 0000604cb03421e0 RDI: 00007bf01a886da0
Jan 08 04:14:38 pve kernel: RBP: 00007bf01a886e50 R08: 00007ffeff57bed4 R09: 0000000000000000
Jan 08 04:14:38 pve kernel: R10: 0000000000000008 R11: 0000000000000202 R12: 0000604cb03421e0
Jan 08 04:14:38 pve kernel: R13: 00007ffeff57a530 R14: 0000000000000000 R15: 00007ffeff57becb
Jan 08 04:14:38 pve kernel: </TASK>
Jan 08 04:17:01 pve CRON[407064]: pam_unix(cron:session): session opened for user root(uid=0) by root(uid=0)
Jan 08 04:17:01 pve CRON[407066]: (root) CMD (cd / && run-parts --report /etc/cron.hourly)
Jan 08 04:17:01 pve CRON[407064]: pam_unix(cron:session): session closed for user root
I am running 9.1.4, let me know if more details would be helpful in narrowing down the issue.
Jan 08 04:14:38 pve kernel: BUG: Bad page state in process proxmox-firewal pfn:d3237e
Jan 08 04:14:38 pve kernel: page: refcount:-1 mapcount:0 mapping:0000000000000000 index:0x0 pfn:0xd3237e
Jan 08 04:14:38 pve kernel: flags: 0x17ffffc0000000(node=0|zone=2|lastcpupid=0x1fffff)
Jan 08 04:14:38 pve kernel: raw: 0017ffffc0000000 dead000000000100 dead000000000122 0000000000000000
Jan 08 04:14:38 pve kernel: raw: 0000000000000000 0000000000000000 ffffffffffffffff 0000000000000000
Jan 08 04:14:38 pve kernel: page dumped because: nonzero _refcount
Jan 08 04:14:38 pve kernel: Modules linked in: veth cmac nls_utf8 cifs cifs_arc4 nls_ucs2_utils rdma_cm iw_cm ib_cm ib_core cifs_md4 netfs tcp_diag inet_diag ebtable_filter ebtables ip6table_raw ip6t_REJECT nf_reject_ipv6 ip6table_filter ip6_tables iptable_raw ipt_REJECT nf_reject_ipv4 xt_mark xt_set xt_physdev xt_addrtype xt_comment xt_multiport xt_conntrack nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 xt_tcpudp iptable_filter ip_set_hash_net ip_set nf_tables softdog sunrpc binfmt_misc bonding tls nfnetlink_log amd_atl intel_rapl_msr intel_rapl_common snd_hda_codec_alc882 snd_hda_codec_realtek_lib snd_hda_codec_generic snd_hda_codec_nvhdmi iwlmvm snd_hda_codec_hdmi sch_fq_codel edac_mce_amd snd_hda_intel kvm_amd snd_hda_codec mac80211 ccp snd_hda_core libarc4 snd_intel_dspcfg snd_intel_sdw_acpi btusb snd_hwdep btrtl kvm snd_pcm btintel snd_timer iwlwifi polyval_clmulni btbcm ghash_clmulni_intel snd btmtk aesni_intel mxm_wmi wmi_bmof rapl ee1004 pcspkr bluetooth k10temp cfg80211 soundcore joydev mac_hid zfs(PO) spl(O) vhost_net vhost
Jan 08 04:14:38 pve kernel: vhost_iotlb tap vfio_pci vfio_pci_core irqbypass vfio_iommu_type1 vfio iommufd efi_pstore nfnetlink dmi_sysfs ip_tables x_tables autofs4 xfs btrfs blake2b_generic xor raid6_pq hid_generic usbhid hid dm_thin_pool dm_persistent_data dm_bio_prison dm_bufio ucsi_ccg typec_ucsi nvme typec xhci_pci thunderbolt ixgbe i2c_piix4 igb nvme_core atlantic xfrm_algo i2c_smbus ahci mdio i2c_algo_bit nvme_keyring xhci_hcd i2c_nvidia_gpu libahci macsec dca nvme_auth i2c_ccgx_ucsi wmi
Jan 08 04:14:38 pve kernel: CPU: 12 UID: 0 PID: 406972 Comm: proxmox-firewal Tainted: P O 6.17.4-2-pve #1 PREEMPT(voluntary)
Jan 08 04:14:38 pve kernel: Tainted: [P]=PROPRIETARY_MODULE, [O]=OOT_MODULE
Jan 08 04:14:38 pve kernel: Hardware name: Micro-Star International Co., Ltd. MS-7C36/PRESTIGE X570 CREATION (MS-7C36), BIOS 1.M0 10/14/2023
Jan 08 04:14:38 pve kernel: Call Trace:
Jan 08 04:14:38 pve kernel: <TASK>
Jan 08 04:14:38 pve kernel: dump_stack_lvl+0x5f/0x90
Jan 08 04:14:38 pve kernel: dump_stack+0x10/0x18
Jan 08 04:14:38 pve kernel: bad_page.cold+0x77/0x8e
Jan 08 04:14:38 pve kernel: __rmqueue_pcplist+0x187/0x1170
Jan 08 04:14:38 pve kernel: get_page_from_freelist+0x360/0x1d70
Jan 08 04:14:38 pve kernel: ? srso_return_thunk+0x5/0x5f
Jan 08 04:14:38 pve kernel: ? srso_return_thunk+0x5/0x5f
Jan 08 04:14:38 pve kernel: ? terminate_walk+0x66/0x100
Jan 08 04:14:38 pve kernel: ? srso_return_thunk+0x5/0x5f
Jan 08 04:14:38 pve kernel: __alloc_frozen_pages_noprof+0x1bd/0x12a0
Jan 08 04:14:38 pve kernel: ? __fput_deferred+0x10e/0x140
Jan 08 04:14:38 pve kernel: ? srso_return_thunk+0x5/0x5f
Jan 08 04:14:38 pve kernel: ? policy_nodemask+0x111/0x190
Jan 08 04:14:38 pve kernel: alloc_pages_mpol+0x80/0x180
Jan 08 04:14:38 pve kernel: alloc_pages_noprof+0x50/0xa0
Jan 08 04:14:38 pve kernel: pgd_alloc+0x3c/0x270
Jan 08 04:14:38 pve kernel: mm_init+0x1d5/0x550
Jan 08 04:14:38 pve kernel: mm_alloc+0x4d/0x70
Jan 08 04:14:38 pve kernel: alloc_bprm+0xf7/0x250
Jan 08 04:14:38 pve kernel: do_execveat_common.isra.0+0x97/0x1d0
Jan 08 04:14:38 pve kernel: __x64_sys_execve+0x39/0x60
Jan 08 04:14:38 pve kernel: x64_sys_call+0x1f58/0x2330
Jan 08 04:14:38 pve kernel: do_syscall_64+0x80/0xa30
Jan 08 04:14:38 pve kernel: ? do_execveat_common.isra.0+0xf6/0x1d0
Jan 08 04:14:38 pve kernel: ? srso_return_thunk+0x5/0x5f
Jan 08 04:14:38 pve kernel: ? __x64_sys_execve+0x39/0x60
Jan 08 04:14:38 pve kernel: ? srso_return_thunk+0x5/0x5f
Jan 08 04:14:38 pve kernel: ? x64_sys_call+0x1f58/0x2330
Jan 08 04:14:38 pve kernel: ? srso_return_thunk+0x5/0x5f
Jan 08 04:14:38 pve kernel: ? do_syscall_64+0xb8/0xa30
Jan 08 04:14:38 pve kernel: ? srso_return_thunk+0x5/0x5f
Jan 08 04:14:38 pve kernel: ? do_open_execat+0x63/0x120
Jan 08 04:14:38 pve kernel: ? srso_return_thunk+0x5/0x5f
Jan 08 04:14:38 pve kernel: ? putname+0x65/0x90
Jan 08 04:14:38 pve kernel: ? srso_return_thunk+0x5/0x5f
Jan 08 04:14:38 pve kernel: ? do_execveat_common.isra.0+0xf6/0x1d0
Jan 08 04:14:38 pve kernel: ? srso_return_thunk+0x5/0x5f
Jan 08 04:14:38 pve kernel: ? __x64_sys_execve+0x39/0x60
Jan 08 04:14:38 pve kernel: ? srso_return_thunk+0x5/0x5f
Jan 08 04:14:38 pve kernel: ? x64_sys_call+0x1f58/0x2330
Jan 08 04:14:38 pve kernel: ? srso_return_thunk+0x5/0x5f
Jan 08 04:14:38 pve kernel: ? do_syscall_64+0xb8/0xa30
Jan 08 04:14:38 pve kernel: ? srso_return_thunk+0x5/0x5f
Jan 08 04:14:38 pve kernel: ? filp_flush+0x5e/0xb0
Jan 08 04:14:38 pve kernel: ? srso_return_thunk+0x5/0x5f
Jan 08 04:14:38 pve kernel: ? __set_task_blocked+0x29/0x80
Jan 08 04:14:38 pve kernel: ? _copy_from_user+0x27/0x70
Jan 08 04:14:38 pve kernel: ? srso_return_thunk+0x5/0x5f
Jan 08 04:14:38 pve kernel: ? __x64_sys_rt_sigprocmask+0xd8/0x160
Jan 08 04:14:38 pve kernel: ? srso_return_thunk+0x5/0x5f
Jan 08 04:14:38 pve kernel: ? x64_sys_call+0x178d/0x2330
Jan 08 04:14:38 pve kernel: ? srso_return_thunk+0x5/0x5f
Jan 08 04:14:38 pve kernel: ? do_syscall_64+0xb8/0xa30
Jan 08 04:14:38 pve kernel: ? ksys_dup3+0x9d/0x120
Jan 08 04:14:38 pve kernel: ? srso_return_thunk+0x5/0x5f
Jan 08 04:14:38 pve kernel: ? __x64_sys_dup2+0x2e/0xd0
Jan 08 04:14:38 pve kernel: ? srso_return_thunk+0x5/0x5f
Jan 08 04:14:38 pve kernel: ? x64_sys_call+0x1361/0x2330
Jan 08 04:14:38 pve kernel: ? srso_return_thunk+0x5/0x5f
Jan 08 04:14:38 pve kernel: ? do_syscall_64+0xb8/0xa30
Jan 08 04:14:38 pve kernel: ? srso_return_thunk+0x5/0x5f
Jan 08 04:14:38 pve kernel: ? irqentry_exit_to_user_mode+0x2e/0x290
Jan 08 04:14:38 pve kernel: ? srso_return_thunk+0x5/0x5f
Jan 08 04:14:38 pve kernel: ? irqentry_exit+0x43/0x50
Jan 08 04:14:38 pve kernel: ? srso_return_thunk+0x5/0x5f
Jan 08 04:14:38 pve kernel: ? exc_page_fault+0x90/0x1b0
Jan 08 04:14:38 pve kernel: entry_SYSCALL_64_after_hwframe+0x76/0x7e
Jan 08 04:14:38 pve kernel: RIP: 0033:0x7bf01a0e7dd7
Jan 08 04:14:38 pve kernel: Code: af e8 ed ed 03 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 00 48 8b 05 d9 91 10 00 48 8b 10 e9 01 00 00 00 90 b8 3b 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d f1 8f 10 00 f7 d8 64 89 01 48
Jan 08 04:14:38 pve kernel: RSP: 002b:00007bf01a886d98 EFLAGS: 00000202 ORIG_RAX: 000000000000003b
Jan 08 04:14:38 pve kernel: RAX: ffffffffffffffda RBX: 0000604cb0346a80 RCX: 00007bf01a0e7dd7
Jan 08 04:14:38 pve kernel: RDX: 00007ffeff57a530 RSI: 0000604cb03421e0 RDI: 00007bf01a886da0
Jan 08 04:14:38 pve kernel: RBP: 00007bf01a886e50 R08: 00007ffeff57bed4 R09: 0000000000000000
Jan 08 04:14:38 pve kernel: R10: 0000000000000008 R11: 0000000000000202 R12: 0000604cb03421e0
Jan 08 04:14:38 pve kernel: R13: 00007ffeff57a530 R14: 0000000000000000 R15: 00007ffeff57becb
Jan 08 04:14:38 pve kernel: </TASK>
Jan 08 04:17:01 pve CRON[407064]: pam_unix(cron:session): session opened for user root(uid=0) by root(uid=0)
Jan 08 04:17:01 pve CRON[407066]: (root) CMD (cd / && run-parts --report /etc/cron.hourly)
Jan 08 04:17:01 pve CRON[407064]: pam_unix(cron:session): session closed for user root