[SOLVED] BUG: kernel NULL pointer dereference, address: 0000000000000000

ukro · Jan 9, 2022

Cheers, updated from pve6 to pve7 and have this random error 1x in 4-36hours.
This thread seems similar but its not fixed on my HW.
https://forum.proxmox.com/threads/p...error_code-0x0000-no-web-access-no-ssh.96598/
I have web access and no ssh access, my VMs are runnning but the host only network between vms and host are down.
The web access is with errors like grayed out icons and cant shutdown VMs/CTs
The error occured maybe in high IO load of one of the VMs.

FYI: I updated because pve6 have some random errors(didn't want to report) that i thought pve7 would fix

CPU(s)

16 x Intel(R) Core(TM) i7-10700 CPU @ 2.90GHz (1 Socket)

Kernel Version

Linux 5.13.19-2-pve #1 SMP PVE 5.13.19-4 (Mon, 29 Nov 2021 12:10:09 +0100)

PVE Manager Version

pve-manager/7.1-8/5b267f33

Code:

Jan  9 04:11:21 matrix kernel: [10250.198771] BUG: kernel NULL pointer dereference, address: 0000000000000000
Jan  9 04:11:21 matrix kernel: [10250.198796] #PF: supervisor read access in kernel mode
Jan  9 04:11:21 matrix kernel: [10250.198806] #PF: error_code(0x0000) - not-present page
Jan  9 04:11:21 matrix kernel: [10250.198817] PGD 0 P4D 0
Jan  9 04:11:21 matrix kernel: [10250.198825] Oops: 0000 [#1] SMP NOPTI
Jan  9 04:11:21 matrix kernel: [10250.198834] CPU: 5 PID: 757 Comm: arc_prune Tainted: P           O      5.13.19-2-pve #1
Jan  9 04:11:21 matrix kernel: [10250.198849] Hardware name: ASUS System Product Name/PRIME B460M-A, BIOS 1605 04/07/2021
Jan  9 04:11:21 matrix kernel: [10250.198863] RIP: 0010:__list_lru_walk_one+0x9c/0x110
Jan  9 04:11:21 matrix kernel: [10250.198876] Code: 8b 45 d0 e8 36 ef d6 00 83 f8 02 74 58 77 16 85 c0 74 74 8b 03 85 c0 74 7a 48 83 45 c0 01 48 83 6b 28 01 eb b0 83 f8 03 75 2a <49> 8b 07 4d 89 fe 4d 39 fd 74 0c 49 89 c7 49 8b 04 24 48 85 c0 75
Jan  9 04:11:21 matrix kernel: [10250.198906] RSP: 0018:ffffa84cc0897bf0 EFLAGS: 00010246
Jan  9 04:11:21 matrix kernel: [10250.198917] RAX: 0000000000000003 RBX: ffff8f21d9c7eb00 RCX: ffffa84cc0897c78
Jan  9 04:11:21 matrix kernel: [10250.198930] RDX: ffff8f21d9c7eb00 RSI: ffff8f21d9c7eb08 RDI: ffff8f275ef2e588
Jan  9 04:11:21 matrix kernel: [10250.198943] RBP: ffffa84cc0897c30 R08: ffff8f275c0935c0 R09: ffffa84cc0897d58
Jan  9 04:11:21 matrix kernel: [10250.198955] R10: 0000000000000000 R11: 0000000b81d8146c R12: ffffa84cc0897d58
Jan  9 04:11:21 matrix kernel: [10250.198968] R13: ffff8f21d9c7eb08 R14: ffff8f275ef2e608 R15: 0000000000000000
Jan  9 04:11:21 matrix kernel: [10250.198981] FS:  0000000000000000(0000) GS:ffff8f28cf340000(0000) knlGS:0000000000000000
Jan  9 04:11:21 matrix kernel: [10250.198995] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Jan  9 04:11:21 matrix kernel: [10250.199006] CR2: 0000000000000000 CR3: 00000001b9c10001 CR4: 00000000007726e0
Jan  9 04:11:21 matrix kernel: [10250.199020] PKRU: 55555554
Jan  9 04:11:21 matrix kernel: [10250.199026] Call Trace:
Jan  9 04:11:21 matrix kernel: [10250.199034]  ? ilookup+0x110/0x110
Jan  9 04:11:21 matrix kernel: [10250.199043]  ? ilookup+0x110/0x110
Jan  9 04:11:21 matrix kernel: [10250.199051]  list_lru_walk_one+0x4c/0x70
Jan  9 04:11:21 matrix kernel: [10250.199060]  prune_icache_sb+0x50/0xb0
Jan  9 04:11:21 matrix kernel: [10250.199068]  super_cache_scan+0x169/0x1f0
Jan  9 04:11:21 matrix kernel: [10250.199080]  zfs_prune+0x399/0x440 [zfs]
Jan  9 04:11:21 matrix kernel: [10250.199167]  ? psi_task_switch+0xd2/0x250
Jan  9 04:11:21 matrix kernel: [10250.199180]  ? __switch_to+0x11d/0x460
Jan  9 04:11:21 matrix kernel: [10250.199191]  ? __switch_to_asm+0x36/0x70
Jan  9 04:11:21 matrix kernel: [10250.199204]  zpl_prune_sb+0x35/0x50 [zfs]
Jan  9 04:11:21 matrix kernel: [10250.199267]  arc_prune_task+0x22/0x40 [zfs]
Jan  9 04:11:21 matrix kernel: [10250.199330]  taskq_thread+0x2b7/0x500 [spl]
Jan  9 04:11:21 matrix kernel: [10250.199345]  ? wake_up_q+0xa0/0xa0
Jan  9 04:11:21 matrix kernel: [10250.199704]  ? taskq_thread_spawn+0x60/0x60 [spl]
Jan  9 04:11:21 matrix kernel: [10250.200068]  kthread+0x128/0x150
Jan  9 04:11:21 matrix kernel: [10250.200430]  ? set_kthread_struct+0x50/0x50
Jan  9 04:11:21 matrix kernel: [10250.200785]  ret_from_fork+0x1f/0x30
Jan  9 04:11:21 matrix kernel: [10250.201142] Modules linked in: nfsd auth_rpcgss nfs_acl lockd grace veth tcp_diag inet_diag ebtable_filter ebtables ip_set ip6table_raw iptable_raw ip6table_filter ip6_tables iptable_filter bpfilter snd_hda_codec_hdmi snd_hda_codec_realtek snd_hda_codec_generic softdog intel_rapl_msr intel_rapl_common intel_tcc_cooling snd_sof_pci_intel_cnl snd_sof_intel_hda_common soundwire_intel soundwire_generic_allocation x86_pkg_temp_thermal nfnetlink_log soundwire_cadence intel_powerclamp ppdev snd_sof_intel_hda nfnetlink snd_sof_pci snd_sof_xtensa_dsp snd_sof snd_soc_hdac_hda snd_hda_ext_core snd_soc_acpi_intel_match snd_soc_acpi soundwire_bus ledtrig_audio snd_soc_core kvm_intel snd_compress ac97_bus snd_pcm_dmaengine snd_hda_intel snd_intel_dspcfg zfs(PO) mei_hdcp snd_intel_sdw_acpi kvm zunicode(PO) i915 snd_hda_codec crct10dif_pclmul zzstd(O) ghash_clmulni_intel snd_hda_core aesni_intel zlua(O) snd_hwdep crypto_simd zavl(PO) cryptd snd_pcm icp(PO) drm_kms_helper rapl snd_timer cec eeepc_wmi
Jan  9 04:11:21 matrix kernel: [10250.201174]  snd asus_wmi rc_core soundcore intel_cstate sparse_keymap mxm_wmi wmi_bmof i2c_algo_bit joydev pcspkr ee1004 fb_sys_fops efi_pstore input_leds syscopyarea sysfillrect mei_me sysimgblt mei zcommon(PO) znvpair(PO) spl(O) parport_pc parport vhost_net vhost mac_hid vhost_iotlb acpi_pad tap acpi_tad ib_iser rdma_cm iw_cm ib_cm ib_core iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi wireguard curve25519_x86_64 libchacha20poly1305 chacha_x86_64 poly1305_x86_64 libblake2s blake2s_x86_64 libcurve25519_generic libchacha libblake2s_generic ip6_udp_tunnel udp_tunnel nct6775 hwmon_vid coretemp vfio_pci vfio_virqfd irqbypass vfio_iommu_type1 vfio drm sunrpc ip_tables x_tables autofs4 btrfs blake2b_generic xor zstd_compress raid6_pq dm_thin_pool dm_persistent_data dm_bio_prison dm_bufio libcrc32c hid_generic usbkbd uas usbhid usb_storage hid crc32_pclmul r8169 nvme xhci_pci i2c_i801 ahci xhci_pci_renesas i2c_smbus nvme_core realtek xhci_hcd libahci wmi video pinctrl_sunrisepoint
Jan  9 04:11:21 matrix kernel: [10250.207517] CR2: 0000000000000000
Jan  9 04:11:21 matrix kernel: [10250.208009] ---[ end trace 700e123ade4624b3 ]---
Jan  9 04:11:21 matrix kernel: [10250.303829] RIP: 0010:__list_lru_walk_one+0x9c/0x110
Jan  9 04:11:21 matrix kernel: [10250.304337] Code: 8b 45 d0 e8 36 ef d6 00 83 f8 02 74 58 77 16 85 c0 74 74 8b 03 85 c0 74 7a 48 83 45 c0 01 48 83 6b 28 01 eb b0 83 f8 03 75 2a <49> 8b 07 4d 89 fe 4d 39 fd 74 0c 49 89 c7 49 8b 04 24 48 85 c0 75
Jan  9 04:11:21 matrix kernel: [10250.305399] RSP: 0018:ffffa84cc0897bf0 EFLAGS: 00010246
Jan  9 04:11:21 matrix kernel: [10250.305934] RAX: 0000000000000003 RBX: ffff8f21d9c7eb00 RCX: ffffa84cc0897c78
Jan  9 04:11:21 matrix kernel: [10250.306459] RDX: ffff8f21d9c7eb00 RSI: ffff8f21d9c7eb08 RDI: ffff8f275ef2e588
Jan  9 04:11:21 matrix kernel: [10250.306989] RBP: ffffa84cc0897c30 R08: ffff8f275c0935c0 R09: ffffa84cc0897d58
Jan  9 04:11:21 matrix kernel: [10250.307516] R10: 0000000000000000 R11: 0000000b81d8146c R12: ffffa84cc0897d58
Jan  9 04:11:21 matrix kernel: [10250.308033] R13: ffff8f21d9c7eb08 R14: ffff8f275ef2e608 R15: 0000000000000000
Jan  9 04:11:21 matrix kernel: [10250.308555] FS:  0000000000000000(0000) GS:ffff8f28cf340000(0000) knlGS:0000000000000000
Jan  9 04:11:21 matrix kernel: [10250.309084] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Jan  9 04:11:21 matrix kernel: [10250.309607] CR2: 0000000000000000 CR3: 000000010b1f8005 CR4: 00000000007726e0
Jan  9 04:11:21 matrix kernel: [10250.310147] PKRU: 55555554
Jan  9 04:11:46 matrix kernel: [10276.182809] watchdog: BUG: soft lockup - CPU#1 stuck for 26s! [smbd:2375920]

Code:

root@matrix:~# pveversion --verbose
proxmox-ve: 7.1-1 (running kernel: 5.13.19-2-pve)
pve-manager: 7.1-8 (running version: 7.1-8/5b267f33)
pve-kernel-helper: 7.1-6
pve-kernel-5.13: 7.1-5
pve-kernel-5.4: 6.4-11
pve-kernel-5.13.19-2-pve: 5.13.19-4
pve-kernel-5.4.157-1-pve: 5.4.157-1
pve-kernel-5.4.143-1-pve: 5.4.143-1
pve-kernel-5.4.124-1-pve: 5.4.124-2
pve-kernel-5.4.119-1-pve: 5.4.119-1
pve-kernel-5.4.114-1-pve: 5.4.114-1
pve-kernel-5.4.106-1-pve: 5.4.106-1
pve-kernel-5.4.98-1-pve: 5.4.98-1
pve-kernel-5.4.73-1-pve: 5.4.73-1
ceph-fuse: 14.2.21-1
corosync: 3.1.5-pve2
criu: 3.15-1+pve-1
glusterfs-client: 9.2-1
ifupdown: 0.8.36+pve1
ksm-control-daemon: 1.4-1
libjs-extjs: 7.0.0-1
libknet1: 1.22-pve2
libproxmox-acme-perl: 1.4.0
libproxmox-backup-qemu0: 1.2.0-1
libpve-access-control: 7.1-5
libpve-apiclient-perl: 3.2-1
libpve-common-perl: 7.0-14
libpve-guest-common-perl: 4.0-3
libpve-http-server-perl: 4.0-4
libpve-storage-perl: 7.0-15
libqb0: 1.0.5-1
libspice-server1: 0.14.3-2.1
lvm2: 2.03.11-2.1
lxc-pve: 4.0.11-1
lxcfs: 4.0.11-pve1
novnc-pve: 1.3.0-1
proxmox-backup-client: 2.1.2-1
proxmox-backup-file-restore: 2.1.2-1
proxmox-mini-journalreader: 1.3-1
proxmox-widget-toolkit: 3.4-4
pve-cluster: 7.1-3
pve-container: 4.1-3
pve-docs: 7.1-2
pve-edk2-firmware: 3.20210831-2
pve-firewall: 4.2-5
pve-firmware: 3.3-4
pve-ha-manager: 3.3-1
pve-i18n: 2.6-2
pve-qemu-kvm: 6.1.0-3
pve-xtermjs: 4.12.0-1
qemu-server: 7.1-4
smartmontools: 7.2-pve2
spiceterm: 3.2-2
swtpm: 0.7.0~rc1+2
vncterm: 1.7-1
zfsutils-linux: 2.1.1-pve3

ukro · Jan 10, 2022

ukro · Jan 12, 2022

reported to bugzilla

ukro · Jan 16, 2022

After change of the network adapter from virtio to E1000 the problem went away instantly. For 6 days no errors/problems.

ukro · Jan 28, 2022

Aaaand its back

Emilien · Jan 28, 2022

Maybe update your BIOS

Stoiko Ivanov · Jan 28, 2022

for reference: https://bugzilla.proxmox.com/show_bug.cgi?id=3821

I'll update the report there (and would suggest to keep the discussion in one place from now on, to not overlook any details)

tristank · Mar 27, 2022

Has this been fixed? Because I think I'm having the same issue on a Z490M Pro4 with a i9-10850K when I'm trying to use SR-IOV with a Intel 82576.

```
[ 480.881090] BUG: kernel NULL pointer dereference, address: 0000000000000008
[ 480.881108] #PF: supervisor read access in kernel mode
[ 480.881119] #PF: error_code(0x0000) - not-present page
```

ukro · Mar 27, 2022

tristank said:
Has this been fixed? Because I think I'm having the same issue on a Z490M Pro4 with a i9-10850K when I'm trying to use SR-IOV with a Intel 82576.

```
[ 480.881090] BUG: kernel NULL pointer dereference, address: 0000000000000008
[ 480.881108] #PF: supervisor read access in kernel mode
[ 480.881119] #PF: error_code(0x0000) - not-present page
```

IDK if it was fixed, but i did fix it for myself. The error was when i was using TruenasVM with connected HDD.
After i connect the whole PCI controler(that have the drive) the error disapear (Fingers crossed-its 14days already with 0 errors)

louispve · May 4, 2024

Hi, I have the same problem on Proxmox 8.2.2

May 04 16:38:59 ns3026656 kernel: ---[ end trace 0000000000000000 ]---
May 04 16:38:59 ns3026656 kernel: </TASK>
May 04 16:38:59 ns3026656 kernel: R13: 0000000000000001 R14: 0000572906ebcfa8 R15: 0000000000000000
May 04 16:38:59 ns3026656 kernel: R10: 0000000000000000 R11: 0000000000000246 R12: 000057290701ab50
May 04 16:38:59 ns3026656 kernel: RBP: 000057290701aa68 R08: 0000000000000000 R09: 0000000000000008
May 04 16:38:59 ns3026656 kernel: RDX: 0000000000000000 RSI: 0000000000000007 RDI: 000000000000002a
May 04 16:38:59 ns3026656 kernel: RAX: ffffffffffffffda RBX: 000057290701aa60 RCX: 0000719e21021b95
May 04 16:38:59 ns3026656 kernel: RSP: 002b:0000719e137fafa8 EFLAGS: 00000246 ORIG_RAX: 00000000000001aa
May 04 16:38:59 ns3026656 kernel: Code: 00 00 00 44 89 d0 41 b9 08 00 00 00 83 c8 10 f6 87 d0 00 00 00 01 8b bf cc 00 00 00 44 0f 45 d0 45 31 c0 b8 aa 01 00 00 0f 05 <c3> 66 2e 0f 1f 84 00 00 00 00 00 41 83 e>
May 04 16:38:59 ns3026656 kernel: RIP: 0033:0x719e21021b95
May 04 16:38:59 ns3026656 kernel: rewind_stack_and_make_dead+0x17/0x20
May 04 16:38:59 ns3026656 kernel: make_task_dead+0x83/0x170
May 04 16:38:59 ns3026656 kernel: ? _printk+0x60/0x90
May 04 16:38:59 ns3026656 kernel: ? do_exit+0x72/0xae0
May 04 16:38:59 ns3026656 kernel: ? do_exit+0x8dd/0xae0
May 04 16:38:59 ns3026656 kernel: ? asm_exc_invalid_op+0x1b/0x20
May 04 16:38:59 ns3026656 kernel: ? exc_invalid_op+0x18/0x80
May 04 16:38:59 ns3026656 kernel: ? handle_bug+0x46/0x90
May 04 16:38:59 ns3026656 kernel: ? report_bug+0x17e/0x1b0
May 04 16:38:59 ns3026656 kernel: ? do_exit+0x8dd/0xae0
May 04 16:38:59 ns3026656 kernel: ? __warn+0x89/0x160
May 04 16:38:59 ns3026656 kernel: ? show_regs+0x6d/0x80
May 04 16:38:59 ns3026656 kernel: <TASK>
May 04 16:38:59 ns3026656 kernel: Call Trace:
May 04 16:38:59 ns3026656 kernel: PKRU: 55555554
May 04 16:38:59 ns3026656 kernel: CR2: 0000000000000008 CR3: 000000010ad0c000 CR4: 0000000000f50ef0
May 04 16:38:59 ns3026656 kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
May 04 16:38:59 ns3026656 kernel: FS: 0000719e138006c0(0000) GS:ffff9f35fe080000(0000) knlGS:0000000000000000
May 04 16:38:59 ns3026656 kernel: R13: 0000000000000009 R14: ffff9f2704f5f380 R15: 0000000000000000
May 04 16:38:59 ns3026656 kernel: R10: 0000000000000000 R11: 0000000000000000 R12: ffff9f271204d100
May 04 16:38:59 ns3026656 kernel: RBP: ffffbce152e07f20 R08: 0000000000000000 R09: 0000000000000000
May 04 16:38:59 ns3026656 kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
May 04 16:38:59 ns3026656 kernel: RAX: 0000000000000000 RBX: ffff9f27eae78000 RCX: 0000000000000000
May 04 16:38:59 ns3026656 kernel: RSP: 0018:ffffbce152e07ec8 EFLAGS: 00010282
May 04 16:38:59 ns3026656 kernel: Code: e9 42 f8 ff ff 48 8b bb e0 09 00 00 31 f6 e8 9a e0 ff ff e9 ee fd ff ff 4c 89 ee bf 05 06 00 00 e8 08 3a 01 00 e9 6e f8 ff ff <0f> 0b e9 9c f7 ff ff 0f 0b e9 55 f7 ff f>
May 04 16:38:59 ns3026656 kernel: RIP: 0010:do_exit+0x8dd/0xae0
May 04 16:38:59 ns3026656 kernel: Hardware name: To Be Filled By O.E.M. To Be Filled By O.E.M./X470D4U2-2T, BIOS L4.03J 03/24/2023
May 04 16:38:59 ns3026656 kernel: CPU: 1 PID: 4799 Comm: kvm Tainted: P D O 6.8.4-2-pve #1
May 04 16:38:59 ns3026656 kernel: Modules linked in: tcp_diag inet_diag veth ebtable_filter ebtables ip_set ip6table_raw iptable_raw ip6table_filter ip6_tables iptable_filter softdog nf_tables sunrpc binfmt_m>
May 04 16:38:59 ns3026656 kernel: WARNING: CPU: 1 PID: 4799 at kernel/exit.c:820 do_exit+0x8dd/0xae0
May 04 16:38:59 ns3026656 kernel: ------------[ cut here ]------------
May 04 16:38:59 ns3026656 kernel: note: kvm[4799] exited with preempt_count 1
May 04 16:38:59 ns3026656 kernel: note: kvm[4799] exited with irqs disabled
May 04 16:38:59 ns3026656 kernel: PKRU: 55555554
May 04 16:38:59 ns3026656 kernel: CR2: 0000000000000008 CR3: 000000010ad0c000 CR4: 0000000000f50ef0
May 04 16:38:59 ns3026656 kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
May 04 16:38:59 ns3026656 kernel: FS: 0000719e138006c0(0000) GS:ffff9f35fe080000(0000) knlGS:0000000000000000
May 04 16:38:59 ns3026656 kernel: R13: ffff9f2716c450c0 R14: ffff9f2716c450d0 R15: ffff9f27175c8398
May 04 16:38:59 ns3026656 kernel: R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000029801
May 04 16:38:59 ns3026656 kernel: RBP: ffffbce152e07ac0 R08: 0000000000000000 R09: 0000000000000000
May 04 16:38:59 ns3026656 kernel: RDX: ffff9f2716c450e0 RSI: ffff9f2718070648 RDI: 0000000000000000
May 04 16:38:59 ns3026656 kernel: RAX: 0000000000000000 RBX: ffff9f2718070600 RCX: ffff9f2718070648
May 04 16:38:59 ns3026656 kernel: RSP: 0018:ffffbce152e07a80 EFLAGS: 00010046
May 04 16:38:59 ns3026656 kernel: Code: 0f b6 f6 49 8d 56 01 49 c1 e6 04 4d 01 ee 48 c1 e2 04 49 8b 4e 10 4c 01 ea 48 39 ca 74 2b 48 8b 4b 50 48 8b 7b 48 48 8d 73 48 <48> 89 4f 08 48 89 39 49 8b 4e 18 49 89 7>
May 04 16:38:59 ns3026656 kernel: RIP: 0010:blk_flush_complete_seq+0x291/0x2d0
May 04 16:38:59 ns3026656 kernel: ---[ end trace 0000000000000000 ]---
May 04 16:38:59 ns3026656 kernel: CR2: 0000000000000008
May 04 16:38:59 ns3026656 kernel: Modules linked in: tcp_diag inet_diag veth ebtable_filter ebtables ip_set ip6table_raw iptable_raw ip6table_filter ip6_tables iptable_filter softdog nf_tables sunrpc binfmt_m>
lines 1921-1971
May 04 16:38:59 ns3026656 kernel: </TASK>
May 04 16:38:59 ns3026656 kernel: R13: 0000000000000001 R14: 0000572906ebcfa8 R15: 0000000000000000
May 04 16:38:59 ns3026656 kernel: R10: 0000000000000000 R11: 0000000000000246 R12: 000057290701ab50
May 04 16:38:59 ns3026656 kernel: RBP: 000057290701aa68 R08: 0000000000000000 R09: 0000000000000008
May 04 16:38:59 ns3026656 kernel: RDX: 0000000000000000 RSI: 0000000000000007 RDI: 000000000000002a
May 04 16:38:59 ns3026656 kernel: RAX: ffffffffffffffda RBX: 000057290701aa60 RCX: 0000719e21021b95
May 04 16:38:59 ns3026656 kernel: RSP: 002b:0000719e137fafa8 EFLAGS: 00000246 ORIG_RAX: 00000000000001aa
May 04 16:38:59 ns3026656 kernel: Code: 00 00 00 44 89 d0 41 b9 08 00 00 00 83 c8 10 f6 87 d0 00 00 00 01 8b bf cc 00 00 00 44 0f 45 d0 45 31 c0 b8 aa 01 00 00 0f 05 <c3> 66 2e 0f 1f 84 00 00 00 00 00 41 83 e>
May 04 16:38:59 ns3026656 kernel: RIP: 0033:0x719e21021b95
May 04 16:38:59 ns3026656 kernel: entry_SYSCALL_64_after_hwframe+0x73/0x7b
May 04 16:38:59 ns3026656 kernel: ? common_interrupt+0x54/0xb0
May 04 16:38:59 ns3026656 kernel: ? srso_alias_return_thunk+0x5/0xfbef5
May 04 16:38:59 ns3026656 kernel: ? irqentry_exit+0x43/0x50
May 04 16:38:59 ns3026656 kernel: do_syscall_64+0x87/0x180
May 04 16:38:59 ns3026656 kernel: __x64_sys_io_uring_enter+0x22/0x40
May 04 16:38:59 ns3026656 kernel: ? handle_irq_event+0x52/0x80
May 04 16:38:59 ns3026656 kernel: ? srso_alias_return_thunk+0x5/0xfbef5
May 04 16:38:59 ns3026656 kernel: __do_sys_io_uring_enter+0x57c/0xbf0
May 04 16:38:59 ns3026656 kernel: io_submit_sqes+0x549/0x680
May 04 16:38:59 ns3026656 kernel: blk_finish_plug+0x31/0x50
May 04 16:38:59 ns3026656 kernel: __blk_flush_plug+0xc1/0x130
May 04 16:38:59 ns3026656 kernel: raid1_unplug+0x3c/0xf0 [raid1]
May 04 16:38:59 ns3026656 kernel: flush_bio_list+0x108/0x110 [raid1]
May 04 16:38:59 ns3026656 kernel: md_bitmap_unplug+0x99/0x200
May 04 16:38:59 ns3026656 kernel: filemap_write_page+0x5b/0x70
May 04 16:38:59 ns3026656 kernel: write_sb_page+0x148/0x300
May 04 16:38:59 ns3026656 kernel: md_super_write+0xcf/0x110
May 04 16:38:59 ns3026656 kernel: submit_bio+0xb2/0x110
May 04 16:38:59 ns3026656 kernel: submit_bio_noacct+0x1f3/0x650
May 04 16:38:59 ns3026656 kernel: submit_bio_noacct_nocheck+0x2b7/0x390
May 04 16:38:59 ns3026656 kernel: __submit_bio+0xb3/0x1c0
May 04 16:38:59 ns3026656 kernel: blk_mq_submit_bio+0x641/0x750
May 04 16:38:59 ns3026656 kernel: blk_insert_flush+0xce/0x220
May 04 16:38:59 ns3026656 kernel: ? wbt_wait+0x33/0x100
May 04 16:38:59 ns3026656 kernel: ? __blk_mq_alloc_requests+0x3e7/0x450
May 04 16:38:59 ns3026656 kernel: ? srso_alias_return_thunk+0x5/0xfbef5
May 04 16:38:59 ns3026656 kernel: ? blk_flush_complete_seq+0x291/0x2d0
May 04 16:38:59 ns3026656 kernel: ? asm_exc_page_fault+0x27/0x30
May 04 16:38:59 ns3026656 kernel: ? exc_page_fault+0x83/0x1b0
May 04 16:38:59 ns3026656 kernel: ? do_user_addr_fault+0x2f9/0x6b0
May 04 16:38:59 ns3026656 kernel: ? md_submit_bio+0x63/0xb0
May 04 16:38:59 ns3026656 kernel: ? srso_alias_return_thunk+0x5/0xfbef5
May 04 16:38:59 ns3026656 kernel: ? page_fault_oops+0x176/0x500
May 04 16:38:59 ns3026656 kernel: ? __die+0x24/0x80
May 04 16:38:59 ns3026656 kernel: ? show_regs+0x6d/0x80
May 04 16:38:59 ns3026656 kernel: <TASK>
May 04 16:38:59 ns3026656 kernel: Call Trace:
May 04 16:38:59 ns3026656 kernel: PKRU: 55555554
May 04 16:38:59 ns3026656 kernel: CR2: 0000000000000008 CR3: 000000010ad0c000 CR4: 0000000000f50ef0
May 04 16:38:59 ns3026656 kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
May 04 16:38:59 ns3026656 kernel: FS: 0000719e138006c0(0000) GS:ffff9f35fe080000(0000) knlGS:0000000000000000
May 04 16:38:59 ns3026656 kernel: R13: ffff9f2716c450c0 R14: ffff9f2716c450d0 R15: ffff9f27175c8398
May 04 16:38:59 ns3026656 kernel: R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000029801
May 04 16:38:59 ns3026656 kernel: RBP: ffffbce152e07ac0 R08: 0000000000000000 R09: 0000000000000000
May 04 16:38:59 ns3026656 kernel: RDX: ffff9f2716c450e0 RSI: ffff9f2718070648 RDI: 0000000000000000
May 04 16:38:59 ns3026656 kernel: RAX: 0000000000000000 RBX: ffff9f2718070600 RCX: ffff9f2718070648
May 04 16:38:59 ns3026656 kernel: RSP: 0018:ffffbce152e07a80 EFLAGS: 00010046
May 04 16:38:59 ns3026656 kernel: Code: 0f b6 f6 49 8d 56 01 49 c1 e6 04 4d 01 ee 48 c1 e2 04 49 8b 4e 10 4c 01 ea 48 39 ca 74 2b 48 8b 4b 50 48 8b 7b 48 48 8d 73 48 <48> 89 4f 08 48 89 39 49 8b 4e 18 49 89 7>
May 04 16:38:59 ns3026656 kernel: RIP: 0010:blk_flush_complete_seq+0x291/0x2d0
May 04 16:38:59 ns3026656 kernel: Hardware name: To Be Filled By O.E.M. To Be Filled By O.E.M./X470D4U2-2T, BIOS L4.03J 03/24/2023
May 04 16:38:59 ns3026656 kernel: CPU: 1 PID: 4799 Comm: kvm Tainted: P O 6.8.4-2-pve #1
May 04 16:38:59 ns3026656 kernel: Oops: 0002 [#1] PREEMPT SMP NOPTI
May 04 16:38:59 ns3026656 kernel: PGD 0 P4D 0
May 04 16:38:59 ns3026656 kernel: #PF: error_code(0x0002) - not-present page
May 04 16:38:59 ns3026656 kernel: #PF: supervisor write access in kernel mode
May 04 16:38:59 ns3026656 kernel: BUG: kernel NULL pointer dereference, address: 0000000000000008

positive · May 26, 2024

+1 pve8.2

Code:

BUG: kernel NULL pointer dereference, address: 0000000000000004

Taz-Matt · May 31, 2024

+1
(PVE 8.2.2)
Any news from anyone else?

Same thing here too. Running on Asus P11C-M-10G-2T. Bios is outdated, I have asked my provider to update it. Will let you know if it helps but it took probably a week before failing, sometimes longer.

Thanks!

nikolay5s · Jun 3, 2024

Have the same issue. For us helped downgrade kernel version to 6.5
Look at this thread: https://forum.proxmox.com/threads/random-6-8-4-2-pve-kernel-crashes.145760/

Taz-Matt · Jun 6, 2024

The BIOS of my server was upgraded. In the days preceeding the upgrade, it crashed almost every day or two. It now has been up two days without any crash. I will continue to monitor and post again if it crashes or maybe in a week or two if it still runs well.

Taz-Matt · Jun 13, 2024

I has been over a week without any issues. I consider it resolved for now. If ever the issue comes back. I'll post it here.

fweber · Jun 25, 2024

Hi, this particular NULL pointer dereference error with RIP pointing to blk_flush_complete_seq ...

louispve said:
May 04 16:38:59 ns3026656 kernel: RIP: 0010:blk_flush_complete_seq+0x291/0x2d0

... should be fixed in proxmox-kernel-6.8.8-1-pve and higher, see [1] for more information.

[¹] https://forum.proxmox.com/threads/random-6-8-4-2-pve-kernel-crashes.145760/page-8#post-674842

Taz-Matt · Jun 28, 2024

Also an update on my end, 23 days without any issues. Will reboot today to apply updates but the issue was resolved by updating the BIOS on my end. Thanks!

aliang · Sep 10, 2024

I always have this problem when docker continues high CPU load
kernel：Linux 6.8.8-3-pve (2024-07-16T16:16Z)
cpu: 12600T
any suggestions？

Code:

Sep 09 23:08:06 pve kernel: BUG: kernel NULL pointer dereference, address: 0000000000000620
Sep 09 23:08:06 pve kernel: #PF: supervisor read access in kernel mode
Sep 09 23:08:06 pve kernel: #PF: error_code(0x0000) - not-present page
Sep 09 23:08:06 pve kernel: PGD 0 P4D 0
Sep 09 23:08:06 pve kernel: Oops: 0000 [#1] PREEMPT SMP NOPTI
Sep 09 23:08:06 pve kernel: CPU: 5 PID: 3545995 Comm: pt_main_thread Tainted: P     U     OE      6.8.8-3-pve #1
Sep 09 23:08:06 pve kernel: Hardware name: ASUS System Product Name/ROG STRIX Z690-I GAMING WIFI, BIOS 3302 02/21/2024
Sep 09 23:08:06 pve kernel: RIP: 0010:folio_lruvec_lock_irqsave+0x4e/0xa0
Sep 09 23:08:06 pve kernel: Code: 8b 17 48 c1 ea 36 48 8b 14 d5 60 de 37 bc 66 90 48 63 8a 40 9e 02 00 48 85 c0 48 0f 44 05 8a a6 11 02 48 8b 9c c8 90 08 00 00 <48> 3b 93 20 06 00 00 75 31 48 8d 7b 50 e8 b0 a0 ce 00 49 89 04 24
Sep 09 23:08:06 pve kernel: RSP: 0018:ffffb1413431b638 EFLAGS: 00010286
Sep 09 23:08:06 pve kernel: RAX: ffff8ef7ca3e0000 RBX: 0000000000000000 RCX: 0000000000000000
Sep 09 23:08:06 pve kernel: RDX: ffff8f063f7d5000 RSI: ffffb1413431b680 RDI: ffffed60e62a2c00
Sep 09 23:08:06 pve kernel: RBP: ffffb1413431b648 R08: 0000000000000000 R09: 0000000000000000
Sep 09 23:08:06 pve kernel: R10: 0000000000000000 R11: 0000000000000000 R12: ffffb1413431b680
Sep 09 23:08:06 pve kernel: R13: ffff8ef6ae088000 R14: ffff8efcca535648 R15: 0000000000000007
Sep 09 23:08:06 pve kernel: FS:  00007582ed600640(0000) GS:ffff8f05ff480000(0000) knlGS:0000000000000000
Sep 09 23:08:06 pve kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Sep 09 23:08:06 pve kernel: CR2: 0000000000000620 CR3: 00000002506c4000 CR4: 0000000000f52ef0
Sep 09 23:08:06 pve kernel: PKRU: 55555554
Sep 09 23:08:06 pve kernel: Call Trace:
Sep 09 23:08:06 pve kernel:  <TASK>
Sep 09 23:08:06 pve kernel:  ? show_regs+0x6d/0x80
Sep 09 23:08:06 pve kernel:  ? __die+0x24/0x80
Sep 09 23:08:06 pve kernel:  ? page_fault_oops+0x176/0x500
Sep 09 23:08:06 pve kernel:  ? do_user_addr_fault+0x2f9/0x6b0
Sep 09 23:08:06 pve kernel:  ? exc_page_fault+0x83/0x1b0
Sep 09 23:08:06 pve kernel:  ? asm_exc_page_fault+0x27/0x30
Sep 09 23:08:06 pve kernel:  ? folio_lruvec_lock_irqsave+0x4e/0xa0
Sep 09 23:08:06 pve kernel:  release_pages+0x267/0x4c0
Sep 09 23:08:06 pve kernel:  ? smp_call_function_many_cond+0x113/0x500
Sep 09 23:08:06 pve kernel:  free_pages_and_swap_cache+0x4a/0x60
Sep 09 23:08:06 pve kernel:  tlb_batch_pages_flush+0x43/0x80
Sep 09 23:08:06 pve kernel:  tlb_flush_mmu+0x3d/0x110
Sep 09 23:08:06 pve kernel:  unmap_page_range+0xd36/0x11c0
Sep 09 23:08:06 pve kernel:  unmap_single_vma+0x89/0xf0
Sep 09 23:08:06 pve kernel:  unmap_vmas+0xb5/0x190
Sep 09 23:08:06 pve kernel:  unmap_region+0xe8/0x180
Sep 09 23:08:06 pve kernel:  do_vmi_align_munmap+0x3e8/0x5b0
Sep 09 23:08:06 pve kernel:  do_vmi_munmap+0xdf/0x190
Sep 09 23:08:06 pve kernel:  __vm_munmap+0xad/0x180
Sep 09 23:08:06 pve kernel:  __x64_sys_munmap+0x27/0x40
Sep 09 23:08:06 pve kernel:  x64_sys_call+0x1b1f/0x24b0
Sep 09 23:08:06 pve kernel:  do_syscall_64+0x81/0x170
Sep 09 23:08:06 pve kernel:  ? __update_load_avg_cfs_rq+0x380/0x3f0
Sep 09 23:08:06 pve kernel:  ? update_load_avg+0x82/0x830
Sep 09 23:08:06 pve kernel:  ? trigger_load_balance+0x167/0x370
Sep 09 23:08:06 pve kernel:  ? scheduler_tick+0x134/0x320
Sep 09 23:08:06 pve kernel:  ? account_user_time+0xa2/0xc0
Sep 09 23:08:06 pve kernel:  ? update_process_times+0x8e/0xb0
Sep 09 23:08:06 pve kernel:  ? tick_sched_handle+0x32/0x70
Sep 09 23:08:06 pve kernel:  ? timerqueue_add+0xa6/0xd0
Sep 09 23:08:06 pve kernel:  ? ktime_get+0x45/0xc0
Sep 09 23:08:06 pve kernel:  ? __pfx_tick_nohz_highres_handler+0x10/0x10
Sep 09 23:08:06 pve kernel:  ? lapic_next_deadline+0x2c/0x50
Sep 09 23:08:06 pve kernel:  ? clockevents_program_event+0xb3/0x140
Sep 09 23:08:06 pve kernel:  ? tick_program_event+0x43/0xa0
Sep 09 23:08:06 pve kernel:  ? hrtimer_interrupt+0x11f/0x250
Sep 09 23:08:06 pve kernel:  ? irqentry_exit_to_user_mode+0x7e/0x260
Sep 09 23:08:06 pve kernel:  ? irqentry_exit+0x43/0x50
Sep 09 23:08:06 pve kernel:  entry_SYSCALL_64_after_hwframe+0x78/0x80
Sep 09 23:08:06 pve kernel: RIP: 0033:0x7583f3c98a7b
Sep 09 23:08:06 pve kernel: Code: 8b 15 b9 b3 0f 00 f7 d8 64 89 02 48 c7 c0 ff ff ff ff c3 66 2e 0f 1f 84 00 00 00 00 00 66 90 f3 0f 1e fa b8 0b 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 85 b3 0f 00 f7 d8 64 89 01 48
Sep 09 23:08:06 pve kernel: RSP: 002b:00007582ed5fd348 EFLAGS: 00000206 ORIG_RAX: 000000000000000b
Sep 09 23:08:06 pve kernel: RAX: ffffffffffffffda RBX: ffffffffffffff80 RCX: 00007583f3c98a7b
Sep 09 23:08:06 pve kernel: RDX: 0000000000000000 RSI: 0000000008955000 RDI: 0000758126c00000
Sep 09 23:08:06 pve kernel: RBP: 0000000000000022 R08: 0000758126c00000 R09: 0000000000000000
Sep 09 23:08:06 pve kernel: R10: 00000000000000ee R11: 0000000000000206 R12: 0000758126c00030
Sep 09 23:08:06 pve kernel: R13: 0000000000000000 R14: 0000758126c00080 R15: 0000758126c00068
Sep 09 23:08:06 pve kernel:  </TASK>
Sep 09 23:08:06 pve kernel: Modules linked in: uas usb_storage msr macvlan nf_conntrack_netlink xt_nat nft_chain_nat xt_MASQUERADE nf_nat xfrm_user xfrm_algo overlay ip6t_REJECT nf_reject_ipv6 xt_hl ip6t_rt ipt_REJECT nf_reject_ipv4 xt_LOG nf_log_syslog xt_multiport nft_limit xt_limit xt_addrtype xt_tcpudp xt_conntrack nf_conntrack nft_compat tcp_diag inet_diag veth ebtable_filter ebtables ip_set ip6table_raw iptable_raw ip6table_filter ip6_tables iptable_filter xt_TPROXY nf_tproxy_ipv6 nf_tproxy_ipv4 nf_defrag_ipv6 nf_defrag_ipv4 softdog nf_tables nvme_fabrics sunrpc binfmt_misc bonding tls nfnetlink_log nfnetlink snd_hda_codec_hdmi intel_rapl_msr intel_rapl_common intel_uncore_frequency intel_uncore_frequency_common intel_tcc_cooling i915(OE) snd_sof_pci_intel_tgl snd_sof_intel_hda_common soundwire_intel snd_sof_intel_hda_mlink soundwire_cadence snd_sof_intel_hda snd_sof_pci snd_sof_xtensa_dsp snd_sof x86_pkg_temp_thermal intel_powerclamp snd_sof_utils snd_soc_hdac_hda snd_hda_ext_core snd_soc_acpi_intel_match kvm_intel
Sep 09 23:08:06 pve kernel:  snd_soc_acpi soundwire_generic_allocation soundwire_bus kvm xe iwlmvm snd_soc_core snd_compress ac97_bus snd_pcm_dmaengine snd_usb_audio snd_hda_intel crct10dif_pclmul drm_gpuvm snd_intel_dspcfg polyval_clmulni drm_exec polyval_generic snd_intel_sdw_acpi mac80211 snd_usbmidi_lib ghash_clmulni_intel gpu_sched snd_ump sha256_ssse3 snd_hda_codec drm_buddy snd_rawmidi sha1_ssse3 drm_suballoc_helper aesni_intel btusb drm_ttm_helper snd_seq_device snd_hda_core btrtl ttm mc snd_hwdep crypto_simd btintel mei_pxp mei_hdcp libarc4 cryptd snd_pcm btbcm drm_display_helper btmtk snd_timer cec rapl cmdlinepart bluetooth iwlwifi intel_pmc_core spi_nor rc_core mei_me snd ecdh_generic intel_cstate pcspkr eeepc_wmi asus_nb_wmi wmi_bmof cfg80211 pmt_telemetry mtd mei i2c_algo_bit soundcore ecc intel_vsec plx_dma pmt_class zfs(PO) acpi_pad acpi_tad joydev input_leds mac_hid spl(O) vhost_net vhost vhost_iotlb tap pkcs8_key_parser vfio_pci vfio_pci_core irqbypass vfio_iommu_type1 vfio iommufd nct6775 nct6775_core hwmon_vid
Sep 09 23:08:06 pve kernel:  coretemp efi_pstore dmi_sysfs ip_tables x_tables autofs4 btrfs blake2b_generic xor raid6_pq libcrc32c hid_generic usbmouse usbkbd usbhid hid mfd_aaeon asus_wmi ledtrig_audio sparse_keymap nvme xhci_pci platform_profile crc32_pclmul xhci_pci_renesas spi_intel_pci i2c_i801 thunderbolt intel_lpss_pci ahci nvme_core spi_intel intel_lpss i2c_smbus xhci_hcd igc libahci idma64 vmd nvme_auth video wmi pinctrl_alderlake [last unloaded: cpuid]
Sep 09 23:08:06 pve kernel: CR2: 0000000000000620
Sep 09 23:08:06 pve kernel: ---[ end trace 0000000000000000 ]---
Sep 09 23:08:06 pve kernel: RIP: 0010:folio_lruvec_lock_irqsave+0x4e/0xa0
Sep 09 23:08:06 pve kernel: Code: 8b 17 48 c1 ea 36 48 8b 14 d5 60 de 37 bc 66 90 48 63 8a 40 9e 02 00 48 85 c0 48 0f 44 05 8a a6 11 02 48 8b 9c c8 90 08 00 00 <48> 3b 93 20 06 00 00 75 31 48 8d 7b 50 e8 b0 a0 ce 00 49 89 04 24
Sep 09 23:08:06 pve kernel: RSP: 0018:ffffb1413431b638 EFLAGS: 00010286
Sep 09 23:08:06 pve kernel: RAX: ffff8ef7ca3e0000 RBX: 0000000000000000 RCX: 0000000000000000
Sep 09 23:08:06 pve kernel: RDX: ffff8f063f7d5000 RSI: ffffb1413431b680 RDI: ffffed60e62a2c00
Sep 09 23:08:06 pve kernel: RBP: ffffb1413431b648 R08: 0000000000000000 R09: 0000000000000000
Sep 09 23:08:06 pve kernel: R10: 0000000000000000 R11: 0000000000000000 R12: ffffb1413431b680
Sep 09 23:08:06 pve kernel: R13: ffff8ef6ae088000 R14: ffff8efcca535648 R15: 0000000000000007
Sep 09 23:08:06 pve kernel: FS:  00007582ed600640(0000) GS:ffff8f05ff480000(0000) knlGS:0000000000000000
Sep 09 23:08:06 pve kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Sep 09 23:08:06 pve kernel: CR2: 0000000000000620 CR3: 00000002506c4000 CR4: 0000000000f52ef0
Sep 09 23:08:06 pve kernel: PKRU: 55555554
Sep 09 23:08:06 pve kernel: note: pt_main_thread[3545995] exited with irqs disabled

Kosh · Sep 10, 2024

I will share this information

This problem is typical for desktop hardware, I tested it on the CPU - i9 14900K and Ryzen 9 7900X
When the hypervisor crashed, there was always a docker inside the VM and the VM had direct access to the CPU (otherwise the docker does not work)
If you remove direct access to the CPU, the problem goes away, but then the docker does not work
Our solution was to abandon the desktop hardware and move critical VMs to server platforms, there is no such problem

aliang · Sep 11, 2024

Kosh said:
I will share this information

This problem is typical for desktop hardware, I tested it on the CPU - i9 14900K and Ryzen 9 7900X
When the hypervisor crashed, there was always a docker inside the VM and the VM had direct access to the CPU (otherwise the docker does not work)
If you remove direct access to the CPU, the problem goes away, but then the docker does not work
Our solution was to abandon the desktop hardware and move critical VMs to server platforms, there is no such problem

however, I just want to have a mini server in my home.
now i'm trying update kernel to 6.8.12-1-pve, Someone said they fixed this problem in the new version

[SOLVED] BUG: kernel NULL pointer dereference, address: 0000000000000000

Member

Attachments

Member

Member

Member

Member

Member

Proxmox Staff Member

Active Member

Member

New Member

New Member

New Member

New Member

New Member

New Member

Proxmox Staff Member

New Member

New Member

Active Member

New Member