Hello,
I am new Proxmox user. Fresh installation using kernel 6.1.6-1-pve.
I only installed openmediavault and started to configure my second VM which is rtorrent client running on debian testing.
I noticed my omv went down.
Here is the crash dump:
I then restarted pve but omv was unable to start again. I disabled "Start on boot" for both VMs and now it is running after I started them one by one.
longer crash dump:
https://pastebin.com/d2BAeRj1
PS. Crashed again this time whole PVE. With kinda different error:
https://pastebin.com/BEZqF2R2
Will try older kernel.
Strange it was working fine whole day.
Maybe this crashes started when I initiated nfs connection between rtorrent -> omv.
I think I saw somewhere that somebody reported that the issue was virtio network driver and switching to e1000 fixed the issue. I will try that after I try kernel 5.x.
I am new Proxmox user. Fresh installation using kernel 6.1.6-1-pve.
I only installed openmediavault and started to configure my second VM which is rtorrent client running on debian testing.
I noticed my omv went down.
Here is the crash dump:
Code:
Feb 4 23:56:00 pve kernel: [31985.271164] BUG: kernel NULL pointer dereference, address: 0000000000000005
Feb 4 23:56:00 pve kernel: [31985.271171] #PF: supervisor write access in kernel mode
Feb 4 23:56:00 pve kernel: [31985.271174] #PF: error_code(0x0002) - not-present page
Feb 4 23:56:00 pve kernel: [31985.271176] PGD 0 P4D 0
Feb 4 23:56:00 pve kernel: [31985.271180] Oops: 0002 [#1] PREEMPT SMP PTI
Feb 4 23:56:00 pve kernel: [31985.271184] CPU: 3 PID: 1238 Comm: kvm Tainted: P IO 6.1.6-1-pve #1
Feb 4 23:56:00 pve kernel: [31985.271188] Hardware name: To Be Filled By O.E.M. To Be Filled By O.E.M./Z170 Extreme4, BIOS P7.50 01/22/2018
Feb 4 23:56:00 pve kernel: [31985.271193] RIP: 0010:__bio_split_to_limits+0x226/0x490
Feb 4 23:56:00 pve kernel: [31985.271197] Code: f7 de 23 74 24 24 89 74 24 24 c1 ee 09 48 8b 0c 24 ba 00 0c 00 00 4c 89 e7 e8 06 39 ff ff 49 89 c5 48 85 c0 0f 84 fd 00 00 00 <41> 81 4d 10 00 40 00 00 41 8b 5d 28 48 b8 00 00 00 00 00 00 00 80
Feb 4 23:56:00 pve kernel: [31985.271203] RSP: 0018:ffffbb3c8087f898 EFLAGS: 00010282
Feb 4 23:56:00 pve kernel: [31985.271205] RAX: ffff929289635201 RBX: 0000000000050000 RCX: 00000000227dc003
Feb 4 23:56:00 pve kernel: [31985.271208] RDX: 00000000227da003 RSI: 43999fd73259a0ba RDI: 000000000003a150
Feb 4 23:56:00 pve kernel: [31985.271211] RBP: ffffbb3c8087f900 R08: ffff929289635900 R09: ffff929290036860
Feb 4 23:56:00 pve kernel: [31985.271213] R10: 0000000000000000 R11: 0000000000000002 R12: ffff929289635940
Feb 4 23:56:00 pve kernel: [31985.271216] R13: fffffffffffffff5 R14: 0000000000000000 R15: ffff929290e69bb8
Feb 4 23:56:00 pve kernel: [31985.271219] FS: 00007f41db84f1c0(0000) GS:ffff9295e66c0000(0000) knlGS:0000000000000000
Feb 4 23:56:00 pve kernel: [31985.271222] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Feb 4 23:56:00 pve kernel: [31985.271224] CR2: 0000000000000005 CR3: 000000010ec4e001 CR4: 00000000003726e0
Feb 4 23:56:00 pve kernel: [31985.271227] Call Trace:
Feb 4 23:56:00 pve kernel: [31985.271229] <TASK>
Feb 4 23:56:00 pve kernel: [31985.271232] blk_mq_submit_bio+0xae/0x590
Feb 4 23:56:00 pve kernel: [31985.271236] ? __iov_iter_get_pages_alloc+0x149/0x900
Feb 4 23:56:00 pve kernel: [31985.271240] __submit_bio+0xff/0x190
Feb 4 23:56:00 pve kernel: [31985.271243] submit_bio_noacct_nocheck+0x257/0x2a0
Feb 4 23:56:00 pve kernel: [31985.271247] submit_bio_noacct+0x20d/0x610
Feb 4 23:56:00 pve kernel: [31985.271250] submit_bio+0x28/0x90
Feb 4 23:56:00 pve kernel: [31985.271253] __blkdev_direct_IO_async+0x124/0x200
Feb 4 23:56:00 pve kernel: [31985.271257] blkdev_direct_IO+0x49/0x90
Feb 4 23:56:00 pve kernel: [31985.271260] generic_file_direct_write+0x9f/0x1e0
Feb 4 23:56:00 pve kernel: [31985.271265] __generic_file_write_iter+0xa1/0x1b0
Feb 4 23:56:00 pve kernel: [31985.271268] blkdev_write_iter+0x10f/0x1c0
Feb 4 23:56:00 pve kernel: [31985.271272] io_write+0x130/0x440
Feb 4 23:56:00 pve kernel: [31985.271276] ? get_sigset_argpack.constprop.0+0x70/0x70
Feb 4 23:56:00 pve kernel: [31985.271280] ? fget+0x83/0xb0
Feb 4 23:56:00 pve kernel: [31985.271283] ? io_read+0x510/0x510
Feb 4 23:56:00 pve kernel: [31985.271286] io_issue_sqe+0x6b/0x410
Feb 4 23:56:00 pve kernel: [31985.271289] io_submit_sqes+0x21b/0x650
Feb 4 23:56:00 pve kernel: [31985.271292] ? __fget_light.part.0+0x8c/0xd0
Feb 4 23:56:00 pve kernel: [31985.271296] __do_sys_io_uring_enter+0x39d/0xa50
Feb 4 23:56:00 pve kernel: [31985.271300] __x64_sys_io_uring_enter+0x29/0x30
Feb 4 23:56:00 pve kernel: [31985.271303] do_syscall_64+0x59/0x90
Feb 4 23:56:00 pve kernel: [31985.271307] ? handle_mm_fault+0xe6/0x2d0
Feb 4 23:56:00 pve kernel: [31985.271310] ? exit_to_user_mode_prepare+0x37/0x180
Feb 4 23:56:00 pve kernel: [31985.271313] ? irqentry_exit_to_user_mode+0x9/0x20
Feb 4 23:56:00 pve kernel: [31985.271317] ? irqentry_exit+0x3b/0x50
Feb 4 23:56:00 pve kernel: [31985.271320] ? exc_page_fault+0x87/0x180
Feb 4 23:56:00 pve kernel: [31985.271323] entry_SYSCALL_64_after_hwframe+0x63/0xcd
Feb 4 23:56:00 pve kernel: [31985.271326] RIP: 0033:0x7f41e60312e9
Feb 4 23:56:00 pve kernel: [31985.271329] Code: 00 c3 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 77 8b 0d 00 f7 d8 64 89 01 48
Feb 4 23:56:00 pve kernel: [31985.271334] RSP: 002b:00007ffe73825b18 EFLAGS: 00000216 ORIG_RAX: 00000000000001aa
Feb 4 23:56:00 pve kernel: [31985.271337] RAX: ffffffffffffffda RBX: 00007f416d89f860 RCX: 00007f41e60312e9
Feb 4 23:56:00 pve kernel: [31985.271340] RDX: 0000000000000000 RSI: 0000000000000001 RDI: 0000000000000016
Feb 4 23:56:00 pve kernel: [31985.271342] RBP: 0000000000000000 R08: 0000000000000000 R09: 0000000000000008
Feb 4 23:56:00 pve kernel: [31985.271344] R10: 0000000000000000 R11: 0000000000000216 R12: 00005577a2335778
Feb 4 23:56:00 pve kernel: [31985.271347] R13: 00005577a2335830 R14: 00005577a2335770 R15: 0000000000000000
Feb 4 23:56:00 pve kernel: [31985.271351] </TASK>
Feb 4 23:56:00 pve kernel: [31985.271352] Modules linked in: nls_utf8 ntfs veth ebtable_filter ebtables ip_set ip6table_raw iptable_raw ip6table_filter ip6_tables iptable_filter bpfilter nf_tables bonding tls softdog nfnetlink_log nfnetlink snd_hda_codec_hdmi snd_hda_codec_realtek snd_hda_codec_generic ledtrig_audio intel_rapl_msr intel_rapl_common intel_tcc_cooling x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel i915 kvm snd_hda_intel drm_buddy snd_intel_dspcfg ttm snd_intel_sdw_acpi irqbypass drm_display_helper crct10dif_pclmul snd_hda_codec polyval_clmulni polyval_generic cec ghash_clmulni_intel snd_hda_core sha512_ssse3 rc_core snd_hwdep aesni_intel drm_kms_helper mei_hdcp mei_pxp snd_pcm crypto_simd i2c_algo_bit cryptd snd_timer fb_sys_fops rapl syscopyarea intel_cstate sysfillrect snd mei_me zfs(PO) sysimgblt joydev soundcore intel_wmi_thunderbolt input_leds pcspkr efi_pstore ee1004 mxm_wmi mei intel_pch_thermal zunicode(PO) zzstd(O) zlua(O) mac_hid zavl(PO) acpi_pad icp(PO) zcommon(PO) znvpair(PO)
Feb 4 23:56:00 pve kernel: [31985.271396] spl(O) vhost_net vhost vhost_iotlb tap ib_iser rdma_cm iw_cm ib_cm ib_core iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi drm sunrpc ip_tables x_tables autofs4 btrfs blake2b_generic xor raid6_pq zstd_compress simplefb hid_logitech_hidpp hid_logitech_dj hid_generic usbkbd usbmouse usbhid uas hid usb_storage dm_thin_pool dm_persistent_data dm_bio_prison dm_bufio libcrc32c nvme xhci_pci crc32_pclmul nvme_core i2c_i801 xhci_pci_renesas e1000e i2c_smbus ahci nvme_common libahci xhci_hcd video wmi
Feb 4 23:56:00 pve kernel: [31985.271447] CR2: 0000000000000005
Feb 4 23:56:00 pve kernel: [31985.271449] ---[ end trace 0000000000000000 ]---
Feb 4 23:56:00 pve kernel: [31986.390750] RIP: 0010:__bio_split_to_limits+0x226/0x490
Feb 4 23:56:00 pve kernel: [31986.390756] Code: f7 de 23 74 24 24 89 74 24 24 c1 ee 09 48 8b 0c 24 ba 00 0c 00 00 4c 89 e7 e8 06 39 ff ff 49 89 c5 48 85 c0 0f 84 fd 00 00 00 <41> 81 4d 10 00 40 00 00 41 8b 5d 28 48 b8 00 00 00 00 00 00 00 80
Feb 4 23:56:00 pve kernel: [31986.390761] RSP: 0018:ffffbb3c8087f898 EFLAGS: 00010282
Feb 4 23:56:00 pve kernel: [31986.390764] RAX: ffff929289635201 RBX: 0000000000050000 RCX: 00000000227dc003
Feb 4 23:56:00 pve kernel: [31986.390766] RDX: 00000000227da003 RSI: 43999fd73259a0ba RDI: 000000000003a150
Feb 4 23:56:00 pve kernel: [31986.390769] RBP: ffffbb3c8087f900 R08: ffff929289635900 R09: ffff929290036860
Feb 4 23:56:00 pve kernel: [31986.390771] R10: 0000000000000000 R11: 0000000000000002 R12: ffff929289635940
Feb 4 23:56:00 pve kernel: [31986.390773] R13: fffffffffffffff5 R14: 0000000000000000 R15: ffff929290e69bb8
Feb 4 23:56:00 pve kernel: [31986.390776] FS: 00007f41db84f1c0(0000) GS:ffff9295e66c0000(0000) knlGS:0000000000000000
Feb 4 23:56:00 pve kernel: [31986.390779] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Feb 4 23:56:00 pve kernel: [31986.390781] CR2: 0000000000000005 CR3: 000000010ec4e001 CR4: 00000000003726e0
Feb 4 23:56:00 pve kernel: [31986.390815] ------------[ cut here ]------------
Feb 4 23:56:00 pve kernel: [31986.390817] WARNING: CPU: 3 PID: 1238 at kernel/exit.c:765 do_exit+0x88a/0xab0
Feb 4 23:56:00 pve kernel: [31986.390822] Modules linked in: nls_utf8 ntfs veth ebtable_filter ebtables ip_set ip6table_raw iptable_raw ip6table_filter ip6_tables iptable_filter bpfilter nf_tables bonding tls softdog nfnetlink_log nfnetlink snd_hda_codec_hdmi snd_hda_codec_realtek snd_hda_codec_generic ledtrig_audio intel_rapl_msr intel_rapl_common intel_tcc_cooling x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel i915 kvm snd_hda_intel drm_buddy snd_intel_dspcfg ttm snd_intel_sdw_acpi irqbypass drm_display_helper crct10dif_pclmul snd_hda_codec polyval_clmulni polyval_generic cec ghash_clmulni_intel snd_hda_core sha512_ssse3 rc_core snd_hwdep aesni_intel drm_kms_helper mei_hdcp mei_pxp snd_pcm crypto_simd i2c_algo_bit cryptd snd_timer fb_sys_fops rapl syscopyarea intel_cstate sysfillrect snd mei_me zfs(PO) sysimgblt joydev soundcore intel_wmi_thunderbolt input_leds pcspkr efi_pstore ee1004 mxm_wmi mei intel_pch_thermal zunicode(PO) zzstd(O) zlua(O) mac_hid zavl(PO) acpi_pad icp(PO) zcommon(PO) znvpair(PO)
Feb 4 23:56:00 pve kernel: [31986.390864] spl(O) vhost_net vhost vhost_iotlb tap ib_iser rdma_cm iw_cm ib_cm ib_core iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi drm sunrpc ip_tables x_tables autofs4 btrfs blake2b_generic xor raid6_pq zstd_compress simplefb hid_logitech_hidpp hid_logitech_dj hid_generic usbkbd usbmouse usbhid uas hid usb_storage dm_thin_pool dm_persistent_data dm_bio_prison dm_bufio libcrc32c nvme xhci_pci crc32_pclmul nvme_core i2c_i801 xhci_pci_renesas e1000e i2c_smbus ahci nvme_common libahci xhci_hcd video wmi
Feb 4 23:56:00 pve kernel: [31986.390940] CPU: 3 PID: 1238 Comm: kvm Tainted: P D IO 6.1.6-1-pve #1
Feb 4 23:56:00 pve kernel: [31986.390945] Hardware name: To Be Filled By O.E.M. To Be Filled By O.E.M./Z170 Extreme4, BIOS P7.50 01/22/2018
Feb 4 23:56:00 pve kernel: [31986.390949] RIP: 0010:do_exit+0x88a/0xab0
Feb 4 23:56:00 pve kernel: [31986.390953] Code: e9 a1 f9 ff ff 48 8b bb f0 09 00 00 31 f6 e8 5d d9 ff ff e9 89 fd ff ff 4c 89 ee bf 05 06 00 00 e8 db fe 00 00 e9 90 f8 ff ff <0f> 0b e9 e0 f7 ff ff 48 89 df e8 87 9b 10 00 e9 88 f9 ff ff 48 8b
Feb 4 23:56:00 pve kernel: [31986.390961] RSP: 0018:ffffbb3c8087fed8 EFLAGS: 00010286
Feb 4 23:56:00 pve kernel: [31986.390964] RAX: 0000000000000000 RBX: ffff929285d41a00 RCX: 0000000000000000
Feb 4 23:56:00 pve kernel: [31986.390968] RDX: 0000000000000001 RSI: 0000000000000027 RDI: ffff929280e1f380
Feb 4 23:56:00 pve kernel: [31986.390972] RBP: ffffbb3c8087ff28 R08: 0000000000000003 R09: 0000000000000001
Feb 4 23:56:00 pve kernel: [31986.390975] R10: 000000000000000a R11: 3030303030302043 R12: ffff92928b8cbf00
Feb 4 23:56:00 pve kernel: [31986.390979] R13: 0000000000000009 R14: ffff929280e1f380 R15: 0000000000000046
Feb 4 23:56:00 pve kernel: [31986.390982] FS: 00007f41db84f1c0(0000) GS:ffff9295e66c0000(0000) knlGS:0000000000000000
Feb 4 23:56:00 pve kernel: [31986.390987] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Feb 4 23:56:00 pve kernel: [31986.390990] CR2: 0000000000000005 CR3: 000000010ec4e001 CR4: 00000000003726e0
Feb 4 23:56:00 pve kernel: [31986.390994] Call Trace:
Feb 4 23:56:00 pve kernel: [31986.390996] <TASK>
Feb 4 23:56:00 pve kernel: [31986.391000] make_task_dead+0x5a/0x60
Feb 4 23:56:00 pve kernel: [31986.391004] rewind_stack_and_make_dead+0x17/0x20
Feb 4 23:56:00 pve kernel: [31986.391009] RIP: 0033:0x7f41e60312e9
Feb 4 23:56:00 pve kernel: [31986.391013] Code: 00 c3 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 77 8b 0d 00 f7 d8 64 89 01 48
Feb 4 23:56:00 pve kernel: [31986.391020] RSP: 002b:00007ffe73825b18 EFLAGS: 00000216 ORIG_RAX: 00000000000001aa
Feb 4 23:56:00 pve kernel: [31986.391025] RAX: ffffffffffffffda RBX: 00007f416d89f860 RCX: 00007f41e60312e9
Feb 4 23:56:00 pve kernel: [31986.391029] RDX: 0000000000000000 RSI: 0000000000000001 RDI: 0000000000000016
Feb 4 23:56:00 pve kernel: [31986.391032] RBP: 0000000000000000 R08: 0000000000000000 R09: 0000000000000008
Feb 4 23:56:00 pve kernel: [31986.391036] R10: 0000000000000000 R11: 0000000000000216 R12: 00005577a2335778
Feb 4 23:56:00 pve kernel: [31986.391039] R13: 00005577a2335830 R14: 00005577a2335770 R15: 0000000000000000
Feb 4 23:56:00 pve kernel: [31986.391044] </TASK>
Feb 4 23:56:00 pve kernel: [31986.391046] ---[ end trace 0000000000000000 ]---I then restarted pve but omv was unable to start again. I disabled "Start on boot" for both VMs and now it is running after I started them one by one.
longer crash dump:
https://pastebin.com/d2BAeRj1
PS. Crashed again this time whole PVE. With kinda different error:
https://pastebin.com/BEZqF2R2
Will try older kernel.
Strange it was working fine whole day.
Maybe this crashes started when I initiated nfs connection between rtorrent -> omv.
I think I saw somewhere that somebody reported that the issue was virtio network driver and switching to e1000 fixed the issue. I will try that after I try kernel 5.x.
Last edited: