[SOLVED] PVE Kernel 5.3.18-1 breaks PCI Passthrough

[mstinsky]

Greatings,

I upgraded one of my servers today and the kernel upgrade from 5.3.13-3 to 5.3.18-1 breaks PCI passthrough for my VM. The VM startup gets stuck with no response, strace on the process shows no activity.

The error message which seems to be responsible for that is the following:

Feb 10 20:39:50 pve01 pvedaemon[3667]: start VM 204: UPID:pve01:00000E53:000023DA:5E41B186:qmstart:204:root@pam:
Feb 10 20:39:50 pve01 kernel: [   91.928584] general protection fault: 0000 [#1] SMP PTI
Feb 10 20:39:50 pve01 kernel: [   91.928607] CPU: 1 PID: 3667 Comm: task UPID:pve01 Tainted: P           O      5.3.18-1-pve #1
Feb 10 20:39:50 pve01 kernel: [   91.928627] Hardware name: ASUSTeK COMPUTER INC. P10S-C Series/P10S-C Series, BIOS 4402 03/07/2018
Feb 10 20:39:50 pve01 kernel: [   91.928651] RIP: 0010:remove_files.isra.1+0x24/0x70
Feb 10 20:39:50 pve01 kernel: [   91.928664] Code: 00 00 00 0f 1f 00 0f 1f 44 00 00 55 48 89 e5 41 55 49 89 d5 41 54 49 89 fc 53 48 85 f6 74 24 48 8b 06 48 89 f3 48 85 c0 74 19 <48> 8b 30 31 d2 4c 89 e7 48 83 c3 08 e8 db d3 ff ff 48 8b 03 48 85
Feb 10 20:39:50 pve01 kernel: [   91.928705] RSP: 0018:ffffa0db0dc53c18 EFLAGS: 00010206
Feb 10 20:39:50 pve01 kernel: [   91.928719] RAX: 5efbb6e90f91173f RBX: ffff8edf542f7a80 RCX: 0000000000000000
Feb 10 20:39:50 pve01 kernel: [   91.928736] RDX: ffff8edf5d2c9488 RSI: ffff8edf542f7a80 RDI: ffff8edf543886e8
Feb 10 20:39:50 pve01 kernel: [   91.928753] RBP: ffffa0db0dc53c30 R08: 0000000000000000 R09: ffff8edf543882c0
Feb 10 20:39:50 pve01 kernel: [   91.928770] R10: 0000000000000000 R11: 0000000000000001 R12: ffff8edf543886e8
Feb 10 20:39:50 pve01 kernel: [   91.928787] R13: ffff8edf5d2c9488 R14: ffff8edf5d18a0b0 R15: fffffffffffffff2
Feb 10 20:39:50 pve01 kernel: [   91.928805] FS:  00007f8c7a9a51c0(0000) GS:ffff8edf5fb00000(0000) knlGS:0000000000000000
Feb 10 20:39:50 pve01 kernel: [   91.928824] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Feb 10 20:39:50 pve01 kernel: [   91.928838] CR2: 00007f8c6e3c4db8 CR3: 00000003d8a38004 CR4: 00000000003606e0
Feb 10 20:39:50 pve01 kernel: [   91.928855] Call Trace:
Feb 10 20:39:50 pve01 kernel: [   91.928866]  sysfs_remove_group+0x44/0x90
Feb 10 20:39:50 pve01 kernel: [   91.928878]  sysfs_remove_groups+0x2e/0x50
Feb 10 20:39:50 pve01 kernel: [   91.928890]  device_remove_attrs+0x3e/0x70
Feb 10 20:39:50 pve01 kernel: [   91.928901]  device_del+0x160/0x370
Feb 10 20:39:50 pve01 kernel: [   91.928911]  cdev_device_del+0x1a/0x40
Feb 10 20:39:50 pve01 kernel: [   91.928922]  posix_clock_unregister+0x26/0x50
Feb 10 20:39:50 pve01 kernel: [   91.928934]  ptp_clock_unregister+0x6f/0x80
Feb 10 20:39:50 pve01 kernel: [   91.928951]  igb_ptp_stop+0x26/0x50 [igb]
Feb 10 20:39:50 pve01 kernel: [   91.928964]  igb_remove+0x4b/0x130 [igb]
Feb 10 20:39:50 pve01 kernel: [   91.928975]  pci_device_remove+0x3e/0xc0
Feb 10 20:39:50 pve01 kernel: [   91.928986]  device_release_driver_internal+0xe0/0x1b0
Feb 10 20:39:50 pve01 kernel: [   91.928999]  device_driver_detach+0x14/0x20
Feb 10 20:39:50 pve01 kernel: [   91.929011]  unbind_store+0xf9/0x130
Feb 10 20:39:50 pve01 kernel: [   91.929021]  drv_attr_store+0x27/0x40
Feb 10 20:39:50 pve01 kernel: [   91.929032]  sysfs_kf_write+0x3b/0x40
Feb 10 20:39:50 pve01 kernel: [   91.929043]  kernfs_fop_write+0xda/0x1c0
Feb 10 20:39:50 pve01 kernel: [   91.929053]  __vfs_write+0x1b/0x40
Feb 10 20:39:50 pve01 kernel: [   91.929063]  vfs_write+0xab/0x1b0
Feb 10 20:39:50 pve01 kernel: [   91.929073]  ksys_write+0x61/0xe0
Feb 10 20:39:50 pve01 kernel: [   91.929082]  __x64_sys_write+0x1a/0x20
Feb 10 20:39:50 pve01 kernel: [   91.929093]  do_syscall_64+0x5a/0x130
Feb 10 20:39:50 pve01 kernel: [   91.929554]  entry_SYSCALL_64_after_hwframe+0x44/0xa9
Feb 10 20:39:50 pve01 kernel: [   91.930019] RIP: 0033:0x7f8c7abb2471
Feb 10 20:39:50 pve01 kernel: [   91.930477] Code: 00 00 75 05 48 83 c4 58 c3 e8 0b 4d ff ff 66 2e 0f 1f 84 00 00 00 00 00 90 8b 05 da ef 00 00 85 c0 75 16 b8 01 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 57 c3 66 0f 1f 44 00 00 41 54 49 89 d4 55 48
Feb 10 20:39:50 pve01 kernel: [   91.931438] RSP: 002b:00007ffcb70c1d18 EFLAGS: 00000246 ORIG_RAX: 0000000000000001
Feb 10 20:39:50 pve01 kernel: [   91.931926] RAX: ffffffffffffffda RBX: 00005590e1c9d260 RCX: 00007f8c7abb2471
Feb 10 20:39:50 pve01 kernel: [   91.932418] RDX: 000000000000000c RSI: 00005590e87d35f0 RDI: 000000000000000b
Feb 10 20:39:50 pve01 kernel: [   91.932915] RBP: 00005590e87d35f0 R08: 0000000000000000 R09: aaaaaaaaaaaaaaab
Feb 10 20:39:50 pve01 kernel: [   91.933419] R10: 00005590e87b5df8 R11: 0000000000000246 R12: 000000000000000c
Feb 10 20:39:50 pve01 kernel: [   91.933914] R13: 00005590e1c9d260 R14: 000000000000000b R15: 00005590e87d2df0
Feb 10 20:39:50 pve01 kernel: [   91.934405] Modules linked in: ebtable_filter ebtables ip_set ip6table_raw iptable_raw ip6table_filter ip6_tables sctp xt_conntrack xt_MASQUERADE nf_conntrack_netlink xfrm_user xfrm_algo xt_addrtype iptable_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 aufs iptable_filter bpfilter overlay softdog nfnetlink_log nfnetlink intel_rapl_msr intel_rapl_common x86_pkg_temp_thermal intel_powerclamp kvm_intel kvm crct10dif_pclmul crc32_pclmul ghash_clmulni_intel ast aesni_intel aes_x86_64 crypto_simd cryptd drm_vram_helper glue_helper intel_cstate ttm intel_rapl_perf drm_kms_helper pcspkr drm fb_sys_fops syscopyarea sysfillrect sysimgblt mei_me mei intel_pch_thermal mac_hid vhost_net vhost tap ib_iser rdma_cm iw_cm ib_cm ib_core iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi vfio_pci vfio_virqfd irqbypass vfio_iommu_type1 vfio nct6775 hwmon_vid coretemp sunrpc ip_tables x_tables autofs4 zfs(PO) zunicode(PO) zlua(PO) zavl(PO) icp(PO) zcommon(PO) znvpair(PO) spl(O) btrfs xor
Feb 10 20:39:50 pve01 kernel: [   91.934430]  zstd_compress raid6_pq libcrc32c i2c_i801 igb ahci i2c_algo_bit dca libahci video
Feb 10 20:39:50 pve01 kernel: [   91.939179] ---[ end trace f32eff54a40d7579 ]---
Feb 10 20:39:51 pve01 pvedaemon[2088]: <root@pam> end task UPID:pve01:00000E53:000023DA:5E41B186:qmstart:204:root@pam: unable to read tail (got 0 bytes)
Feb 10 20:39:51 pve01 kernel: [   93.231548] RIP: 0010:remove_files.isra.1+0x24/0x70
Feb 10 20:39:51 pve01 kernel: [   93.231550] Code: 00 00 00 0f 1f 00 0f 1f 44 00 00 55 48 89 e5 41 55 49 89 d5 41 54 49 89 fc 53 48 85 f6 74 24 48 8b 06 48 89 f3 48 85 c0 74 19 <48> 8b 30 31 d2 4c 89 e7 48 83 c3 08 e8 db d3 ff ff 48 8b 03 48 85
Feb 10 20:39:51 pve01 kernel: [   93.231551] RSP: 0018:ffffa0db0dc53c18 EFLAGS: 00010206
Feb 10 20:39:51 pve01 kernel: [   93.231552] RAX: 5efbb6e90f91173f RBX: ffff8edf542f7a80 RCX: 0000000000000000
Feb 10 20:39:51 pve01 kernel: [   93.231552] RDX: ffff8edf5d2c9488 RSI: ffff8edf542f7a80 RDI: ffff8edf543886e8
Feb 10 20:39:51 pve01 kernel: [   93.231553] RBP: ffffa0db0dc53c30 R08: 0000000000000000 R09: ffff8edf543882c0
Feb 10 20:39:51 pve01 kernel: [   93.231553] R10: 0000000000000000 R11: 0000000000000001 R12: ffff8edf543886e8
Feb 10 20:39:51 pve01 kernel: [   93.231554] R13: ffff8edf5d2c9488 R14: ffff8edf5d18a0b0 R15: fffffffffffffff2
Feb 10 20:39:51 pve01 kernel: [   93.231554] FS:  00007f8c7a9a51c0(0000) GS:ffff8edf5fb00000(0000) knlGS:0000000000000000
Feb 10 20:39:51 pve01 kernel: [   93.231555] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Feb 10 20:39:51 pve01 kernel: [   93.231555] CR2: 00007f8c6e3c4db8 CR3: 00000003d8a38004 CR4: 00000000003606e0

After I reverted to Kernel 5.3.13-3 everything is working as expected again. Any idea what could cause that?

 

[C.G.B. Spender]

Was this a Windows vm by any chance? I have similar/same issue with my win10 vm (made a post a few posts earlier), but my other vm (freebsd) with 3x pci passthrough worked fine on 5.3.18-1.

 

[mstinsky]

No, its a freebsd vm (opnsense) vm with 2 onboard nic ports passed through.

 

[mstinsky]

I tested the upgrade on a different server which also has a opnsense vm but with a complete seperate NIC passed through instead of single onboard nic ports. Surprisingly that VM just starts fine with Kernel 5.3.18-1 so it seems like the error is dependent on what PCI device you are passing through?

 

[C.G.B. Spender]

Sounds likely (can't reboot the server atm to test if the win10 vm would work without passthrough), but my freebsd machine has 3 standalone pci(e) cards in passthrough and that worked fine with the new kernel, the win10 vm that didn't work has a onboard nic passed through.

 

[Zok]

Had this same issue last night upon upgrading the kernel to 5.3.18-1 from 5.3.13-3. Have an Untangle VM with two I211 NICs passed through to it. Reverting to 5.3.13-3 fixed it, but now my Promox install is telling me that I have a broken install because pve-kernel-5.3 depends on pve-kernel-5.3.18-1-pve.

 

[sleel]

Hello brother, can you tell me how to restore the kernel

Had this same issue last night upon upgrading the kernel to 5.3.18-1 from 5.3.13-3. Have an Untangle VM with two I211 NICs passed through to it. Reverting to 5.3.13-3 fixed it, but now my Promox install is telling me that I have a broken install because pve-kernel-5.3 depends on pve-kernel-5.3.18-1-pve.

Hello brother, can you tell me how to restore the kernel ?

I also encountered the same problem

 

[mstinsky]

I switched my grub config to boot from the old kernel.
In my setup the old Kernel could be booted from grub menu 1 submenu 2 or something like that so i changed "GRUB_DEFAULT" in /etc/default/grub to "1>2".

 

[lingguchong]

The PCI pass-through FC card in my two nodes is not working properly now, and it will not work properly after restoring to 5.3.13-1. Is there any configuration that needs to be adjusted?

 

[Zok]

Hello brother, can you tell me how to restore the kernel ?

I also encountered the same problem

Instead of deleting the 5.3.18-1 kernel as I did previously, I chose to run untangle apt script to fix my broken install and then use GRUB to boot into the 5.3.13-3 kernel, as 5.3.18-1 hasn't been fixed yet.

This is what I did:
grep gnulinux /boot/grub/grub.cfg

Which returns:

set default="gnulinux-5.3.13-3-pve-advanced-8569c167-51c3-4bed-9ded-e5a5f74878e2"
menuentry 'Proxmox Virtual Environment GNU/Linux' --class proxmox --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-simple-8569c167-51c3-4bed-9ded-e5a5f74878e2' {
submenu 'Advanced options for Proxmox Virtual Environment GNU/Linux' $menuentry_id_option 'gnulinux-advanced-8569c167-51c3-4bed-9ded-e5a5f74878e2' {
        menuentry 'Proxmox Virtual Environment GNU/Linux, with Linux 5.3.18-1-pve' --class proxmox --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-5.3.18-1-pve-advanced-8569c167-51c3-4bed-9ded-e5a5f74878e2' {
        menuentry 'Proxmox Virtual Environment GNU/Linux, with Linux 5.3.13-3-pve' --class proxmox --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-5.3.13-3-pve-advanced-8569c167-51c3-4bed-9ded-e5a5f74878e2' {
        menuentry 'Proxmox Virtual Environment GNU/Linux, with Linux 5.3.10-1-pve' --class proxmox --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-5.3.10-1-pve-advanced-8569c167-51c3-4bed-9ded-e5a5f74878e2' {

Use Vi/nano/whatever to edit /etc/default/grub and add the menuentry and submenu menuentry together into the GRUB_DEFAULT:
#Bypass 5.3.18-1 bug with IOMMU by using 5.3.13-3
GRUB_DEFAULT="gnulinux-advanced-8569c167-51c3-4bed-9ded-e5a5f74878e2>gnulinux-5.3.13-3-pve-advanced-8569c167-51c3-4bed-9ded-e5a5f74878e2"

Run update-grub

Reboot

 

[Zok]

For what it's worth, this is what my /var/log/messages looks like under 5.3.18-1:

Feb 15 04:01:59 pve kernel: [   56.458240] general protection fault: 0000 [#1] SMP PTI
Feb 15 04:01:59 pve kernel: [   56.458269] CPU: 1 PID: 1276 Comm: task UPID:pve:0 Tainted: P           O      5.3.18-1-pve #1
Feb 15 04:01:59 pve kernel: [   56.458293] Hardware name: Default string Default string/Default string, BIOS 5.12 04/07/2019
Feb 15 04:01:59 pve kernel: [   56.458320] RIP: 0010:remove_files.isra.1+0x24/0x70
Feb 15 04:01:59 pve kernel: [   56.458336] Code: 00 00 00 0f 1f 00 0f 1f 44 00 00 55 48 89 e5 41 55 49 89 d5 41 54 49 89 fc 53 48 85 f6 74 24 48 8b 06 48 89 f3 48 85 c0 74 19 <48> 8b 30 31 d2 4c 89 e7 48 83 c3 08 e8 db d3 ff ff 48 8b 03 48 85
Feb 15 04:01:59 pve kernel: [   56.458386] RSP: 0018:ffffab3400857c18 EFLAGS: 00010282
Feb 15 04:01:59 pve kernel: [   56.458411] RAX: 82ab0c7e24a6d6e5 RBX: ffff93695224dec0 RCX: 0000000000000000
Feb 15 04:01:59 pve kernel: [   56.458440] RDX: ffff93695c6a1488 RSI: ffff93695224dec0 RDI: ffff936952358110
Feb 15 04:01:59 pve kernel: [   56.458470] RBP: ffffab3400857c30 R08: 0000000000000000 R09: ffff936952149678
Feb 15 04:01:59 pve kernel: [   56.458521] R10: 0000000000000000 R11: 0000000000000001 R12: ffff936952358110
Feb 15 04:01:59 pve kernel: [   56.458542] R13: ffff93695c6a1488 R14: ffff93695bd2b0b0 R15: fffffffffffffff2
Feb 15 04:01:59 pve kernel: [   56.458563] FS:  00007f8ee94401c0(0000) GS:ffff93695ea80000(0000) knlGS:0000000000000000
Feb 15 04:01:59 pve kernel: [   56.458586] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Feb 15 04:01:59 pve kernel: [   56.458604] CR2: 00007f8edec64800 CR3: 000000041b7da002 CR4: 00000000003606e0
Feb 15 04:01:59 pve kernel: [   56.458624] Call Trace:
Feb 15 04:01:59 pve kernel: [   56.458638]  sysfs_remove_group+0x44/0x90
Feb 15 04:01:59 pve kernel: [   56.458653]  sysfs_remove_groups+0x2e/0x50
Feb 15 04:01:59 pve kernel: [   56.458669]  device_remove_attrs+0x3e/0x70
Feb 15 04:01:59 pve kernel: [   56.458684]  device_del+0x160/0x370
Feb 15 04:01:59 pve kernel: [   56.458698]  cdev_device_del+0x1a/0x40
Feb 15 04:01:59 pve kernel: [   56.458713]  posix_clock_unregister+0x26/0x50
Feb 15 04:01:59 pve kernel: [   56.458729]  ptp_clock_unregister+0x6f/0x80
Feb 15 04:01:59 pve kernel: [   56.458750]  igb_ptp_stop+0x26/0x50 [igb]
Feb 15 04:01:59 pve kernel: [   56.458765]  igb_remove+0x4b/0x130 [igb]
Feb 15 04:01:59 pve kernel: [   56.458780]  pci_device_remove+0x3e/0xc0
Feb 15 04:01:59 pve kernel: [   56.458794]  device_release_driver_internal+0xe0/0x1b0
Feb 15 04:01:59 pve kernel: [   56.458811]  device_driver_detach+0x14/0x20
Feb 15 04:01:59 pve kernel: [   56.458826]  unbind_store+0xf9/0x130
Feb 15 04:01:59 pve kernel: [   56.458840]  drv_attr_store+0x27/0x40
Feb 15 04:01:59 pve kernel: [   56.458853]  sysfs_kf_write+0x3b/0x40
Feb 15 04:01:59 pve kernel: [   56.458867]  kernfs_fop_write+0xda/0x1c0
Feb 15 04:01:59 pve kernel: [   56.458882]  __vfs_write+0x1b/0x40
Feb 15 04:01:59 pve kernel: [   56.458896]  vfs_write+0xab/0x1b0
Feb 15 04:01:59 pve kernel: [   56.458909]  ksys_write+0x61/0xe0
Feb 15 04:01:59 pve kernel: [   56.458922]  __x64_sys_write+0x1a/0x20
Feb 15 04:01:59 pve kernel: [   56.458936]  do_syscall_64+0x5a/0x130
Feb 15 04:01:59 pve kernel: [   56.458951]  entry_SYSCALL_64_after_hwframe+0x44/0xa9
Feb 15 04:01:59 pve kernel: [   56.458972] RIP: 0033:0x7f8ee964d471
Feb 15 04:01:59 pve kernel: [   56.458986] Code: 00 00 75 05 48 83 c4 58 c3 e8 0b 4d ff ff 66 2e 0f 1f 84 00 00 00 00 00 90 8b 05 da ef 00 00 85 c0 75 16 b8 01 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 57 c3 66 0f 1f 44 00 00 41 54 49 89 d4 55 48
Feb 15 04:01:59 pve kernel: [   56.459031] RSP: 002b:00007ffcff1e81d8 EFLAGS: 00000246 ORIG_RAX: 0000000000000001
Feb 15 04:01:59 pve kernel: [   56.459052] RAX: ffffffffffffffda RBX: 00005642215d8260 RCX: 00007f8ee964d471
Feb 15 04:01:59 pve kernel: [   56.459073] RDX: 000000000000000c RSI: 00005642280d83a0 RDI: 000000000000000c
Feb 15 04:01:59 pve kernel: [   56.459094] RBP: 00005642280d83a0 R08: 0000000000000000 R09: aaaaaaaaaaaaaaab
Feb 15 04:01:59 pve kernel: [   56.459114] R10: 00005642280bf568 R11: 0000000000000246 R12: 000000000000000c
Feb 15 04:01:59 pve kernel: [   56.459134] R13: 00005642215d8260 R14: 000000000000000c R15: 00005642280d7250
Feb 15 04:01:59 pve kernel: [   56.460169] Modules linked in: tcp_diag inet_diag ebtable_filter ebtables ip_set ip6table_raw iptable_raw ip6table_filter ip6_tables iptable_filter bpfilter softdog nfnetlink_log nfnetlink snd_hda_codec_hdmi intel_rapl_msr intel_rapl_common x86_pkg_temp_thermal intel_powerclamp kvm_intel kvm crct10dif_pclmul snd_soc_skl crc32_pclmul snd_soc_hdac_hda snd_hda_ext_core snd_soc_skl_ipc snd_soc_sst_ipc ghash_clmulni_intel snd_soc_sst_dsp snd_soc_acpi_intel_match snd_soc_acpi snd_soc_core snd_compress ac97_bus snd_pcm_dmaengine snd_hda_intel snd_hda_codec snd_hda_core snd_hwdep i915 joydev input_leds snd_pcm 8250_dw drm_kms_helper aesni_intel drm aes_x86_64 zfs(PO) snd_timer crypto_simd fb_sys_fops mei_me syscopyarea sysfillrect cryptd sysimgblt snd zunicode(PO) glue_helper intel_xhci_usb_role_switch mei idma64 roles soundcore virt_dma intel_pch_thermal zlua(PO) intel_cstate zavl(PO) intel_rapl_perf icp(PO) intel_wmi_thunderbolt pcspkr acpi_pad mac_hid zcommon(PO) znvpair(PO) spl(O) vfio_pci
Feb 15 04:01:59 pve kernel: [   56.460191]  vfio_virqfd irqbypass vfio_iommu_type1 vfio vhost_net vhost tap ib_iser rdma_cm iw_cm ib_cm ib_core iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi it87 hwmon_vid coretemp sunrpc ip_tables x_tables autofs4 btrfs xor zstd_compress raid6_pq dm_thin_pool dm_persistent_data dm_bio_prison dm_bufio libcrc32c hid_logitech_hidpp hid_logitech_dj hid_generic usbkbd usbmouse usbhid hid i2c_i801 igb ahci i2c_algo_bit dca intel_lpss_pci libahci intel_lpss wmi video pinctrl_sunrisepoint pinctrl_intel
Feb 15 04:01:59 pve kernel: [   56.470058] ---[ end trace 667423bab399d6b3 ]---
Feb 15 04:01:59 pve kernel: [   56.471958] RIP: 0010:remove_files.isra.1+0x24/0x70
Feb 15 04:01:59 pve kernel: [   56.473433] Code: 00 00 00 0f 1f 00 0f 1f 44 00 00 55 48 89 e5 41 55 49 89 d5 41 54 49 89 fc 53 48 85 f6 74 24 48 8b 06 48 89 f3 48 85 c0 74 19 <48> 8b 30 31 d2 4c 89 e7 48 83 c3 08 e8 db d3 ff ff 48 8b 03 48 85
Feb 15 04:01:59 pve kernel: [   56.474925] RSP: 0018:ffffab3400857c18 EFLAGS: 00010282
Feb 15 04:01:59 pve kernel: [   56.476473] RAX: 82ab0c7e24a6d6e5 RBX: ffff93695224dec0 RCX: 0000000000000000
Feb 15 04:01:59 pve kernel: [   56.478018] RDX: ffff93695c6a1488 RSI: ffff93695224dec0 RDI: ffff936952358110
Feb 15 04:01:59 pve kernel: [   56.479561] RBP: ffffab3400857c30 R08: 0000000000000000 R09: ffff936952149678
Feb 15 04:01:59 pve kernel: [   56.481120] R10: 0000000000000000 R11: 0000000000000001 R12: ffff936952358110
Feb 15 04:01:59 pve kernel: [   56.482644] R13: ffff93695c6a1488 R14: ffff93695bd2b0b0 R15: fffffffffffffff2
Feb 15 04:01:59 pve kernel: [   56.484186] FS:  00007f8ee94401c0(0000) GS:ffff93695ea80000(0000) knlGS:0000000000000000
Feb 15 04:01:59 pve kernel: [   56.485747] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Feb 15 04:01:59 pve kernel: [   56.487286] CR2: 00007f8edec64800 CR3: 000000041b7da002 CR4: 00000000003606e0

 

[kisfc]

same question

Is there an official restoration plan?

 

[Zok]

Looks like 5.3.18-2 has fixed this issue.

 

[mstinsky]

I can confirm 5.3.18-2 fixed it for me aswell.

 

[frotella]

Seems the problem is present on 5.3.18-3 (efi). Any feedback for this bug/version/configuration?

 

[C.G.B. Spender]

Seems the problem is present on 5.3.18-3 (efi). Any feedback for this bug/version/configuration?

Is this Windows VM ? I am currently running 5.3.18-3 and on it 1 freebsd vm with 3 pci passthroughs and 1 linux vm with 1 pci passthrough and those work, can't try Windows atm though.

 

[frotella]

It's an openwrt vm (so linux based). If you give me 15 min i'll try to passthrough to a freebsd and windows vms

PS. Can you give me and hint on how to boot on a previous kernel if using efi? Maybe adding (and efi refresh) the kernel override (that Zok provide) to cmdline?

 

[C.G.B. Spender]

Oh I missed the efi bit, I boot in normal mode so not sure whether that affects it, but I would presume the grub entries are the same, if you go to the advanced options or w/e it's called on the grub screen (the 2nd option) it should list few previous kernels to boot into.

 

[frotella]

Hard times with backup yesterday. Made some test this morning, I confirm the problem on 5.3.18-3 (efi)
5.3.18-2 Everything works as aexpected (freebsd/linux/windows)
5.3.18-3 Nothing works

Here's a trace

May 12 06:12:30 pv0-it kernel: [  125.937690] invalid opcode: 0000 [#1] SMP PTI
May 12 06:12:30 pv0-it kernel: [  125.937700] CPU: 2 PID: 3618 Comm: task UPID:pv0-i Tainted: P           O      5.3.18-3-pve #1
May 12 06:12:30 pv0-it kernel: [  125.937717] Hardware name: IBM IBM xSeries High Volume Towers x3100 M4  -[2582K1G]-/00D8867, BIOS -[JQE164AUS-1.07]- 12/09/2013
May 12 06:12:30 pv0-it kernel: [  125.937742] RIP: 0010:free_msi_irqs+0x17b/0x1b0
May 12 06:12:30 pv0-it kernel: [  125.937752] Code: 84 e1 fe ff ff 45 31 f6 eb 11 41 83 c6 01 44 39 73 14 0f 86 ce fe ff ff 8b 7b 10 44 01 f7 e8 6c 1f b8 ff 48 83 78 70 00 74 e0 <0f> 0b 49 8d b5 b0 00 00 00 e8 07 da b8 ff e9 cf fe ff ff 48 8b 78
May 12 06:12:30 pv0-it kernel: [  125.937787] RSP: 0018:ffffb6e915b5bcf8 EFLAGS: 00010286
May 12 06:12:30 pv0-it kernel: [  125.937798] RAX: ffff937df98d8400 RBX: ffff937e0a765d80 RCX: 0000000000000000
May 12 06:12:30 pv0-it kernel: [  125.937812] RDX: 0000000000000000 RSI: 0000000000000024 RDI: ffffffffa5466940
May 12 06:12:30 pv0-it kernel: [  125.937826] RBP: ffffb6e915b5bd28 R08: ffff937e1c001ff0 R09: ffff937e1c002138
May 12 06:12:30 pv0-it kernel: [  125.937840] R10: 0000000000000000 R11: ffffffffa5466948 R12: ffff937e1b68f2c0
May 12 06:12:30 pv0-it kernel: [  125.937854] R13: ffff937e1b68f000 R14: 0000000000000000 R15: fffffffffffffff2
May 12 06:12:30 pv0-it kernel: [  125.937869] FS:  00007ff8dadc21c0(0000) GS:ffff937e1fa80000(0000) knlGS:0000000000000000
May 12 06:12:30 pv0-it kernel: [  125.937884] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
May 12 06:12:30 pv0-it kernel: [  125.937896] CR2: 000055fcc8db0d3c CR3: 000000081a1e4005 CR4: 00000000001606e0
May 12 06:12:30 pv0-it kernel: [  125.937910] Call Trace:
May 12 06:12:30 pv0-it kernel: [  125.937920]  pci_disable_msi+0xfa/0x120
May 12 06:12:30 pv0-it kernel: [  125.937935]  e1000e_reset_interrupt_capability+0x52/0x60 [e1000e]
May 12 06:12:30 pv0-it kernel: [  125.937951]  e1000_remove+0xb9/0x170 [e1000e]
May 12 06:12:30 pv0-it kernel: [  125.937962]  pci_device_remove+0x3e/0xc0
May 12 06:12:30 pv0-it kernel: [  125.937971]  device_release_driver_internal+0xe0/0x1b0
May 12 06:12:30 pv0-it kernel: [  125.937983]  device_driver_detach+0x14/0x20
May 12 06:12:30 pv0-it kernel: [  125.937993]  unbind_store+0xf9/0x130
May 12 06:12:30 pv0-it kernel: [  125.938001]  drv_attr_store+0x27/0x40
May 12 06:12:30 pv0-it kernel: [  125.938011]  sysfs_kf_write+0x3b/0x40
May 12 06:12:30 pv0-it kernel: [  125.938019]  kernfs_fop_write+0xda/0x1c0
May 12 06:12:30 pv0-it kernel: [  125.938029]  __vfs_write+0x1b/0x40
May 12 06:12:30 pv0-it kernel: [  125.938037]  vfs_write+0xab/0x1b0
May 12 06:12:30 pv0-it kernel: [  125.938045]  ksys_write+0x61/0xe0
May 12 06:12:30 pv0-it kernel: [  125.938052]  __x64_sys_write+0x1a/0x20
May 12 06:12:30 pv0-it kernel: [  125.938062]  do_syscall_64+0x5a/0x130
May 12 06:12:30 pv0-it kernel: [  125.938072]  entry_SYSCALL_64_after_hwframe+0x44/0xa9
May 12 06:12:30 pv0-it kernel: [  125.938083] RIP: 0033:0x7ff8dafcf471
May 12 06:12:30 pv0-it kernel: [  125.938092] Code: 00 00 75 05 48 83 c4 58 c3 e8 0b 4d ff ff 66 2e 0f 1f 84 00 00 00 00 00 90 8b 05 da ef 00 00 85 c0 75 16 b8 01 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 57 c3 66 0f 1f 44 00 00 41 54 49 89 d4 55 48
May 12 06:12:30 pv0-it kernel: [  125.938127] RSP: 002b:00007fff4f3f18a8 EFLAGS: 00000246 ORIG_RAX: 0000000000000001
May 12 06:12:30 pv0-it kernel: [  125.938142] RAX: ffffffffffffffda RBX: 000055e4a69eb260 RCX: 00007ff8dafcf471
May 12 06:12:30 pv0-it kernel: [  125.938156] RDX: 000000000000000c RSI: 000055e4ad84ced0 RDI: 000000000000000d
May 12 06:12:30 pv0-it kernel: [  125.938170] RBP: 000055e4ad84ced0 R08: 0000000000000000 R09: aaaaaaaaaaaaaaab
May 12 06:12:30 pv0-it kernel: [  125.938184] R10: 000055e4ad842458 R11: 0000000000000246 R12: 000000000000000c
May 12 06:12:30 pv0-it kernel: [  125.938198] R13: 000055e4a69eb260 R14: 000000000000000d R15: 000055e4ad84a980
May 12 06:12:30 pv0-it kernel: [  125.938212] Modules linked in: nfsv3 nfs_acl rpcsec_gss_krb5 auth_rpcgss nfsv4 nfs lockd grace fscache ebtable_filter ebtables ip_set ip6table_raw iptable_raw ip6table_filter ip6_tables iptable_filter bpfilter bonding softdog nfnetlink_log nfnetlink intel_rapl_msr intel_rapl_common x86_pkg_temp_thermal intel_powerclamp coretemp mgag200 drm_vram_helper ttm kvm_intel drm_kms_helper kvm drm i2c_algo_bit fb_sys_fops syscopyarea ipmi_ssif sysfillrect crct10dif_pclmul crc32_pclmul ghash_clmulni_intel sysimgblt cdc_ether aesni_intel usbnet input_leds joydev mii aes_x86_64 crypto_simd cryptd ie31200_edac glue_helper ipmi_si ipmi_devintf mac_hid pcspkr intel_cstate ipmi_msghandler intel_rapl_perf sch_fq vhost_net vhost tap ib_iser rdma_cm iw_cm ib_cm ib_core iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi vfio_pci sunrpc vfio_virqfd irqbypass vfio_iommu_type1 vfio ip_tables x_tables autofs4 zfs(PO) zunicode(PO) zlua(PO) zavl(PO) icp(PO) zcommon(PO) znvpair(PO) spl(O) btrfs xor zstd_compress
May 12 06:12:30 pv0-it kernel: [  125.938237]  raid6_pq libcrc32c wmi hid_sunplus hid_generic usbkbd usbmouse usbhid gpio_ich ahci i2c_i801 hid libahci lpc_ich e1000e
May 12 06:12:30 pv0-it kernel: [  125.941666] ---[ end trace 6d6d6578e1c43408 ]---
May 12 06:12:30 pv0-it kernel: [  125.942380] RIP: 0010:free_msi_irqs+0x17b/0x1b0
May 12 06:12:30 pv0-it kernel: [  125.943070] Code: 84 e1 fe ff ff 45 31 f6 eb 11 41 83 c6 01 44 39 73 14 0f 86 ce fe ff ff 8b 7b 10 44 01 f7 e8 6c 1f b8 ff 48 83 78 70 00 74 e0 <0f> 0b 49 8d b5 b0 00 00 00 e8 07 da b8 ff e9 cf fe ff ff 48 8b 78
May 12 06:12:30 pv0-it kernel: [  125.944529] RSP: 0018:ffffb6e915b5bcf8 EFLAGS: 00010286
May 12 06:12:30 pv0-it kernel: [  125.945261] RAX: ffff937df98d8400 RBX: ffff937e0a765d80 RCX: 0000000000000000
May 12 06:12:30 pv0-it kernel: [  125.946018] RDX: 0000000000000000 RSI: 0000000000000024 RDI: ffffffffa5466940
May 12 06:12:30 pv0-it kernel: [  125.946779] RBP: ffffb6e915b5bd28 R08: ffff937e1c001ff0 R09: ffff937e1c002138
May 12 06:12:30 pv0-it kernel: [  125.947509] R10: 0000000000000000 R11: ffffffffa5466948 R12: ffff937e1b68f2c0
May 12 06:12:30 pv0-it kernel: [  125.948240] R13: ffff937e1b68f000 R14: 0000000000000000 R15: fffffffffffffff2
May 12 06:12:30 pv0-it kernel: [  125.948960] FS:  00007ff8dadc21c0(0000) GS:ffff937e1fa80000(0000) knlGS:0000000000000000
May 12 06:12:30 pv0-it kernel: [  125.949698] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
May 12 06:12:30 pv0-it kernel: [  125.950415] CR2: 000055fcc8db0d3c CR3: 000000081a1e4005 CR4: 00000000001606e0
May 12 06:12:30 pv0-it pvedaemon[3168]: <root@pam> end task UPID:pv0-it:00000E22:00003117:5EBA3E4E:qmstart:100:root@pam: unable to read tail (got 0 bytes)

[\code]

 

[C.G.B. Spender]

Can't really offer any more advice, but I'd suggest starting a new thread as this one is marked solved as the issue is not present for the thread participants anymore.