[SOLVED] kernel 5.15.83-1 breaks vm with gpu pcie passthrough

[pengu1n]

Hi. I'm in real trouble. My broken VM is our house main computer. A similar situation to https://forum.proxmox.com/threads/kernel-5-15-64-breaks-my-vms.117988/
I updated to kernel 5.15.83-1 and my VM with a pcie gpu passed through fails to start. I only noticed when my wife notified me. There was some odd behaviour. The only recent change to suspect was what I had done on proxmox. I had been trying to move the UEFI disk from one storage to another. I had to deal with the problems from that and whist working on Proxmox, I had installed the suggested upgrade in kernel.
From the changes to the VM, it took me a long while to realize the problem wasn't the change anymore, that problem was solved, leaving me with the other change, the kernel one. So here we are. I seek assistance please, I can't get the VM to start anymore and had to take day off work today to fix it and I'm half way through with no progress.
pve version 7.3-4
VM fails to start with kernels 5.15.83-1 , 5.15.74-1 .
VM is Ubuntu 22.04 with an AMD GPU passed through. I had nothing included or blacklisted in modules, and no kernel parameters in grub. GPU passthrough was working "out of the box". Just added it via the proxmox User Interface.
When on last two kernels, when the VM is started automatisch at boot, it remains off on the UI. Digging in logs, I can see the following errors:
On proxmox UI attempting to start manually:
stopped: unable to read tail (got 0 bytes)
And on dmesg:

Spoiler: GPU error

[ 226.723972] VFIO - User Level meta-driver version: 0.3
[ 226.745451] sdd: sdd1
[ 226.761592] sdc: sdc1
[ 226.933645] amdgpu 0000:08:00.0: amdgpu: amdgpu: finishing device.
[ 227.045695] Console: switching to colour dummy device 80x25
[ 231.048632] amdgpu: cp queue pipe 4 queue 0 preemption failed
[ 231.050698] amdgpu 0000:08:00.0: amdgpu: Fail to disable thermal alert!
[ 231.050803] amdgpu 0000:08:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0012 address=0xfedab0a0 flags=0x0020]
[ 231.050811] amdgpu 0000:08:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0012 address=0xfedab0c0 flags=0x0020]
[ 231.050815] amdgpu 0000:08:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0012 address=0xfedab0e0 flags=0x0020]
[ 231.050822] amdgpu 0000:08:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0012 address=0xfedab100 flags=0x0020]
[ 231.050825] amdgpu 0000:08:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0012 address=0xfedab120 flags=0x0020]
[ 231.050828] amdgpu 0000:08:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0012 address=0xfedab140 flags=0x0020]
[ 231.050831] amdgpu 0000:08:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0012 address=0xfedab160 flags=0x0020]
[ 231.050837] amdgpu 0000:08:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0012 address=0xfedab180 flags=0x0020]
[ 231.051016] BUG: kernel NULL pointer dereference, address: 000000000000013c
[ 231.051019] #PF: supervisor read access in kernel mode
[ 231.051020] #PF: error_code(0x0000) - not-present page
[ 231.051022] PGD 0 P4D 0
[ 231.051024] Oops: 0000 [#1] SMP NOPTI
[ 231.051026] CPU: 23 PID: 2675 Comm: task UPIDve:0 Tainted: P O 5.15.83-1-pve #1
[ 231.051028] Hardware name: ASUS System Product Name/ROG STRIX B550-F GAMING, BIOS 2423 08/10/2021
[ 231.051030] RIP: 0010:amdgpu_dm_fini+0x184/0x240 [amdgpu]
[ 231.051140] Code: 01 00 48 85 ff 74 10 e8 0a f1 26 00 48 c7 83 e0 4e 01 00 00 00 00 00 4c 8b 83 48 4f 01 00 4d 85 c0 74 66 48 8b 93 40 3e 01 00 <8b> 82 3c 01 00 00 85 c0 74 42 45 31 e4 49 63 c4 4c 8d 2c 40 4b 8b
[ 231.051142] RSP: 0018:ffffb957c1ddbb50 EFLAGS: 00010286
[ 231.051143] RAX: 0000000000000000 RBX: ffff987a8d060000 RCX: 00000000810000d5
[ 231.051145] RDX: 0000000000000000 RSI: 00000000810000d5 RDI: 0000000000000000
[ 231.051146] RBP: ffffb957c1ddbb68 R08: ffff987a8ccfd960 R09: 0000000000000001
[ 231.051147] R10: ffff987a8adb8b00 R11: dead000000000122 R12: 0000000000000005
[ 231.051148] R13: ffff987a8d075518 R14: 0000000000000005 R15: 0000000000000001
[ 231.051149] FS: 00007f2664196280(0000) GS:ffff98819efc0000(0000) knlGS:0000000000000000
[ 231.051150] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 231.051151] CR2: 000000000000013c CR3: 0000000169bac000 CR4: 0000000000750ee0
[ 231.051153] PKRU: 55555554
[ 231.051154] Call Trace:
[ 231.051155] <TASK>
[ 231.051157] dm_hw_fini+0x23/0x30 [amdgpu]
[ 231.051235] amdgpu_device_fini_hw+0x3c7/0x50b [amdgpu]
[ 231.051326] amdgpu_driver_unload_kms+0x69/0x90 [amdgpu]
[ 231.051379] amdgpu_pci_remove+0x27/0x50 [amdgpu]
[ 231.051430] pci_device_remove+0x3e/0xb0
[ 231.051434] __device_release_driver+0x1ab/0x2a0
[ 231.051437] device_driver_detach+0x56/0xe0
[ 231.051439] unbind_store+0x12a/0x140
[ 231.051441] drv_attr_store+0x24/0x40
[ 231.051442] sysfs_kf_write+0x3f/0x50
[ 231.051445] kernfs_fop_write_iter+0x13f/0x1d0
[ 231.051447] new_sync_write+0x114/0x1b0
[ 231.051449] vfs_write+0x1d9/0x270
[ 231.051451] ksys_write+0x67/0xf0
[ 231.051452] __x64_sys_write+0x1a/0x20
[ 231.051454] do_syscall_64+0x5c/0xc0
[ 231.051457] ? exit_to_user_mode_prepare+0x37/0x1b0
[ 231.051460] ? syscall_exit_to_user_mode+0x27/0x50
[ 231.051462] ? __x64_sys_newfstat+0x16/0x20
[ 231.051464] ? do_syscall_64+0x69/0xc0
[ 231.051465] ? exit_to_user_mode_prepare+0x37/0x1b0
[ 231.051466] ? irqentry_exit_to_user_mode+0x9/0x20
[ 231.051468] ? irqentry_exit+0x1d/0x30
[ 231.051469] ? exc_page_fault+0x89/0x170
[ 231.051471] entry_SYSCALL_64_after_hwframe+0x61/0xcb
[ 231.051473] RIP: 0033:0x7f26643b8fb3
[ 231.051474] Code: 75 05 48 83 c4 58 c3 e8 cb 41 ff ff 66 2e 0f 1f 84 00 00 00 00 00 90 64 8b 04 25 18 00 00 00 85 c0 75 14 b8 01 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 55 c3 0f 1f 40 00 48 83 ec 28 48 89 54 24 18
[ 231.051476] RSP: 002b:00007ffc5a609d98 EFLAGS: 00000246 ORIG_RAX: 0000000000000001
[ 231.051478] RAX: ffffffffffffffda RBX: 00005603c080b650 RCX: 00007f26643b8fb3
[ 231.051479] RDX: 000000000000000c RSI: 00005603c080b650 RDI: 000000000000000e
[ 231.051480] RBP: 000000000000000c R08: 0000000000000000 R09: 00005603b871e3b0
[ 231.051481] R10: 00005603c07ec508 R11: 0000000000000246 R12: 00005603c08080c0
[ 231.051482] R13: 00005603b92fc2a0 R14: 000000000000000e R15: 00005603c08080c0
[ 231.051483] </TASK>
[ 231.051484] Modules linked in: vfio_pci vfio_pci_core vfio_virqfd vfio_iommu_type1 vfio veth ebtable_filter ebtables ip_set ip6table_raw iptable_raw ip6table_filter ip6_tables iptable_filter bpfilter nf_tables bonding tls softdog nfnetlink_log nfnetlink intel_rapl_msr intel_rapl_common amdgpu snd_hda_codec_realtek snd_hda_codec_generic edac_mce_amd nouveau ledtrig_audio iommu_v2 gpu_sched mxm_wmi snd_hda_codec_hdmi drm_ttm_helper kvm_amd ttm snd_hda_intel drm_kms_helper snd_intel_dspcfg cec kvm snd_intel_sdw_acpi snd_hda_codec rc_core i2c_algo_bit irqbypass snd_hda_core fb_sys_fops crct10dif_pclmul syscopyarea snd_hwdep ghash_clmulni_intel sysfillrect sysimgblt snd_pcm aesni_intel snd_timer crypto_simd cryptd joydev snd eeepc_wmi input_leds asus_wmi rapl soundcore ccp platform_profile efi_pstore pcspkr sparse_keymap k10temp video wmi_bmof mac_hid zfs(PO) zunicode(PO) zzstd(O) zlua(O) zavl(PO) icp(PO) zcommon(PO) znvpair(PO) spl(O) vhost_net vhost vhost_iotlb tap ib_iser rdma_cm iw_cm
[ 231.051517] ib_cm ib_core iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi drm sunrpc ip_tables x_tables autofs4 btrfs blake2b_generic xor zstd_compress raid6_pq simplefb dm_thin_pool dm_persistent_data dm_bio_prison dm_bufio libcrc32c hid_generic usbmouse usbkbd usbhid hid crc32_pclmul i2c_piix4 xhci_pci xhci_pci_renesas igc ahci xhci_hcd libahci wmi gpio_amdpt gpio_generic
[ 231.051540] CR2: 000000000000013c
[ 231.051541] ---[ end trace 7dc805a9de1c05ee ]---
[ 231.067912] RIP: 0010:amdgpu_dm_fini+0x184/0x240 [amdgpu]
[ 231.068050] Code: 01 00 48 85 ff 74 10 e8 0a f1 26 00 48 c7 83 e0 4e 01 00 00 00 00 00 4c 8b 83 48 4f 01 00 4d 85 c0 74 66 48 8b 93 40 3e 01 00 <8b> 82 3c 01 00 00 85 c0 74 42 45 31 e4 49 63 c4 4c 8d 2c 40 4b 8b
[ 231.068053] RSP: 0018:ffffb957c1ddbb50 EFLAGS: 00010286
[ 231.068055] RAX: 0000000000000000 RBX: ffff987a8d060000 RCX: 00000000810000d5
[ 231.068056] RDX: 0000000000000000 RSI: 00000000810000d5 RDI: 0000000000000000
[ 231.068058] RBP: ffffb957c1ddbb68 R08: ffff987a8ccfd960 R09: 0000000000000001
[ 231.068059] R10: ffff987a8adb8b00 R11: dead000000000122 R12: 0000000000000005
[ 231.068060] R13: ffff987a8d075518 R14: 0000000000000005 R15: 0000000000000001
[ 231.068061] FS: 00007f2664196280(0000) GS:ffff98819efc0000(0000) knlGS:0000000000000000
[ 231.068062] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 231.068063] CR2: 000000000000013c CR3: 0000000169bac000 CR4: 0000000000750ee0
[ 231.068065] PKRU: 55555554

Tried so far:
I went to use modules in case of regressions in new kernel but it kills my pve filling in the logs and I have to quickly start truncating log files before physically shutting down.
Errors :
On proxmox:
Jan 03 12:42:26 pve kernel: vfio-pci 0000:08:00.0: BAR 0: can't reserve [mem 0xc0000000-0xcfffffff 64bit pref]
On VM:
Jan 03 12:42:25 pve QEMU[6698]: kvm: vfio_region_read(0000:08:00.0:region0+0x4890e8, 4) failed: Device or resource busy

I put in /etc/modprobe.d/blacklist.conf
blacklist amdgpu
blacklist radeon
blacklist nouveau
and in /etc/modprobe.d/vfio.conf
options vfio-pci ids=1002:73ff,1002:ab28 disable_vga=1

Going this way seems to be worse as the vfio errors quickly kill the node. So I'm reversing them, trying again without it to get the actual error for clues.

 

[shanreich]

Have you tried this workaround? [1]

[1] https://forum.proxmox.com/posts/478351/

 

[pengu1n]

Hi. Not yet. Ich will do it next. I'm updating the error. Thank you.

 

[pengu1n]

I've just read the thread. I am really hoping I don't have to get into the vfio thing and be able to use a previous kernel where things were working OK out of the box.
I don't understand the required magic to make gpu passthrough. There are so many variables and old internet posts that scares me due to lack of time to get to the right permutation of options.
I'll have another scan of the thread.
Meantime, shouldn't chosing the previous kernel not have restored the situation?

 

[pengu1n]

Thank you @shanreich for the pointer, following what seemed to be the parts relevant to me, we've been able to start the VM again on the new 5.15.83-1-pve kernel with successful AMD GPU passthrough again.
For reference my changes were:
1. to add to /etc/default/grub: nomodeset video=vesafbff video=efifbff video=simplefbff to the line GRUB_CMDLINE_LINUX_DEFAULT="quiet". GRUB_CMDLINE_LINUX_DEFAULT="quiet" was my previous value.

2. leave blacklisted nvidia:
/etc/modprobe.d/blacklist.conf
#blacklist amdgpu
#blacklist radeon
#blacklist nouveau
blacklist nvidia

3. also commented out a previous vfio config's failed attempt:
/etc/modprobe.d/vfio.conf
# AMD 6600XT: 1002:73ff (rev c1), 1002:ab28
#options vfio-pci ids=1002:73ff,1002:ab28 disable_vga=1

4. create a cron job with the script /root/fix_gpu_pass.sh, which has the contents:

echo 1 > /sys/bus/pci/devices/0000\:08\:00.0/remove
echo 1 > /sys/bus/pci/rescan

Mostly for my own future reference, I am not knowledgeable how the options work together to the success.
I am using an AMD CPU, passing through an AMD GPU to an Ubuntu linux 22.04 VM.
The system also has a dedicated Nvidia basic card but I still can't get the output from pve to it. That is an old problem though. I will need to open a new post for help for that.