Kernel 6.8.12-1-pve breaks USB controller passthrough in a weird way

[leesteken]

I use PCI passthrough of a USB controller [1b21:3241)] to a VM (with a GPU too), including early binding to vfio-pci and a softdep to make sure vfio-pci is loaded before xhci_pci. This has worked for a long time (since before PVE 8.0, currently on an up to date 8.2 no-subscription) up to and including kernel 6.8.8-4-pve.
The new kernel of today with version 6.8.12-1-pve breaks it in a weird way where the keyboard will not work while the mouse is connected and the mouse will not work while the keyboard is connected.

05:00.0 USB controller [0c03]: ASMedia Technology Inc. ASM3241 USB 3.2 Gen 2 Host Controller [1b21:3241]
    Subsystem: Gigabyte Technology Co., Ltd ASM3241 USB 3.2 Gen 2 Host Controller [1458:5007]
    Kernel driver in use: vfio-pci
    Kernel modules: xhci_pci

options vfio_pci ids=1b21:3241 disable_vga=1
softdep xhci_pci pre: vfio-pci

I PCI passthrough another USB controller [1022:149c] to another VM (with another GPU too) and that still works fine. I don't use early binding to vfio-pci for this one and the device resets properly. I use the same keyboard and mouse using a USB-switch (KVM without V so to speak).

10:00.3 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller [1022:149c]
    Subsystem: Gigabyte Technology Co., Ltd Matisse USB 3.0 Host Controller [1458:5007]
    Kernel driver in use: vfio-pci
    Kernel modules: xhci_pci

It's unexpected that the passthrough is influenced by the Proxmox kernel version, when I took all the steps to prevent the Proxmox kernel from touching the device before the VM. Otherwise, I would not be surprised that passthrough sometimes breaks and new work-around are sometimes needed. I can work-around it by pinning kernel 6.8.8-4-pve. And I can do some more testing, like doing the opposite of preventing Proxmox from touching the device, for example.

Has anyone any idea why this might be happening? Did Linux or Ubuntu change something that early binding and softdep on xhci_pci no longer works in Proxmox?

 

[leesteken]

I tested the USB controller (without passthrough) on the host and it works. I also tested not-early binding and that also (unsurprisingly) fails on 6.8.12-1-pve but works on 6.8.8.4-pve (as the device resets properly).
Maybe it's an issue introduced by vfio-pci, since early-binding to vfio-pci (which should be device agnostic) no longer works and nothing else touches the device.

Comparing the journalctl -b 0 from both kernel versions only shows a difference in memory layout (which I would not expect but maybe this is common?):

Spoiler: 6.8.12-1-pve

e820: update [mem 0xb47c9018-0xb47d7e57] usable ==> usable
e820: update [mem 0xb47c9018-0xb47d7e57] usable ==> usable
e820: update [mem 0xb0e3f018-0xb0e4e057] usable ==> usable
e820: update [mem 0xb0e3f018-0xb0e4e057] usable ==> usable
e820: update [mem 0xb0e20018-0xb0e3e257] usable ==> usable
e820: update [mem 0xb0e20018-0xb0e3e257] usable ==> usable
e820: update [mem 0xb0e0f018-0xb0e1f057] usable ==> usable
e820: update [mem 0xb0e0f018-0xb0e1f057] usable ==> usable

software IO TLB: mapped [mem 0x00000000ace0f000-0x00000000b0e0f000] (64MB)

Spoiler: 6.8.8-4-pve

e820: update [mem 0xb47c9018-0xb47d7e57] usable ==> usable
e820: update [mem 0xb47c9018-0xb47d7e57] usable ==> usable
e820: update [mem 0xb0d25018-0xb0d34057] usable ==> usable
e820: update [mem 0xb0d25018-0xb0d34057] usable ==> usable
e820: update [mem 0xb0d06018-0xb0d24257] usable ==> usable
e820: update [mem 0xb0d06018-0xb0d24257] usable ==> usable
e820: update [mem 0xb0cf5018-0xb0d05057] usable ==> usable
e820: update [mem 0xb0cf5018-0xb0d05057] usable ==> usable

software IO TLB: mapped [mem 0x00000000accf5000-0x00000000b0cf5000] (64MB)

This small change in memory addresses does appear in several parts of the logs.

The new kernel identifies the USB controller as:

xhci_hcd 0000:05:00.0: xHCI Host Controller
xhci_hcd 0000:05:00.0: new USB bus registered, assigned bus number 1
xhci_hcd 0000:05:00.0: hcc params 0x0200ef81 hci version 0x110 quirks 0x0000000000000010
xhci_hcd 0000:05:00.0: xHCI Host Controller
xhci_hcd 0000:05:00.0: new USB bus registered, assigned bus number 2
xhci_hcd 0000:05:00.0: Host supports USB 3.2 Enhanced SuperSpeed

The logs inside the VM are less clear (because it's a Linux Mint with graphical desktop) and also only appear to have a memory offset difference. Both keyboard and mouse appear to be detected correctly inside the VM. Either the keypresses are not comming through or the mouse does not move or click. Maybe it's an interrupt or MSI issue with VFIO passthrough?

I fear that my issue is way too specific and the hardware not common enough for anyone to comment on. I guess the next kernel version will point out whether this was a generic bug or whether this is the new normal and this USB controller can unfortunately no longer be used for passthrough.