PCIe USB3 Card Passthrough

shoddyguard

New Member
May 10, 2020
1
0
1
37
Hello,
I'm assessing Proxmox at the moment and trying to passthrough a PCIe USB3 card but it's not working correctly.
I have blacklisted the driver and created a modprobe .conf file to tell virtio to grab the card, however the driver is still reported as xhci_hcd

I read another post who had similar issues, it was suggested to try the passthrough anyway, which does indeed work in my case too, however this results in some weird behaviour inside the guest.

While the guest can see devices plugged into the card they do not behave as expected, for example if I plug a USB stick into the card and try to copy files to the guest then I am given a "the filename you've specified is either invalid or too long" message (even though the file is called test.mp4) though if I play the video file directly from the USB stick in the guest it works.
I've also tried plugging a UPS into the guest, which results in the battery indicator popping up, but when installing the UPS software it cannot detect the UPS...

Both these devices work on bare metal with no issue, I have tried both Ubuntu, Windows Server 2019 and Windows 10 guests, all of which have similar issues (though not identical) so my suspicion is that there is some issue with the passthrough and the xhci_hcd mentioned above.

Also of interest:
I originally tried plugging the devices into the on-board USB ports of the machine and passing them through as USB devices, this also had exactly the same issue.
Which lead me to trying the USB card to rule out on-board shenanigans.
FWIW I also tried a different USB card in case I had a duff one but the fault remains.
I have managed to succesfully passthrough an nVidia GPU to another machine as a test to make sure VFIO was working, this correctly uses the vfio-pci driver and works perfectly as far as I can tell.

I have updated initramfs and rebooted in-between every change I've made to be safe.


Any help would be greatly appreciated as this will unfortunately be a blocker to us rolling out Proxmox org wide due to our reliance on our vendors UPS software :(

Result of lspci -nnk (trimmed down due to forum character limit):
Code:
03:00.0 VGA compatible controller [0300]: NVIDIA Corporation GP107GL [Quadro P400] [10de:1cb3] (rev a1)
        Subsystem: NVIDIA Corporation GP107GL [Quadro P400] [10de:11be]
        Kernel driver in use: vfio-pci
        Kernel modules: nvidiafb, nouveau
03:00.1 Audio device [0403]: NVIDIA Corporation GP107GL High Definition Audio Controller [10de:0fb9] (rev a1)
        Subsystem: NVIDIA Corporation GP107GL High Definition Audio Controller [10de:11be]
        Kernel driver in use: vfio-pci
        Kernel modules: snd_hda_intel
04:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Cedar [Radeon HD 5000/6000/7350/8350 Series] [1002:68f9]
        Subsystem: PC Partner Limited / Sapphire Technology Cedar [Radeon HD 5000/6000/7350/8350 Series] [174b:e164]
        Kernel driver in use: radeon
        Kernel modules: radeon
04:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Cedar HDMI Audio [Radeon HD 5400/6300/7300 Series] [1002:aa68]
        Subsystem: PC Partner Limited / Sapphire Technology Cedar HDMI Audio [Radeon HD 5400/6300/7300 Series] [174b:aa68]
        Kernel driver in use: snd_hda_intel
        Kernel modules: snd_hda_intel
08:00.0 USB controller [0c03]: ASMedia Technology Inc. ASM1142 USB 3.1 Host Controller [1b21:1242]
        Subsystem: ASMedia Technology Inc. ASM1142 USB 3.1 Host Controller [1b21:1242]
        Kernel driver in use: xhci_hcd
0b:00.0 USB controller [0c03]: Renesas Technology Corp. uPD720202 USB 3.0 Host Controller [1912:0015] (rev 02)
        Kernel driver in use: xhci_hcd

Content of /etc/modprobe.d/blacklist.conf:
Code:
blacklist nvidia
blacklist renesas
blacklist ASMedia

Content of /etc/modprobe.d/vfio.conf:
Code:
options vfio-pci ids=10de:1cb3,10de:0fb9 disable_vga=1
options vfio-pci ids=1912:0015,1b21:1242

qm config output:
Code:
bios: ovmf
bootdisk: sata0
cores: 2
efidisk0: local-lvm:vm-103-disk-1,size=4M
hostpci0: 08:00,pcie=1
hostpci1: 0b:00,pcie=1
ide0: local:iso/virtio-latest.iso,media=cdrom,size=685830K
ide2: local:iso/en_windows_10_business_editions_version_1909_updated_april_2020_x64_dvd_aa945e0d.iso,media=cdrom
machine: q35
memory: 2048
name: testw10
net0: e1000=52:3F:C8:17:53:AB,bridge=vmbr0,firewall=1
numa: 0
ostype: win10
sata0: local-lvm:vm-103-disk-0,size=40G
scsihw: virtio-scsi-pci
smbios1: uuid=cfa42a5e-6a18-417d-87d5-a51873bdc8fb
sockets: 1
vga: qxl
vmgenid: 938fe9d4-a2e1-4263-95ff-dd00540acb2e
 
No real help, sorry, but I also tried that a while ago and there are adapters that work and other that just don't work.
 
I'm going to test with that:

07:00.0 USB controller [0c03]: Renesas Technology Corp. uPD720201 USB 3.0 Host Controller [1912:0014] (rev 03)
Kernel driver in use: xhci_hcd
Kernel modules: xhci_pci

any result about Renesas product?

-----
it's looks like the IOMMU group of my card is share with a lots of other stuff on my PC, including my ethernet card. so that complicate.
 
Last edited:
it's looks like the IOMMU group of my card is share with a lots of other stuff on my PC, including my ethernet card. so that complicate.

That is always a problem. I have also a board with only one PCIe and one MiniPCIe and they share the same IOMMU group with a lot of onboard devices rendering the machine totally useless for PCIe-passthrough stuff :-/

Have you tried changing the PCIe slot?
 
Have you tried changing the PCIe slot?

Unfortunately I did my tests onto a Micro ATX motherboard, so having only 3 slot.

By switching to a smaller graphic card I achieve to test the middle slot, but this one give me the same IOMMU.

This proxmox server should had only been used as a supper cheap proxmox server node, So I do not have a lot of expectation on it, but since the gfx passthrough allready work fine on it, I'm preaty happy with this PC (B450 chipset + Ryzen 3700X + 64Go RAM)

I'm building a lot of new script to allow me to hotplug USB to the VM that get the GFX, I think that next week I will be able to hotplug USB.

Breaking IOMMU groups is for my next Proxmox node, this one is already in use. We always need an extra PC that we do not have ...
 
Last edited:
Switch IOMMU bios option from *Auto* to *Enabled* should create more IOMMU groups.
I would like to test that, but I can not restart my Proxmox server for now...
 
I have a working script in nodejs:

Code:
npx proxmox-hotplug

This script should support multiple identical devices, but I need to find tww of them to test it.
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!