Dual GPU Passthrough (Radeon 580 & Nvidia 710)

[mimesot]

Hi everyone!
I again started a GPU Passthrough experiment and encounter some difficulties I hope you can help me with.

I have Proxmox 5.2 running with two virtual machines. Both have a GPU passed through to it. One is running fine, the other one won't boot up.

VM-160:

• Win 10
• Radeon 580 in 01:00 passed through successfully

VM-161:

• Linux Mint 18.3
• Nvidia 710 in 07:00 (the third PCIe x16 slot on my MoBo, the second is tied to slot 01:00)

Are there any known issues with multiple gpu passthroughs?

My iommu output can be found in this thread. Essentially 01:00 and 02:00 appear to be tied together, but nothing else.

I will post additional data as soon as I can.

Thanks in advance!
mimesot

 

[dcsapak]

One is running fine, the other one won't boot up.

any error messages?

 

[mimesot]

None that I noticed.

The only difference I found was that, the hiding ecap lines appeared with device 01:00 (Radeon) and not with device 07:00 (GeForce). The enabling device 0000 -> 0003 line appeared with both GPUs.



In addition, i'd like to mention, that there is no access to the Linux VM via SSH. So it is more than just no graphics on a screen.

 

[mimesot]

for a in /sys/kernel/iommu_groups/*; do find $a -type l; done

/sys/kernel/iommu_groups/0/devices/0000:00:00.0
/sys/kernel/iommu_groups/1/devices/0000:00:01.1
/sys/kernel/iommu_groups/1/devices/0000:01:00.1
/sys/kernel/iommu_groups/1/devices/0000:00:01.0
/sys/kernel/iommu_groups/1/devices/0000:02:00.0
/sys/kernel/iommu_groups/1/devices/0000:01:00.0
/sys/kernel/iommu_groups/10/devices/0000:00:1c.2
/sys/kernel/iommu_groups/11/devices/0000:00:1c.4
/sys/kernel/iommu_groups/12/devices/0000:00:1d.0
/sys/kernel/iommu_groups/13/devices/0000:00:1f.3
/sys/kernel/iommu_groups/13/devices/0000:00:1f.4
/sys/kernel/iommu_groups/13/devices/0000:00:1f.2
/sys/kernel/iommu_groups/13/devices/0000:00:1f.0
/sys/kernel/iommu_groups/14/devices/0000:00:1f.6
/sys/kernel/iommu_groups/15/devices/0000:04:00.0
/sys/kernel/iommu_groups/16/devices/0000:05:00.0
/sys/kernel/iommu_groups/17/devices/0000:06:00.0
/sys/kernel/iommu_groups/18/devices/0000:07:00.0
/sys/kernel/iommu_groups/18/devices/0000:07:00.1
/sys/kernel/iommu_groups/19/devices/0000:08:00.0
/sys/kernel/iommu_groups/2/devices/0000:00:02.0
/sys/kernel/iommu_groups/3/devices/0000:00:08.0
/sys/kernel/iommu_groups/4/devices/0000:00:14.2
/sys/kernel/iommu_groups/4/devices/0000:00:14.0
/sys/kernel/iommu_groups/5/devices/0000:00:16.0
/sys/kernel/iommu_groups/6/devices/0000:00:17.0
/sys/kernel/iommu_groups/7/devices/0000:00:1b.0
/sys/kernel/iommu_groups/8/devices/0000:00:1b.3
/sys/kernel/iommu_groups/9/devices/0000:00:1c.0

lspci

00:00.0 Host bridge: Intel Corporation Device 5918 (rev 05)
00:01.0 PCI bridge: Intel Corporation Skylake PCIe Controller (x16) (rev 05)
00:01.1 PCI bridge: Intel Corporation Skylake PCIe Controller (x8) (rev 05)
00:02.0 VGA compatible controller: Intel Corporation Device 591d (rev 04)
00:08.0 System peripheral: Intel Corporation Skylake Gaussian Mixture Model
00:14.0 USB controller: Intel Corporation Sunrise Point-H USB 3.0 xHCI Controller (rev 31)
00:14.2 Signal processing controller: Intel Corporation Sunrise Point-H Thermal subsystem (rev 31)
00:16.0 Communication controller: Intel Corporation Sunrise Point-H CSME HECI #1 (rev 31)
00:17.0 SATA controller: Intel Corporation Sunrise Point-H SATA controller [AHCI mode] (rev 31)
00:1b.0 PCI bridge: Intel Corporation Sunrise Point-H PCI Root Port #17 (rev f1)
00:1b.3 PCI bridge: Intel Corporation Sunrise Point-H PCI Root Port #20 (rev f1)
00:1c.0 PCI bridge: Intel Corporation Sunrise Point-H PCI Express Root Port #1 (rev f1)
00:1c.2 PCI bridge: Intel Corporation Sunrise Point-H PCI Express Root Port #3 (rev f1)
00:1c.4 PCI bridge: Intel Corporation Sunrise Point-H PCI Express Root Port #5 (rev f1)
00:1d.0 PCI bridge: Intel Corporation Sunrise Point-H PCI Express Root Port #9 (rev f1)
00:1f.0 ISA bridge: Intel Corporation Sunrise Point-H LPC Controller (rev 31)
00:1f.2 Memory controller: Intel Corporation Sunrise Point-H PMC (rev 31)
00:1f.3 Audio device: Intel Corporation Sunrise Point-H HD Audio (rev 31)
00:1f.4 SMBus: Intel Corporation Sunrise Point-H SMBus (rev 31)
00:1f.6 Ethernet controller: Intel Corporation Ethernet Connection (2) I219-V (rev 31)
01:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Ellesmere [Radeon RX 470/480] (rev e7)
01:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Device aaf0
02:00.0 USB controller: Renesas Technology Corp. uPD720202 USB 3.0 Host Controller (rev 02)
04:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 07)
05:00.0 USB controller: ASMedia Technology Inc. ASM1142 USB 3.1 Host Controller
06:00.0 SATA controller: ASMedia Technology Inc. ASM1062 Serial ATA Controller (rev 02)
07:00.0 VGA compatible controller: NVIDIA Corporation GK208 [GeForce GT 710B] (rev a1)
07:00.1 Audio device: NVIDIA Corporation GK208 HDMI/DP Audio Controller (rev a1)
08:00.0 Non-Volatile memory controller: Samsung Electronics Co Ltd NVMe SSD Controller (rev 01)

160.conf

#I added machine q35
#I added hostpci 01 00 gfx card radeon 580
#I added hostpci 02 00 usb3 card
#I added hostpci 08 00 nvme ssd samsung 256GB
balloon: 2048
bootdisk: sata0
cores: 6
hostpci0: 01:00,pcie=1,x-vga=on
hostpci1: 02:00,pcie=1
hostpci2: 08:00,pcie=1
ide2: ProxmoxVM:iso/Windows10_DE_64bit_1803.iso,media=cdrom
keyboard: de
machine: q35
memory: 12288
name: zirkon-vm-win10
net0: e1000=52:E1:F1:51:7E:98,bridge=vmbr0
numa: 0
ostype: win10
sata0: ProxmoxVM:160/vm-160-disk-1.qcow2,cache=writeback,size=250G
scsihw: virtio-scsi-pci
smbios1: uuid=95e28013-96a8-44bb-a3cf-f1a7dc4d701b
sockets: 1
usb0: host=2-2,usb3=1

161.conf

#I added -- machine q35
#I added -- hostpci1 07 00,pcie=1,x-vga=on gfx card nvidia 710 kepler 208
balloon: 1024
boot: cdn
bootdisk: virtio0
cores: 2
hostpci0: 07:00,pcie=1,x-vga=on
ide2: ProxmoxVM:iso/linuxmint-18.3-cinnamon-64bit.iso,media=cdrom
keyboard: de
machine: q35
memory: 3072
name: zirkon-vm-mint
net0: virtio=46:E5:CC:53:D0:4E,bridge=vmbr0
numa: 0
ostype: l26
scsihw: virtio-scsi-pci
smbios1: uuid=fac992b4-9276-4f9e-9a4f-aad357930948
sockets: 1
usb0: host=2-1,usb3=1
virtio0: ProxmoxVM:161/vm-161-disk-1.qcow2,cache=writeback,size=60G

dmesg | grep vfio

[    7.506436] vfio-pci 0000:01:00.0: vgaarb: changed VGA decodes: olddecodes=io+mem,decodes=io+mem:owns=none
[    7.524060] vfio_pci: add [1002:67df[ffff:ffff]] class 0x000000/00000000
[    7.544103] vfio_pci: add [1002:aaf0[ffff:ffff]] class 0x000000/00000000
[    7.544651] vfio-pci 0000:07:00.0: vgaarb: changed VGA decodes: olddecodes=io+mem,decodes=io+mem:owns=none
[    7.564036] vfio_pci: add [10de:128b[ffff:ffff]] class 0x000000/00000000
[    7.584050] vfio_pci: add [10de:0e0f[ffff:ffff]] class 0x000000/00000000
[    7.739768] vfio-pci 0000:07:00.0: vgaarb: changed VGA decodes: olddecodes=io+mem,decodes=io+mem:owns=none
[    7.739772] vfio-pci 0000:01:00.0: vgaarb: changed VGA decodes: olddecodes=io+mem,decodes=io+mem:owns=none
[   36.396096] vfio-pci 0000:01:00.0: enabling device (0000 -> 0003)
[   36.396315] vfio_ecap_init: 0000:01:00.0 hiding ecap 0x19@0x270
[   36.396337] vfio_ecap_init: 0000:01:00.0 hiding ecap 0x1b@0x2d0
[   36.396355] vfio_ecap_init: 0000:01:00.0 hiding ecap 0x1e@0x370
[   37.544233] vfio_ecap_init: 0000:08:00.0 hiding ecap 0x19@0x168
[   37.544253] vfio_ecap_init: 0000:08:00.0 hiding ecap 0x1e@0x190
[   85.470357] vfio-pci 0000:07:00.0: enabling device (0000 -> 0003)

The Mainboard is a MSI C236a, which should support AMD 3x Crossfire. It has 3 PCIe slots of x16 size and these are 01:00, 02:00 and 07:00 as far as I can tell. This means to me that the 07:00 slot is intended tio be usef for a graphics device. While 01:00 and 02:00 both appear to be in IOMMU group 1, the 07:00 appears to be alone in group 18.

 

[mimesot]

I just exchanged the Nvidia 710 with a Radeon 230 in 07:00. Radeon is already blacklisted and vfio.conf was adapted with the new info of lspci -n -s 07:00

No effect.

 

[dcsapak]

can you try to boot both vms with ovmf as bios?
also what exactly happens with the second vm? task error? boots but no display output?

 

[mimesot]

can you try to boot both vms with ovmf as bios?

Thank you! I will try this in the evening. Is there a method to convert the Winows 10 VM to OVMF? Does OVMF act in some cind of legacy mode and the installed Windows will just boot up, or do I need to e.g. transform the VM disk tp GPT or something like that? (I heard of Windows being capable of this: "mbr2gpt.exe /allowFullOS /convert")

also what exactly happens with the second vm? task error? boots but no display output?

Unforunately I cannot report much:

• No Task error in the Web-Interface, shows OK
• A plausabible CPU and RAM usage
• No mention of anything suspicious in Syslog
• SSH does not become available. I logged into the Linux via SSH successfully when no gpu was passed through. So I guess it does not boot up. Berhaps it gets stuck in grub?

I need to check if a Keyboard is successfully usb-passed through. Perhaps it doesn't like that?

BTW. I encountered a graphics glitch again, when using Seabios and GPU passthrough. The Same green stripe on toop of the debian-proxmox console, which I encountered before. Another reason for tryong OVMF.

 

[mimesot]

I heard of Windows being capable of this: "mbr2gpt.exe /allowFullOS /convert"

This worked flawlessly with the Windows 10 VM.

Now this is interesting. I set up the Linux Mint VM anew. Then I passed through the GPU. Again no graphical output. But hey. I now can SSH into the blind Linux Mint VM. See ...

@zirkon-vm161-mint ~ $ lspci
00:00.0 Host bridge: Intel Corporation 82G33/G31/P35/P31 Express DRAM Controller
00:1a.0 USB controller: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #4 (rev 03)
00:1a.1 USB controller: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #5 (rev 03)
00:1a.2 USB controller: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #6 (rev 03)
00:1a.7 USB controller: Intel Corporation 82801I (ICH9 Family) USB2 EHCI Controller #2 (rev 03)
00:1b.0 Audio device: Intel Corporation 82801I (ICH9 Family) HD Audio Controller (rev 03)
00:1c.0 PCI bridge: Intel Corporation 7500/5520/5500/X58 I/O Hub PCI Express Root Port 0 (rev 02)
00:1c.1 PCI bridge: Intel Corporation 7500/5520/5500/X58 I/O Hub PCI Express Root Port 0 (rev 02)
00:1c.2 PCI bridge: Intel Corporation 7500/5520/5500/X58 I/O Hub PCI Express Root Port 0 (rev 02)
00:1c.3 PCI bridge: Intel Corporation 7500/5520/5500/X58 I/O Hub PCI Express Root Port 0 (rev 02)
00:1d.0 USB controller: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #1 (rev 03)
00:1d.1 USB controller: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #2 (rev 03)
00:1d.2 USB controller: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #3 (rev 03)
00:1d.7 USB controller: Intel Corporation 82801I (ICH9 Family) USB2 EHCI Controller #1 (rev 03)
00:1e.0 PCI bridge: Intel Corporation 82801 PCI Bridge (rev 92)
00:1f.0 ISA bridge: Intel Corporation 82801IB (ICH9) LPC Interface Controller (rev 02)
00:1f.2 SATA controller: Intel Corporation 82801IR/IO/IH (ICH9R/DO/DH) 6 port SATA Controller [AHCI mode] (rev 02)
00:1f.3 SMBus: Intel Corporation 82801I (ICH9 Family) SMBus Controller (rev 02)
01:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Caicos [Radeon HD 6450/7450/8450 / R5 230 OEM]
01:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Caicos HDMI Audio [Radeon HD 6400 Series]
05:01.0 PCI bridge: Red Hat, Inc. QEMU PCI-PCI bridge
05:02.0 PCI bridge: Red Hat, Inc. QEMU PCI-PCI bridge
05:03.0 PCI bridge: Red Hat, Inc. QEMU PCI-PCI bridge
05:04.0 PCI bridge: Red Hat, Inc. QEMU PCI-PCI bridge
06:03.0 Unclassified device [00ff]: Red Hat, Inc Virtio memory balloon
06:0a.0 SCSI storage controller: Red Hat, Inc Virtio block device
06:12.0 Ethernet controller: Red Hat, Inc Virtio network device
07:1b.0 USB controller: NEC Corporation uPD720200 USB 3.0 Host Controller (rev 03)

@zirkon-vm161-mint ~ $ sudo lspci -vnnn | perl -lne 'print if /^\d+\:.+(\[\S+\:\S+\])/' | grep VGA
01:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Caicos [Radeon HD 6450/7450/8450 / R5 230 OEM] [1002:6779] (prog-if 00 [VGA controller])

@zirkon-vm161-mint ~ $ glxinfo|egrep "OpenGL vendor|OpenGL renderer"
Error: unable to open display

The Linux Mint VM recognizes the AMD/ATI Radeon 230 GPU correctly in PCIe 01:00. A place, where it sould expect a graphics device to be located. But why doesn't it utilize it? I have no output on the second monitor I attached to it's DVI port. Is there any way to determine what the problem is?

Edit: I attached a monitor to the Radeon 230s VGA port and I get something. A black screen with a flashing white underscore. A console? It is unresponsive to text input.

Edit: No, I am wrong. This is even weirder. There is a bright white underscore continuously and locally stable flashing in the upper left corner BUT there is a faint image of console text periodically shifting from left to right. So faint I did not notice it before, almost ghost-like. It is too fast by far, so I can't read it, but it is most definively console text. I tried to capture it with my smartphone camera, but it is not capable of isolating a clear frame.

And what is the 07:1b.0 USB controller: NEC ... i do not recognize that piece of hardware. I passed a single USB-Port through. Is is that one? How can I tell?

 

[mimesot]

This is a part of the ghost-like display output of the passed through GPU

 

[dcsapak]

Edit: I attached a monitor to the Radeon 230s VGA port and I get something. A black screen with a flashing white underscore. A console? It is unresponsive to text input.

Edit: No, I am wrong. This is even weirder. There is a bright white underscore continuously and locally stable flashing in the upper left corner BUT there is a faint image of console text periodically shifting from left to right. So faint I did not notice it before, almost ghost-like. It is too fast by far, so I can't read it, but it is most definively console text. I tried to capture it with my smartphone camera, but it is not capable of isolating a clear frame.

seems like the graphics driver is wrong/not working/not properly configured, you could try to install with the gpu passed through (and see if the installer detects it correctly)

And what is the 07:1b.0 USB controller: NEC ... i do not recognize that piece of hardware. I passed a single USB-Port through. Is is that one? How can I tell?

yes this is the usb3 controller emulated by qemu

 

[mimesot]

seems like the graphics driver is wrong/not working/not properly configured, you could try to install with the gpu passed through (and see if the installer detects it correctly)

It does not work with the Radeon 230 (which is admittedly very old, 2014 i believe). The Grub menu does not even show up. Perhaps it is not UEFI compatible?

I exchanged it with a GTX 950 (which I do not want to keep, because it should be my maintainance device, not my gaming PC). And I can see grub. But now another problem. None of my USB passthroughs do work any more. I have no keyboard access. Damn it. PCIe passthrough and USB passthrough don't exclude each other as far as I know. What might have happened?

My next try Is the GT 710 (which is of 2016) with OVMF enabled. Let's see.

 

[dcsapak]

Perhaps it is not UEFI compatible?

yes probably

PCIe passthrough and USB passthrough don't exclude each other as far as I know. What might have happened?

mhmm... this should work together, usb3 is correctly set (set it only if the device is a usb3 device on a usb3 controller)

 

[mimesot]

My next try Is the GT 710 (which is of 2016) with OVMF enabled.

There is a GRUB screen with the 710 and OVMF. Nice. But there is still the same issue with no keyboard being accessible.

mhmm... this should work together, usb3 is correctly set (set it only if the device is a usb3 device on a usb3 controller)

I am using the usb ports 1-5 (usb 3.0, generic keyboard "hama ak 220") and port 1-13 / 1-14 (usb 1.0, logitech unifying receiver, keyboard k270). The logitech unifying receiver is switched between port 1-13 (belongs to proxmox) and 1-14 (belongs to VM 161 with Linux Mint) using a KVM switch (which also switches VGA between the Intel IGP belonging to Proxmox and the Nvidia 710 belonging to the VM161)

lsusb -t
/:  Bus 06.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/2p, 10000M
/:  Bus 05.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/2p, 480M
/:  Bus 04.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/2p, 5000M
/:  Bus 03.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/2p, 480M
/:  Bus 02.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/10p, 5000M
/:  Bus 01.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/16p, 480M
    |__ Port 5: Dev 11, If 0, Class=Human Interface Device, Driver=usbhid, 1.5M
    |__ Port 5: Dev 11, If 1, Class=Human Interface Device, Driver=usbhid, 1.5M
    |__ Port 13: Dev 17, If 0, Class=Hub, Driver=hub/4p, 480M
        |__ Port 3: Dev 18, If 2, Class=Human Interface Device, Driver=usbhid, 12M
        |__ Port 3: Dev 18, If 0, Class=Human Interface Device, Driver=usbhid, 12M
        |__ Port 3: Dev 18, If 1, Class=Human Interface Device, Driver=usbhid, 12M

I have no indea why port 5 has two interfaces. It is one keyboard on one usb cable plugged into one usb3 port.

This is an excerpt of lsusb -v with only the two existing devices selected. I detached all other usb devices just to make sure only the two keyboards remain.

Bus 001 Device 011: ID 04d9:1503 Holtek Semiconductor, Inc. Keyboard
Device Descriptor:
  bLength                18
  bDescriptorType         1
  bcdUSB               1.10
  bDeviceClass            0 (Defined at Interface level)
  bDeviceSubClass         0
  bDeviceProtocol         0
  bMaxPacketSize0         8
  idVendor           0x04d9 Holtek Semiconductor, Inc.
  idProduct          0x1503 Keyboard
  bcdDevice            3.10
  iManufacturer           1
  iProduct                2 USB Keyboard
  iSerial                 0
  bNumConfigurations      1
  Configuration Descriptor:
    bLength                 9
    bDescriptorType         2
    wTotalLength           59
    bNumInterfaces          2
    bConfigurationValue     1
    iConfiguration          0
    bmAttributes         0xa0
      (Bus Powered)
      Remote Wakeup
    MaxPower              100mA
    Interface Descriptor:
      bLength                 9
      bDescriptorType         4
      bInterfaceNumber        0
      bAlternateSetting       0
      bNumEndpoints           1
      bInterfaceClass         3 Human Interface Device
      bInterfaceSubClass      1 Boot Interface Subclass
      bInterfaceProtocol      1 Keyboard
      iInterface              0
        HID Device Descriptor:
          bLength                 9
          bDescriptorType        33
          bcdHID               1.10
          bCountryCode            0 Not supported
          bNumDescriptors         1
          bDescriptorType        34 Report
          wDescriptorLength      62
         Report Descriptors:
           ** UNAVAILABLE **
      Endpoint Descriptor:
        bLength                 7
        bDescriptorType         5
        bEndpointAddress     0x81  EP 1 IN
        bmAttributes            3
          Transfer Type            Interrupt
          Synch Type               None
          Usage Type               Data
        wMaxPacketSize     0x0008  1x 8 bytes
        bInterval              10
    Interface Descriptor:
      bLength                 9
      bDescriptorType         4
      bInterfaceNumber        1
      bAlternateSetting       0
      bNumEndpoints           1
      bInterfaceClass         3 Human Interface Device
      bInterfaceSubClass      0 No Subclass
      bInterfaceProtocol      0 None
      iInterface              0
        HID Device Descriptor:
          bLength                 9
          bDescriptorType        33
          bcdHID               1.10
          bCountryCode            0 Not supported
          bNumDescriptors         1
          bDescriptorType        34 Report
          wDescriptorLength     101
         Report Descriptors:
           ** UNAVAILABLE **
      Endpoint Descriptor:
        bLength                 7
        bDescriptorType         5
        bEndpointAddress     0x82  EP 2 IN
        bmAttributes            3
          Transfer Type            Interrupt
          Synch Type               None
          Usage Type               Data
        wMaxPacketSize     0x0008  1x 8 bytes
        bInterval              10
Device Status:     0x0000
  (Bus Powered)

Bus 001 Device 018: ID 046d:c52b Logitech, Inc. Unifying Receiver
Device Descriptor:
  bLength                18
  bDescriptorType         1
  bcdUSB               2.00
  bDeviceClass            0 (Defined at Interface level)
  bDeviceSubClass         0
  bDeviceProtocol         0
  bMaxPacketSize0        32
  idVendor           0x046d Logitech, Inc.
  idProduct          0xc52b Unifying Receiver
  bcdDevice           24.01
  iManufacturer           1 Logitech
  iProduct                2 USB Receiver
  iSerial                 0
  bNumConfigurations      1
  Configuration Descriptor:
    bLength                 9
    bDescriptorType         2
    wTotalLength           84
    bNumInterfaces          3
    bConfigurationValue     1
    iConfiguration          4 RQR24.01_B0023
    bmAttributes         0xa0
      (Bus Powered)
      Remote Wakeup
    MaxPower               98mA
    Interface Descriptor:
      bLength                 9
      bDescriptorType         4
      bInterfaceNumber        0
      bAlternateSetting       0
      bNumEndpoints           1
      bInterfaceClass         3 Human Interface Device
      bInterfaceSubClass      1 Boot Interface Subclass
      bInterfaceProtocol      1 Keyboard
      iInterface              0
        HID Device Descriptor:
          bLength                 9
          bDescriptorType        33
          bcdHID               1.11
          bCountryCode            0 Not supported
          bNumDescriptors         1
          bDescriptorType        34 Report
          wDescriptorLength      59
         Report Descriptors:
           ** UNAVAILABLE **
      Endpoint Descriptor:
        bLength                 7
        bDescriptorType         5
        bEndpointAddress     0x81  EP 1 IN
        bmAttributes            3
          Transfer Type            Interrupt
          Synch Type               None
          Usage Type               Data
        wMaxPacketSize     0x0008  1x 8 bytes
        bInterval               8
    Interface Descriptor:
      bLength                 9
      bDescriptorType         4
      bInterfaceNumber        1
      bAlternateSetting       0
      bNumEndpoints           1
      bInterfaceClass         3 Human Interface Device
      bInterfaceSubClass      1 Boot Interface Subclass
      bInterfaceProtocol      2 Mouse
      iInterface              0
        HID Device Descriptor:
          bLength                 9
          bDescriptorType        33
          bcdHID               1.11
          bCountryCode            0 Not supported
          bNumDescriptors         1
          bDescriptorType        34 Report
          wDescriptorLength     148
         Report Descriptors:
           ** UNAVAILABLE **
      Endpoint Descriptor:
        bLength                 7
        bDescriptorType         5
        bEndpointAddress     0x82  EP 2 IN
        bmAttributes            3
          Transfer Type            Interrupt
          Synch Type               None
          Usage Type               Data
        wMaxPacketSize     0x0008  1x 8 bytes
        bInterval               2
    Interface Descriptor:
      bLength                 9
      bDescriptorType         4
      bInterfaceNumber        2
      bAlternateSetting       0
      bNumEndpoints           1
      bInterfaceClass         3 Human Interface Device
      bInterfaceSubClass      0 No Subclass
      bInterfaceProtocol      0 None
      iInterface              0
        HID Device Descriptor:
          bLength                 9
          bDescriptorType        33
          bcdHID               1.11
          bCountryCode            0 Not supported
          bNumDescriptors         1
          bDescriptorType        34 Report
          wDescriptorLength      98
         Report Descriptors:
           ** UNAVAILABLE **
      Endpoint Descriptor:
        bLength                 7
        bDescriptorType         5
        bEndpointAddress     0x83  EP 3 IN
        bmAttributes            3
          Transfer Type            Interrupt
          Synch Type               None
          Usage Type               Data
        wMaxPacketSize     0x0020  1x 32 bytes
        bInterval               2
Device Status:     0x0000
  (Bus Powered)

Finally, just to make sure here is the current <vmid.conf> 161.conf

balloon: 1024
bios: ovmf
boot: dcn
bootdisk: virtio0
cores: 2
efidisk0: ProxmoxVM:161/vm-161-disk-2.qcow2,size=128K
hostpci0: 07:00,pcie=1,x-vga=on
ide2: ProxmoxVM:iso/linuxmint-18.3-cinnamon-64bit.iso,media=cdrom
keyboard: de
machine: q35
memory: 3072
name: zirkon-vm161-mint
net0: virtio=46:E5:CC:53:D0:4E,bridge=vmbr0
numa: 0
ostype: l26
scsihw: virtio-scsi-pci
smbios1: uuid=fac992b4-9276-4f9e-9a4f-aad357930948
sockets: 1
usb0: host=1-5,usb3=1
usb1: host=1-14.3
virtio0: ProxmoxVM:161/vm-161-disk-1.qcow2,cache=writeback,size=60G

Edit 1: I exchanged the installation medium to Ubuntu 18.4, which has a timeout in GRUB. I still can't use the Keyboard in grub, but at least Ubuntu starts. And iside Ubuntu the mouse actually works. But it appears as if the generic driver can't deal with the GT710. It is flickering and parts of the screen get distorted or remain black and after some time it crashes.

Edit 2: I changed the USB3 card from pcie location 02:00 to 04:00 in order to let it have its own iommu group and passed that one to the Linux Mint VM. The USB3 card is a Renesas one and was passed to the Windows 10 VM (which is down right now, so no risk of any conflicts there) before and tested successfully. The GRUB of the Linux Mint installation medium does not recognize any keyboards. I used those keyboards with the Linux Mint GRUB and installer before so that cannot be the reason. This is so strange. I will remove the passthrough of the GPU from the Mint VM again and see if the USB keyboard becomes available again.

Edit 3: No, it does not. GRUB viewed in Web Interface and keyboard physically attached to the PCIe USB 3 card does not work.

 

[dcsapak]

maybe the linux mint iso does not have usb drivers included ?

 

[mimesot]

Yeah ;-)

But seriously, I tried it with Ubuntu 18.4, Mint 18.3 and Debian 9.1

And USB passthrough works with th Windosw 10 VM, right now, which is the most scary thing of all.

If I use a previously installed Linux Mint no graphics appear, but the screen is active and blank. I can SSH into the machine and get following result if i fo lspci -vnnn. Is my Graphics card detected correctly? i am looking at the memory size description which I find suspicious.

01:00.0 VGA compatible controller [0300]: NVIDIA Corporation Device [10de:128b] (rev a1) (prog-if 00 [VGA controller])
        Subsystem: ZOTAC International (MCO) Ltd. Device [19da:5360]
        Physical Slot: 0
        Flags: fast devsel, IRQ 16
        Memory at 90000000 (32-bit, non-prefetchable) [size=16M]
        Memory at 800000000 (64-bit, prefetchable) [size=128M]
        Memory at 808000000 (64-bit, prefetchable) [size=32M]
        I/O ports at d000 [size=128]
        Expansion ROM at 91080000 [disabled] [size=512K]
        Capabilities: [60] Power Management version 3
        Capabilities: [68] MSI: Enable- Count=1/1 Maskable- 64bit+
        Capabilities: [78] Express Legacy Endpoint, MSI 00
        Capabilities: [100] Virtual Channel
        Capabilities: [128] Power Budgeting <?>
        Capabilities: [600] Vendor Specific Information: ID=0001 Rev=1 Len=024 <?>
        Kernel modules: nvidiafb, nouveau, nvidia_304

01:00.1 Audio device [0403]: NVIDIA Corporation GK208 HDMI/DP Audio Controller [10de:0e0f] (rev a1)
        Subsystem: ZOTAC International (MCO) Ltd. GK208 HDMI/DP Audio Controller [19da:5360]
        Physical Slot: 0
        Flags: bus master, fast devsel, latency 0, IRQ 17
        Memory at 91000000 (32-bit, non-prefetchable) [size=16K]
        Capabilities: [60] Power Management version 3
        Capabilities: [68] MSI: Enable- Count=1/1 Maskable- 64bit+
        Capabilities: [78] Express Endpoint, MSI 00
        Kernel driver in use: snd_hda_intel
        Kernel modules: snd_hda_intel

lsmod |grep nvidia delivers no output. Well, this cannot work.

 

[mimesot]

Okay... Ich habe jetzt einen teilweisen Fix...

Wenn ich das Gerät auf USB3 setze, funktioniert es...

PS : Allerdings funktioniert das dann auch nur im SeaBIOS Mode.... Bei OVMF tut sich nix bei USB Passthrough.... Egal was ich mache. Und bei OVMF scheint dies mit reset nicht im syslog zu stehen....

Is this a known OVMF issue and more important a solution?

I did a complete reinstall of proxmox, just to make sure.

Result: The moment I turn on OVMF an GPU Passthrough (whatever graphics card or operating system is used) USB ceases to function. If I use SeaBios and GPU passthrough on the first PCIe slot I have USB support but weird graphics glithes on the host system and it does not work on any PCIe slot except for the first one.

Note: adding device id to vfio.conf?
https://lime-technology.com/forums/...eset-mechanism/?do=findComment&comment=514507

 

[mimesot]

Hi there!
Can anyone enlighten me on how to diagnose the problem? How do you attatch a keyboard to a VM with GPU passthrough and OVMF?

It appears to me that there are two different problems regarding my OVMF-machine.

• USB-passthrough does not work at all (works with SeaBios)
  
• PCIe-Passthrough of a USB-3.0-controller card fails, because the driver xhci_hcd is gererally included in the kernel while vfio-pci is loaded as a module. Further xhci_hcd is loaded before vfio.conf and grabs the USB-controller. As xhci_hcd is not a loaded module, I can't add softdep xhci_hcd pre: vfio-pci to a corresponding .conf file.

Thanks in advance.

 

[dcsapak]

• USB-passthrough does not work at all (works with SeaBios)

this works here without problems (i have a vm with gpu passthrough and usb passthrough with q35 and ovmf running here on my workstation, no problems)

PCIe-Passthrough of a USB-3.0-controller card fails, because the driver xhci_hcd is gererally included in the kernel while vfio-pci is loaded as a module. Further xhci_hcd is loaded before vfio.conf and grabs the USB-controller. As xhci_hcd is not a loaded module, I can't add softdep xhci_hcd pre: vfio-pci to a corresponding .conf file.

this depends highly on the hardware, namely if the usb controller can be reset, then it does not matter if the card is bound to the vfio-pci driver or not (i have such a setup at home)

 

[mimesot]

this works here without problems (i have a vm with gpu passthrough and usb passthrough with q35 and ovmf running here on my workstation, no problems)

Hmmm. So is there any diagnosis data I can hand over to you and the forum users in order to determine what is the crucial difference between my System and your workstation.

this depends highly on the hardware, namely if the usb controller can be reset, then it does not matter if the card is bound to the vfio-pci driver or not (i have such a setup at home)

I have a really messy workaround right now, really messy such that I would not dare qualifying it as a solution, but, what I did was setting up a VM in SeaBios mode, then switching it from BIOS to EFI using "mbr2gpt.exe /allowFullOS /convert" (didn't try with linux guest yet) and restarting the machine using ovmf. Is there a situation, that we have a booting stage, where the passthrough might fail, because the rebinding of the devices can not yet be performed, but as the mein operating system boots up the rebinding is forced?

Thanks a lot for your support.