4 GPUs Passthrough Issue

[d1xt1r]

Hello guys,

I am trying to passthrough (GPU Passthrough) 4 video cards (2xGTX1080 and 2x1080Ti) to 4 separate Windows 10 virtual machines (1 GPU per virtual machine). Because I'm accessing the server via SSH I don't need host GPU.
I'm successfully passing-through 3 of them, but for one of them (the one in the first PCIExpress slot) I'm facing the code 43 issue. I've tried different things, including kvm=off, DDU the drivers, reinstall of proxmox/virtual machines but so far I'm unable to solve the problem.

I've read that you cannot passthrough the primary GPU, but I'm not sure what exactly this means.
With two GPUs installed in PCIExpress slot 3 and 5 I'm able to passthrough both cards.
In my tests when I was trying to figure out why I'm getting code 43 I managed to passthrough (no code 43) when only one GPU was installed in the system (although today with fresh install of proxmox I'm unable to get it working with only one GPU).

Can you help me, please?

Hardware configuration:
Motherboard: ASRock X299 Taichi (with latest available BIOS)
CPU: Intel Core i9-7960X (no integrated GPU)
RAM: Corsair Vengeance LPX 32GB (4x8GB) DDR4 3000MHz
Storage: Samsung 860 EVO 500GB 2.5 Inch SATA III
GPU: 2 x Gigabyte GeForce GTX Windforce 1080, 2 x GeForce GTX 1080 Ti Founders Edition
PSU: Super Flower Leadex 1600W Titanium

Software configuration:
Proxmox 5.3-1

Windows 10 VM config file for the VM with the GPU in the first PCIExpress slot:

bios: ovmf
bootdisk: scsi0
cores: 4
cpu: host,hidden=1
efidisk0: local-lvm:vm-100-disk-1,size=128K
ide0: local:iso/virtio-win-0.1.141.iso,media=cdrom,size=309208K
ide2: local:iso/Windows.iso,media=cdrom
machine: q35
memory: 6144
name: quarkvm1
net0: virtio=22:C2:D8:7E:AF:F0,bridge=vmbr0
numa: 1
ostype: win10
scsi0: local-lvm:vm-100-disk-0,size=100G
scsihw: virtio-scsi-pci
smbios1: uuid=f4e9061b-4aa5-4add-a992-a9a55b34ea28
sockets: 1
vmgenid: 0c820384-601d-434c-b8d7-7543e86ed79f
hostpci0: 65:00,x-vga=on,pcie=1

lspci -v

65:00.0 VGA compatible controller: NVIDIA Corporation GP104 [GeForce GTX 1080] (rev a1) (prog-if 00 [VGA controller])
        Subsystem: Gigabyte Technology Co., Ltd GP104 [GeForce GTX 1080]
        Flags: fast devsel, IRQ 55, NUMA node 0
        Memory at d7000000 (32-bit, non-prefetchable) [size=16M]
        Memory at c0000000 (64-bit, prefetchable) [size=256M]
        Memory at d0000000 (64-bit, prefetchable) [size=32M]
        I/O ports at b000 [size=128]
        Expansion ROM at 000c0000 [disabled] [size=128K]
        Capabilities: [60] Power Management version 3
        Capabilities: [68] MSI: Enable- Count=1/1 Maskable- 64bit+
        Capabilities: [78] Express Legacy Endpoint, MSI 00
        Capabilities: [100] Virtual Channel
        Capabilities: [250] Latency Tolerance Reporting
        Capabilities: [128] Power Budgeting <?>
        Capabilities: [420] Advanced Error Reporting
        Capabilities: [600] Vendor Specific Information: ID=0001 Rev=1 Len=024 <?>
        Capabilities: [900] #19
        Kernel driver in use: vfio-pci
        Kernel modules: nvidiafb, nouveau

65:00.1 Audio device: NVIDIA Corporation GP104 High Definition Audio Controller (rev a1)
        Subsystem: Gigabyte Technology Co., Ltd GP104 High Definition Audio Controller
        Flags: fast devsel, IRQ 60, NUMA node 0
        Memory at d8080000 (32-bit, non-prefetchable) [size=16K]
        Capabilities: [60] Power Management version 3
        Capabilities: [68] MSI: Enable- Count=1/1 Maskable- 64bit+
        Capabilities: [78] Express Endpoint, MSI 00
        Capabilities: [100] Advanced Error Reporting
        Kernel driver in use: vfio-pci
        Kernel modules: snd_hda_intel

 

[dcsapak]

I've read that you cannot passthrough the primary GPU, but I'm not sure what exactly this means.

normally the mainboard/bios grabs a gpu to output an image (e.g. the boot logo), this includes some hardware initialization which touches the
gpu rom, passing such a card through then may result in weird behavior

you could try to boot the card as non-primary, dump the rom (see e.g. https://wiki.archlinux.org/index.php/PCI_passthrough_via_OVMF ) and pass the romfile then
to the device with 'romfile=...' ( see the manpage/documentation for details)

 

[d1xt1r]

Thank you, dcsapak!

I read the bios dump / pass procedure (https://pve.proxmox.com/wiki/Pci_passthrough#How_to_known_if_card_is_UEFI_.28ovmf.29_compatible)

I want to try this method, but before that may I ask you
Is passing the rom = flashing the rom, or it is temporary setting?
Can I brick GPU if passing the rom goes wrong?
I have experience with flashing bios roms on GPUs/Motherboards, but I have never done "passing" rom.

Thank you in advance!

BR,
Denis

 

[dcsapak]

no if you simply configure the rom in the vm config, nothing gets flashed

 

[d1xt1r]

no if you simply configure the rom in the vm config, nothing gets flashed

Thank you very much, dcsapak. I was able to resolve the Code 43 issue on the first PCIE slot using the dumping / passing bios file.