[SOLVED] GPU passthrough working for 2 GPU but not for 1

[jf2021]

Hi !

I'm trying to set up a GPU passtrough to a Win10 VM. I've followed all instructions and procedures, and it's partially working, but I've got a weird case and no more clues...

The host has 2 video cards :

1. A GTX1660 on PCIe slot 1
2. A GT640 (Asus GT640-2GD3) on PCIe slot2

When I add the GTX1660 to the VM, it works, and I get video on the screen connected to it

When I add both the GTX1660 and the GT640 to the VM it works only if the GTX1660 is declared as hostpci0 and GT640 as hostpci1
In this case, I've got dual screen working, one from each card.

If I add the GT640 as hostpci0 and GTX1660 as hostpci1, in this case, I get a signal only from the GTX1660 and the GTA640 has an error 43 in the device manager.

If I add only the GT640 to the VM, I get no signal, and using RDP, I can see an error 43 in the device manager for the GPU.

I've tried several ROM file for my GTA640 :

• original,
• from techpowerup with EFI,
• patched myself with GOPupdater.

The weird thing is even with the "EFI" vbios rom file, if I launch GPU-Z from within the VM, it says the GT640 has not EFI...

What could I do to have this GT640 working alone ?

Any hint or help appreciated !

Here is my config

Host : Proxmox VE 7.0

# /etc/default/grub

...
GRUB_CMDLINE_LINUX_DEFAULT="quiet amd_iommu=on iommu=pt pcie_acs_override=downstream,multifunction nofb nomodeset video=vesafbff video=efifbff"
...

# lspci -v | grep -i nvidia

##lspci -v | grep -i nvidia
04:00.0 VGA compatible controller: NVIDIA Corporation GK107 [GeForce GT 640] (rev a1) (prog-if 00 [VGA controller])
Kernel modules: nvidiafb, nouveau
04:00.1 Audio device: NVIDIA Corporation GK107 HDMI Audio Controller (rev a1)
09:00.0 VGA compatible controller: NVIDIA Corporation TU116 [GeForce GTX 1660] (rev a1) (prog-if 00 [VGA controller])
Kernel modules: nvidiafb, nouveau
09:00.1 Audio device: NVIDIA Corporation TU116 High Definition Audio Controller (rev a1)
09:00.2 USB controller: NVIDIA Corporation TU116 USB 3.1 Host Controller (rev a1) (prog-if 30 [XHCI])
09:00.3 Serial bus controller [0c80]: NVIDIA Corporation TU116 USB Type-C UCSI Controller (rev a1)
Kernel modules: i2c_nvidia_gpu

# lspci -n -s 04:00 ; lspci -n -s 09:00

# lspci -n -s 04:00 ; lspci -n -s 09:00
04:00.0 0300: 10de:0fc1 (rev a1)
04:00.1 0403: 10de:0e1b (rev a1)
09:00.0 0300: 10de:2184 (rev a1)
09:00.1 0403: 10de:1aeb (rev a1)
09:00.2 0c03: 10de:1aec (rev a1)
09:00.3 0c80: 10de:1aed (rev a1)

# /etc/modprobe.d/vfio.conf

# /etc/modprobe.d/vfio.conf
options vfio-pci ids=10de:2184,10de:1aeb,10de:1aec,10de:1aed,10de:0fc1,10de:0e1b disable_vga=1

## CONF WITH 2 GPU - Both WORKING

# /etc/pve/qemu-server/xxx.conf
agent: 1
args: -cpu 'host,+kvm_pv_unhalt,+kvm_pv_eoi,hv_vendor_id=NV43FIX,kvm=off'
balloon: 4096
bios: ovmf
boot: order=ide2;scsi0;net0
cores: 4
cpu: host,hidden=1,flags=+pcid
efidisk0: local-lvm:vm-101-disk-0,size=4M
hostpci0: 0000:09:00,pcie=1,romfile=nvidia1660-tu116.rom
hostpci1: 0000:04:00,pcie=1,romfile=AsusGT640_patched.rom
machine: pc-q35-3.1
memory: 8192
name: Windows10ProGT640
net0: virtio=12:72:27:A1:75:90,bridge=vmbr0,firewall=1
numa: 0
ostype: win10
scsi0: local-lvm:vm-101-disk-1,size=64G
scsihw: virtio-scsi-single
smbios1: uuid=121a8b1a-4847-474b-ac6b-47bd77253cc0
sockets: 1
usb0: host=046d:c31d,usb3=1
usb1: host=045e:00f0,usb3=1
vga: none
vmgenid: d63d0423-415f-4844-b094-989a3f0fe4bd

## CONF WITH 2 GPU - ONLY GTX1660 working

# /etc/pve/qemu-server/xxx.conf
agent: 1
args: -cpu 'host,+kvm_pv_unhalt,+kvm_pv_eoi,hv_vendor_id=NV43FIX,kvm=off'
balloon: 4096
bios: ovmf
boot: order=ide2;scsi0;net0
cores: 4
cpu: host,hidden=1,flags=+pcid
efidisk0: local-lvm:vm-101-disk-0,size=4M
hostpci0: 0000:04:00,pcie=1,romfile=AsusGT640_patched.rom
hostpci1: 0000:09:00,pcie=1,romfile=nvidia1660-tu116.rom
machine: pc-q35-3.1
memory: 8192
name: Windows10ProGT640
net0: virtio=12:72:27:A1:75:90,bridge=vmbr0,firewall=1
numa: 0
ostype: win10
scsi0: local-lvm:vm-101-disk-1,size=64G
scsihw: virtio-scsi-single
smbios1: uuid=121a8b1a-4847-474b-ac6b-47bd77253cc0
sockets: 1
usb0: host=046d:c31d,usb3=1
usb1: host=045e:00f0,usb3=1
vga: none
vmgenid: d63d0423-415f-4844-b094-989a3f0fe4bd

## CONF WITH ONLY GT640 - Not working

# /etc/pve/qemu-server/xxx.conf
agent: 1
args: -cpu 'host,+kvm_pv_unhalt,+kvm_pv_eoi,hv_vendor_id=NV43FIX,kvm=off'
balloon: 4096
bios: ovmf
boot: order=ide2;scsi0;net0
cores: 4
cpu: host,hidden=1,flags=+pcid
efidisk0: local-lvm:vm-101-disk-0,size=4M
hostpci0: 0000:04:00,pcie=1,romfile=AsusGT640_patched.rom
machine: pc-q35-3.1
memory: 8192
name: Windows10ProGT640
net0: virtio=12:72:27:A1:75:90,bridge=vmbr0,firewall=1
numa: 0
ostype: win10
scsi0: local-lvm:vm-101-disk-1,size=64G
scsihw: virtio-scsi-single
smbios1: uuid=121a8b1a-4847-474b-ac6b-47bd77253cc0
sockets: 1
usb0: host=046d:c31d,usb3=1
usb1: host=045e:00f0,usb3=1
vga: none
vmgenid: d63d0423-415f-4844-b094-989a3f0fe4bd

(Edit : marked as solved)

 

[Asano]

I had similar issues with my passthrough setup (but I only used Proxmox 5/6 as you may know from my other thread ). In the end I solved it with the correct ROM file (or at least I'm very sure that this was what actually did it). However getting the ROM file was quite tricky and files from techpowerup and created with tools like GOPupdater did also not work for me. The ROM file that worked was extracted from the host following exactly this procedure (though I'm not sure if you meant that when you said you tried the "original" ROM file as well):

# Ensure the GPU you want to extract the BIOS from is not the primary
# Boot up Windows VM with the GPU passed through and graphic driver installed and working
# Shut down the Windows VM
echo "0000:0b:00.0" > /sys/bus/pci/drivers/vfio-pci/unbind
cd /sys/bus/pci/devices/0000:0b:00.0/
echo 1 > rom
cat rom > /usr/share/kvm/my-gpu-rom-extracted.rom
echo 0 > rom
echo "0000:0b:00.0" > /sys/bus/pci/drivers/vfio-pci/bind

At least that is what I have documented for myself from back than. Booting up the GPU while it is not the primary one at that moment (in this state it should not be required to use a romfile) with a working windows VM and shutting it down before the extraction was especially important and obviously replace 0000:0b:00.0 with the address of your gpu. In case you haven't tried that yet maybe it's worth a shot.

 

[jf2021]

That sounds promising, thanks for the hint. I can’t give it a try right now but I will this weekend !

 

[jf2021]

@Asano, thanks a lot for your tips ! It worked !
For all my previous attempts, I had extracted my "original" GPU rom with GPU-Z, and it seems that it wasn't working properly...
I used your method to extract the rom, then I updated it with GOPUpdate and I was finally able to use my old GT640 alone on a VM (and my other GPU in another VM). Great !

 

[eminempt]

Well for me it was a different thing that was preventing me from successfully having the HDMI output for the VM.


Since i have a CPU w/ no Graphical Processing Unit integrated, (therefore, i only have 1no. GPU), ProxMox Hypervisor was splitting the resources of my GPU for him, and for the VM.

even though i tried to force that not to happen pretty hard, it turns out, something was also missing, that comes as a suggestion on the Troubleshooting section for some problems that can occurr,(that weren't really occurring but somehow i could relate.

on the /etc/default/grub i should also add 'video=efifbff'


it started working as it should right away.

my problem was i was having "Error 43".. i have change the drivers even though they were recent and there has been some changes at Nvidia whereas they now allow GPU passtrough to VMs. no luck with that.


all the other configurations for GPU Passtrough on Proxmox i've done it.

this was the missing point.

I'm pasting this here in case anyone is desperate like i was

Cheers

Marco

 

[eminempt]

by the way, pasting it here aswell, @Asano hope you dont mind.

For me this method worked on a TechPowerUp downloaded .rom file:

https://www.youtube.com/watch?v=1IP-h9IKof0

I find it easy.

Cheers