SOLVED ** GPU Passthrough with NVIDIA GTX 680 ti

[gen2k4k]

Hi,
I am trying to use a Nvidia Gtx 680 in a Windows 10 VM. I've been looking the Wiki multiple times, but i am stuck using the gpu Passthrough
see below the hypervisor configuration:
- Proxmox Virtual Environment 6.2-4
- Gpu gigabyte nvidia geforce gtx 680 4gb updated latest bios UEFI boot compatible
- MB Asus P9X79
- CPU Intel Core i7-3930K 3.2GHz Socket LGA

The MB doesn't have a standard VGA hence the nvidia GPU is the only graphic card in my system

This is my VM config:

bios: ovmf
boot: c
bootdisk: scsi0
cores: 4
cpu: host,hidden=1,flags=+pcid
args: -cpu 'host,+kvm_pv_unhalt,+kvm_pv_eoi,hv_vendor_id=NV43FIX,kvm=off
efidisk0: zvmpool2:vm-103-disk-1,size=1M
hostpci0: 01:00,pcie=1,x-vga=1,romfile=Gigabyte.GTX680.4096.130110.rom
ide2: local:iso/virtio-win-0.1.189.iso,media=cdrom,size=488766K
machine: q35
memory: 8196
name: win10alxpass
net0: virtio=F2:66:04:6DE:81,bridge=vmbr0,firewall=1
numa: 0
ostype: win10
scsi0: zvmpool2:vm-103-disk-0,size=80G
scsihw: virtio-scsi-pci
smbios1: uuid=f1b90ae1-4f17-436f-a902-1a1788f1bc5b
sockets: 1
vga: none
vmgenid: 2028b15b-27bd-4112-8969-b023eae9c6dd

I've done all the steps listed in the forumus and the GPU passthrough beginner guide

when starting the VGA with qm start 104 I receive the following BAR3 error:
kvm: -device vfio-pci,host=0000:01:00.0,id=hostpci0,bus=ich9-pcie-port-1,addr=0x0,romfile=/usr/share/kvm/Gigabyte.GTX680.4096.130110.rom: Failed to mmap 0000:01:00.0 BAR 3. Performance may be slow

after this error the whole system hangs and I need to power off in order to restart
When starting the VM It seems to me that for one second the fan stop running and then the hypervisor hangs.

while clearing the beginner guide steps I see that with lspci -v command I don't have any "Kernel driver in use: vfio-pci" row for the GPU

lspci -v

01:00.0 VGA compatible controller: NVIDIA Corporation GK104 [GeForce GTX 680] (rev a1) (prog-if 00 [VGA controller])
Subsystem: Gigabyte Technology Co., Ltd GK104 [GeForce GTX 680]
Flags: bus master, fast devsel, latency 0, IRQ 11
Memory at fa000000 (32-bit, non-prefetchable) [size=16M]
Memory at f0000000 (64-bit, prefetchable) [size=128M]
Memory at f8000000 (64-bit, prefetchable) [size=32M]
I/O ports at e000
Expansion ROM at 000c0000 [disabled] [size=128K]
Capabilities: [60] Power Management version 3
Capabilities: [68] MSI: Enable- Count=1/1 Maskable- 64bit+
Capabilities: [78] Express Endpoint, MSI 00
Capabilities: [b4] Vendor Specific Information: Len=14 <?>
Capabilities: [100] Virtual Channel
Capabilities: [128] Power Budgeting <?>
Capabilities: [600] Vendor Specific Information: ID=0001 Rev=1 Len=024 <?>
Capabilities: [900] #19
Kernel modules: nvidiafb, nouveau

01:00.1 Audio device: NVIDIA Corporation GK104 HDMI Audio Controller (rev a1)
Subsystem: Gigabyte Technology Co., Ltd GK104 HDMI Audio Controller
Flags: bus master, fast devsel, latency 0, IRQ 63
Memory at fb080000 (32-bit, non-prefetchable) [size=16K]
Capabilities: [60] Power Management version 3
Capabilities: [68] MSI: Enable- Count=1/1 Maskable- 64bit+
Capabilities: [78] Express Endpoint, MSI 00
Kernel driver in use: snd_hda_intel
Kernel modules: snd_hda_intel


Could this be linked to the single graphic card on my system?
of course I've enabled vfio to the gpu + audio

nano /etc/modprobe.d/vfio.conf
options vfio-pci ids=10de:1180,10de:0e0a disable_vga=1

another strange thing is that using the GPU-Z command I can see that the GPU is UEFI compatible
Iìve also contacted Gigabyte and they've confirmed that the loaded bios is uefi compatible
But if I run rom-parser script I have:

root@prmx01:/rom-parser# ./rom-parser /tmp/image.rom
Valid ROM signature found @0h, PCIR offset 190h
PCIR: type 0 (x86 PC-AT), vendor: 10de, device: 1180, class: 030000
PCIR: revision 0, vendor revision: 1
Error, ran off the end



I've tried several other steps googling + YT but always bringing to the same BAR3 error

any idea ?

many thanks in advance for your help
Alex

 

[LnxBil]

Hi and welcome. First, please use CODE-tags for CLI output, which make the output better readable.

args: -cpu 'host,+kvm_pv_unhalt,+kvm_pv_eoi,hv_vendor_id=NV43FIX,kvm=off

Why do you turn kvm off? The is the hardware acceleration?

Could this be linked to the single graphic card on my system?

Could be.

Have you tried another GPU? I also struggled in getting one older GPU to work, after switching to a newer one, it just worked out of the box, just follow the wiki guide.

What about your IOMMU groups?

 

[gen2k4k]

Hi LnxBil,
thanks for helping.

What about your IOMMU groups?

473"]
Why do you turn kvm off? The is the hardware acceleration?
[/QUOTE]

I don't have a reason, while googling I found that string that worked for someone and put that in my configuration hoping could work.
I've just removed that parameter

What about your IOMMU groups?

here the result for IOMMU groups:
find /sys/kernel/iommu_groups/ -type l

/sys/kernel/iommu_groups/55/devices/0000:ff:11.0
/sys/kernel/iommu_groups/17/devices/0000:00:1c.5
/sys/kernel/iommu_groups/45/devices/0000:ff:0f.5
/sys/kernel/iommu_groups/35/devices/0000:ff:0a.3
/sys/kernel/iommu_groups/35/devices/0000:ff:0a.1
/sys/kernel/iommu_groups/35/devices/0000:ff:0a.2
/sys/kernel/iommu_groups/35/devices/0000:ff:0a.0
/sys/kernel/iommu_groups/7/devices/0000:00:05.4
/sys/kernel/iommu_groups/25/devices/0000:08:00.0
/sys/kernel/iommu_groups/53/devices/0000:ff:10.6
/sys/kernel/iommu_groups/15/devices/0000:00:1c.2
/sys/kernel/iommu_groups/43/devices/0000:ff:0f.3
/sys/kernel/iommu_groups/33/devices/0000:ff:09.3
/sys/kernel/iommu_groups/5/devices/0000:00:05.0
/sys/kernel/iommu_groups/23/devices/0000:01:00.1
/sys/kernel/iommu_groups/51/devices/0000:ff:10.4
/sys/kernel/iommu_groups/13/devices/0000:00:1c.0
/sys/kernel/iommu_groups/41/devices/0000:ff:0f.1
/sys/kernel/iommu_groups/31/devices/0000:ff:08.4
/sys/kernel/iommu_groups/3/devices/0000:00:02.0
/sys/kernel/iommu_groups/21/devices/0000:00:1f.2
/sys/kernel/iommu_groups/21/devices/0000:00:1f.0
/sys/kernel/iommu_groups/21/devices/0000:00:1f.3
/sys/kernel/iommu_groups/11/devices/0000:00:1a.0
/sys/kernel/iommu_groups/1/devices/0000:00:01.0
/sys/kernel/iommu_groups/48/devices/0000:ff:10.1
/sys/kernel/iommu_groups/38/devices/0000:ff:0d.1
/sys/kernel/iommu_groups/38/devices/0000:ff:0d.6
/sys/kernel/iommu_groups/38/devices/0000:ff:0d.2
/sys/kernel/iommu_groups/38/devices/0000:ff:0d.0
/sys/kernel/iommu_groups/28/devices/0000:0b:00.0
/sys/kernel/iommu_groups/56/devices/0000:ff:13.0
/sys/kernel/iommu_groups/56/devices/0000:ff:13.5
/sys/kernel/iommu_groups/56/devices/0000:ff:13.1
/sys/kernel/iommu_groups/56/devices/0000:ff:13.6
/sys/kernel/iommu_groups/56/devices/0000:ff:13.4
/sys/kernel/iommu_groups/18/devices/0000:00:1c.7
/sys/kernel/iommu_groups/46/devices/0000:ff:0f.6
/sys/kernel/iommu_groups/36/devices/0000:ff:0b.0
/sys/kernel/iommu_groups/36/devices/0000:ff:0b.3
/sys/kernel/iommu_groups/8/devices/0000:00:11.0
/sys/kernel/iommu_groups/26/devices/0000:09:00.0
/sys/kernel/iommu_groups/54/devices/0000:ff:10.7
/sys/kernel/iommu_groups/16/devices/0000:00:1c.4
/sys/kernel/iommu_groups/44/devices/0000:ff:0f.4
/sys/kernel/iommu_groups/34/devices/0000:ff:09.4
/sys/kernel/iommu_groups/6/devices/0000:00:05.2
/sys/kernel/iommu_groups/24/devices/0000:07:00.0
/sys/kernel/iommu_groups/52/devices/0000:ff:10.5
/sys/kernel/iommu_groups/14/devices/0000:00:1c.1
/sys/kernel/iommu_groups/42/devices/0000:ff:0f.2
/sys/kernel/iommu_groups/32/devices/0000:ff:09.0
/sys/kernel/iommu_groups/4/devices/0000:00:03.0
/sys/kernel/iommu_groups/22/devices/0000:01:00.0
/sys/kernel/iommu_groups/50/devices/0000:ff:10.3
/sys/kernel/iommu_groups/12/devices/0000:00:1b.0
/sys/kernel/iommu_groups/40/devices/0000:ff:0f.0
/sys/kernel/iommu_groups/30/devices/0000:ff:08.3
/sys/kernel/iommu_groups/2/devices/0000:00:01.1
/sys/kernel/iommu_groups/20/devices/0000:00:1e.0
/sys/kernel/iommu_groups/49/devices/0000:ff:10.2
/sys/kernel/iommu_groups/10/devices/0000:00:19.0
/sys/kernel/iommu_groups/39/devices/0000:ff:0e.1
/sys/kernel/iommu_groups/39/devices/0000:ff:0e.0
/sys/kernel/iommu_groups/29/devices/0000:ff:08.0
/sys/kernel/iommu_groups/0/devices/0000:00:00.0
/sys/kernel/iommu_groups/19/devices/0000:00:1d.0
/sys/kernel/iommu_groups/47/devices/0000:ff:10.0
/sys/kernel/iommu_groups/37/devices/0000:ff:0c.6
/sys/kernel/iommu_groups/37/devices/0000:ff:0c.2
/sys/kernel/iommu_groups/37/devices/0000:ff:0c.0
/sys/kernel/iommu_groups/37/devices/0000:ff:0c.7
/sys/kernel/iommu_groups/37/devices/0000:ff:0c.1
/sys/kernel/iommu_groups/9/devices/0000:00:16.0
/sys/kernel/iommu_groups/27/devices/0000:0a:00.0

my GPU iommu groups should be:
/sys/kernel/iommu_groups/23/devices/0000:01:00.1
/sys/kernel/iommu_groups/22/devices/0000:01:00.0


Dual GPU:
if you think that the single GPU could be the problem I can by an additional cheap graphic card in order to leave proxmox with at least one.

Maybe the root problem is that with lspci -v a don't see the "Kernel driver in use: vfio-pci " loaded to the GPU.
this could explain why the VM can't connect to the GPU.
Is there any log possibility to understand why the vfio_pci driver doesn't load for the GPU?

many thanks in advance

 

[LnxBil]

Dual GPU:
if you think that the single GPU could be the problem I can by an additional cheap graphic card in order to leave proxmox with at least one.

Maybe the root problem is that with lspci -v a don't see the "Kernel driver in use: vfio-pci " loaded to the GPU.
this could explain why the VM can't connect to the GPU.
Is there any log possibility to understand why the vfio_pci driver doesn't load for the GPU?

I would at least try the multi GPU thing.

You have to unbind your GPU from the current driver and rebind to vfio_pci in order to try that. You can also try to blacklist the driver.

 

[r.jochum]

Try that:

https://clayfreeman.github.io/gpu-passthrough/#blacklisting-the-gpu

After that you should have no more "BAR 3".

But then you will run in ERROR 43, you need to load the ROM from the GPU maybe Patched:

https://forum.proxmox.com/threads/nvidia-gtx-1050ti-error-43-code-43.75553/

 

[gen2k4k]

Hello,
thanks to your hints I now have my GPU working passthrough on my Win10 VM.
Steps:
- Purchased secondo GPU Geforce GT 710
- Put new GPU in first MB PCIe slot in order to use it with proxmox
- Configured the win10 pro VM:
- no UEFI bios, it didn't work with UEFI
- here the changed config parameters that worked for me

agent: 1
args: -cpu 'host,+kvm_pv_unhalt,+kvm_pv_eoi,hv_vendor_id=NV43FIX,kvm=off'
balloon: 1024
cpu: host,hidden=1,flags=+pcid
hostpci0: 02:00,pcie=1,x-vga=1
machine: q35
net0: virtio=56:40:FA:72:132,bridge=vmbr0,firewall=1
numa: 0
ostype: win10

many thanks again