Proxmox 6 and Nvidia GPU pass through issue

[scrapiron]

I've been reading/searching, but I can't figure it out.

Running ProxMox 6.1-7 on a Dell T7500 (dual Intel CPU, 96GB RAM, one Nvidia 9500GT for the host, one Nvidia GTX 1060 for a VM).
VM is Fedora 31. VM boots up, I see the boot screen, then goes to black. If I boot in 'safe' mode, I can get into the desktop, I see the 1060 as a device inside the VM.

Virtual machine config:
agent: 1
balloon: 0
bios: ovmf
bootdisk: scsi0
cores: 2
cpu: kvm64
efidisk0: local-lvm:vm-102-disk-0,size=4M
hostpci0: 04:00,pcie=1,x-vga=1
hostpci1: 04:00,pcie=1
ide2: local:iso/Fedora-Workstation-Live-x86_64-31-1.9.iso,media=cdrom
machine: q35
memory: 16032
name: FedoraTEST
net0: virtio=02:29: B:97:4A:65,bridge=vmbr0,firewall=1
numa: 0
ostype: l26
scsi0: Fedora:vm-102-disk-0,size=164G
scsihw: virtio-scsi-pci
smbios1: uuid=56de08ce-8fae-4b5c-8d68-f3c70294d16f
sockets: 2
vga: std
vmgenid: 55097e4a-6e24-404e-b594-6f2761d9437e

GRUB:
RUB_DEFAULT=0
GRUB_TIMEOUT=5
GRUB_DISTRIBUTOR="Proxmox Virtual Environment"
GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on iommu=pt pcie_acs_override=downstream,multifunction nofb nomodeset video=vesafb:ff,efifb:ff"
GRUB_CMDLINE_LINUX=""

Output of lspci:
03:00.0 VGA compatible controller: NVIDIA Corporation G96 [GeForce 9500 GT] (rev a1)
Subsystem: eVga.com. Corp. G96C [GeForce 9500 GT]
Kernel modules: nvidiafb, nouveau
04:00.0 VGA compatible controller: NVIDIA Corporation GP106 [GeForce GTX 1060 6GB] (rev a1)
Subsystem: Micro-Star International Co., Ltd. [MSI] GP106 [GeForce GTX 1060 6GB]
Kernel driver in use: vfio-pci
Kernel modules: nvidiafb, nouveau
04:00.1 Audio device: NVIDIA Corporation GP106 High Definition Audio Controller (rev a1)
Subsystem: Micro-Star International Co., Ltd. [MSI] GP106 High Definition Audio Controller
Kernel driver in use: vfio-pci
Kernel modules: snd_hda_intel


If I remove the PCIe card from the VM, it boots fine. What am I missing? Thanks.

 

[scrapiron]

Thanks, I'm not sure what happened there... probably multi-tasking with too many open windows)

This post was referencing my double post for the same issue.

 

[Veeh]

Hello,

Could you share your resolution ?
thanks

 

[dcsapak]

what do you want to accomplish?

VM is Fedora 31. VM boots up, I see the boot screen, then goes to black. If I boot in 'safe' mode, I can get into the desktop, I see the 1060 as a device inside the VM.

which screen? the virtual one or one attached to the gpu?

hostpci0: 04:00,pcie=1,x-vga=1
hostpci1: 04:00,pcie=1

please remove the line with hostpci1, or else you try to pass through the device twice

 

[scrapiron]

I want the Fedora vm to utilize the GTX 1060 as the GPU. This is the virtual screen (see the Fedora boot screen, then goes black).

removed the hostpci1 line- nothing changed, same black screen.

For reference:
IOMMU Group 15 03:00.0 VGA compatible controller [0300]: NVIDIA Corporation G96 [GeForce 9500 GT] [10de:0640] (rev a1)
IOMMU Group 16 04:00.0 VGA compatible controller [0300]: NVIDIA Corporation GP106 [GeForce GTX 1060 6GB] [10de:1c03] (rev a1)
IOMMU Group 17 04:00.1 Audio device [0403]: NVIDIA Corporation GP106 High Definition Audio Controller [10de:10f1] (rev a1)

 

[dcsapak]

This is the virtual screen (see the Fedora boot screen, then goes black).

and on the physical screen?

 

[scrapiron]

ah ha! It works... well, I don't have keyboard/mouse control (with either the physical or remote devices), but I get the login screen.

When I add the PCIe device (GPU), my memory spikes from 0% to 93%).

okay, so what setting have I goofed?

 

[dcsapak]

When I add the PCIe device (GPU), my memory spikes from 0% to 93%).

okay, so what setting have I goofed?

nothing, kvm/qemu has to map the memory when you passthrough a real pci device

 

[scrapiron]

nothing, kvm/qemu hast to map the memory when you passthrough a real pci device

so what do I need to change to allow VNC to display?

 

[dcsapak]

so what do I need to change to allow VNC to display?

well this depends on the guest os, i guess that fedora simply chooses the real graphics card as primary and puts the output there..
you could install e.g. ssh while not passing through and then configuring via ssh
or create an xorg.conf (wayland?) to not use the passthrough card

 

[scrapiron]

Thanks! I'll see which option works better. When I have a resolution, I'll post back. I'll also try another VM OS. Appreciate the assistance!

 

[scrapiron]

okay, I'm more confused.
When I VNC into the VM (not using the console (change vnc 0.0.0.0:100), I get TWO screens- the VNC window on my laptop as primary, and the physical monitor as the secondary screen. I switched the physical monitor to be the primary, as the VNC window to the VM is black.

The monitor is plugged into the secondary GPU. When I plug into the primary 'host' GPU, I see the host machine login (as I should). Aargh, so close!

 

[Veeh]

Hello Scrapiron

I've a similar issue.
If you've a display on your physical monitor and VNC it's basicly a win. I do not need VNC (in the web interface) so like you i selected in windows to only use the physical monitor and VNC screen is black.
If you prefer to use VNC, you can configure it in fedora. Do you have a wey to extend your display ?

 

[scrapiron]

Hello Scrapiron

I've a similar issue.
If you've a display on your physical monitor and VNC it's basicly a win. I do not need VNC (in the web interface) so like you i selected in windows to only use the physical monitor and VNC screen is black.
If you prefer to use VNC, you can configure it in fedora. Do you have a wey to extend your display ?

True- it’s a win and but doesn’t meet my needs. When I’m in the guest OS, I can configure the screens- but when I make VNC the only or primary- it’s solid black. Actually any setting VNC is black. Though that’s where my mouse needs to be to be able to move it.

thought about using a dummy plug on the video card output, but that seems like cheating.
It’s got to be a config somewhere....

 

[Veeh]

Have you tried to start the vm without primary gpu (x-vga) selected ?
And the display to default ? (delete line : vga: std)

 

[scrapiron]

Just tried that. no change. I wonder if it's something in my grub:

GRUB_DEFAULT=0
GRUB_TIMEOUT=5
GRUB_DISTRIBUTOR="Proxmox Virtual Environment"
GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on pcie_acs_override=downstream,multifunction video=vesafbff video=efifbff"
GRUB_CMDLINE_LINUX=""

 

[scrapiron]

Well, I tried Ubuntu20.4 and it works while passing the GPU. 'works' means that it displays in the console/VNC window, has the GPU, and is not using the connected monitor.

So it's the guest OS.. I'll look again at dcsapak's advice to see what should be changed.

 

[Veeh]

He's mine for comparaison.

GRUB_DISTRIBUTOR="Proxmox Virtual Environment"
GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on pcie_acs_override=downstream,multifunction video=efifb:eek:ff"
GRUB_CMDLINE_LINUX=""

It might be due to the OS. In my case both VM are windows10 VM.

 

[scrapiron]

Okay. So I noticed the guest OS was using llvmpipe as the graphics driver- even when using HDMI out from the GTX1060- I was getting 4k- looked good. So I went and updated the Nvidia drivers to the latest... well, now it's still using llvmpipe as the driver:

llvmpipe LLVM 9.0, 128 bits) /llvmpipe (LLVM 9.0, 128 bits)

BUT I'm getting VNC to work. No video out on GTX 1060.