Single GPU setup - display console on boot and passthrough to VM

[skylowe]

Hi.

I have been searching for a solution to this use case - at one time this did work but after a few updates to proxmox something changed.

As it stands I have my VM's working nicely with a single Nvidia GPU passed through.

My GRUB conf line is

GRUB_CMDLINE_LINUX_DEFAULT="amd_iommu=on pcie_no_flr=1022:149c,1022:1487 video=efifb:off consoleblank=1"

lspci -nnv is

0b:00.0 VGA compatible controller [0300]: NVIDIA Corporation Device [10de:1e81] (rev a1) (prog-if 00 [VGA controller])
    Subsystem: Gigabyte Technology Co., Ltd Device [1458:3ff4]
    Flags: bus master, fast devsel, latency 0, IRQ 85
    Memory at fb000000 (32-bit, non-prefetchable) [size=16M]
    Memory at d0000000 (64-bit, prefetchable) [size=256M]
    Memory at e0000000 (64-bit, prefetchable) [size=32M]
    I/O ports at d000 [size=128]
    Expansion ROM at 000c0000 [disabled] [size=128K]
    Capabilities: [60] Power Management version 3
    Capabilities: [68] MSI: Enable- Count=1/1 Maskable- 64bit+
    Capabilities: [78] Express Legacy Endpoint, MSI 00
    Capabilities: [100] Virtual Channel
    Capabilities: [258] L1 PM Substates
    Capabilities: [128] Power Budgeting <?>
    Capabilities: [420] Advanced Error Reporting
    Capabilities: [600] Vendor Specific Information: ID=0001 Rev=1 Len=024 <?>
    Capabilities: [900] #19
    Capabilities: [bb0] #15
    Kernel driver in use: vfio-pci
    Kernel modules: nvidiafb, nouveau

What I want to know is, is there a way to have the host system boot with console output to GPU, and then when a VM is booted it is passed to the VM, and when the VM is stopped have the GPU passed back to console

Thanks for the feedback in advance

 

[skylowe]

Perhaps there is a command line that I can use to reset the card and force it to use a different driver like nouveau?

 

[leesteken]

What I want to know is, is there a way to have the host system boot with console output to GPU, and then when a VM is booted it is passed to the VM, and when the VM is stopped have the GPU passed back to console

I got the first part working with a RX570 by NOT using video=efifb:off and NOT blacklisting amdgpu. It just shows console output until I start the VM.
For the second part, maybe this post can help you? I have not yet tried it with my setup, but intend to.

 

[skylowe]

I got the first part working with a RX570 by NOT using video=efifb:off and NOT blacklisting amdgpu. It just shows console output until I start the VM.
For the second part, maybe this post can help you? I have not yet tried it with my setup, but intend to.

Yes I did this - and lspci shows kernel driver - nouveau. Then I run the VM and vfio-pci is bound. When I run that reset code, the kernel driver is still vfio-pci and I dont see output. How do I get the nouveau driver back - I think that is the key.

 

[leesteken]

Have you tried unbinding the vfio-pci driver by using echo '0000:0b:00.0' >/sys/bus/pci/drivers/vfio-pci/unbind?
Maybe you need to unbind the other multi-function devices (like 0b:00.1 etc.) also.

PS: I tried it, and got it working in my case (using hookscripts). Maybe these steps will help you too:
In my case, I cannot reset like the mentioned post and I needed to explicitly bind the GPU to the driver. (I did not need to unbind 0b:00.1.)
In your case, this would be: echo '0000:0b:00.0' >/sys/bus/pci/drivers/nouveau/bind. (After unbinding 0b:00.0.)
Then I do see the Proxmox console and can actually login, if I also unbind the USB controller and binding it to xhci_hcd again.

PPS: In my case I had to use vendor-reset to be able to start the VM again after unbind it from vfio-pci and rebind it to amdgp, because it was stuck in D3.

 

[skylowe]

Have you tried unbinding the vfio-pci driver by using echo '0000:0b:00.0' >/sys/bus/pci/drivers/vfio-pci/unbind?
Maybe you need to unbind the other multi-function devices (like 0b:00.1 etc.) also.

PS: I tried it, and got it working in my case (using hookscripts). Maybe these steps will help you too:
In my case, I cannot reset like the mentioned post and I needed to explicitly bind the GPU to the driver. (I did not need to unbind 0b:00.1.)
In your case, this would be: echo '0000:0b:00.0' >/sys/bus/pci/drivers/nouveau/bind. (After unbinding 0b:00.0.)
Then I do see the Proxmox console and can actually login, if I also unbind the USB controller and binding it to xhci_hcd again.

PPS: In my case I had to use vendor-reset to be able to start the VM again after unbind it from vfio-pci and rebind it to amdgp, because it was stuck in D3.

Hi.

I tried all the suggestions with limited success - every now and then something would work and the console output would come back after unbinding the vfio driver, but the behaviour is not consistent.

I also forgot to state that my GPU is in PCI slot 2 - if this makes any difference and I do not have onboard graphics

 

[skylowe]

So some more testing. I removed all the driver and vfio specific config. When proxmox boots it displays the text console. At this point I can change drivers (from nouveau to vfio-pci at will) with no problem.

However when I launch the windows 10 vm - it launches correctly and the GTX 2080 super is passed though to the guest OS and everything functions as expected. Then I shutdown the VM and unload vfio-pci drivers load nouveau drivers and remove then rescan the pci device.

dmesg gives a strange error - nouveau training display failed.

If I could get to the root of this error I think that my problems will be solved.

Does any one have any more ideas??