GPU passthrough

[Basosa]

Dear all,

I am new and would like to use proxmox to have two VM one with Windows for office and gaming and one with unbuntu for tensorflow work.

I managed to install proxmox as well as both os. I also followed those steps for gpu passthrough for an nvidia 1080TI:

1. Enable iommu on bios
2. Change GRUB_CMDLINE_LINUX_DEFAULT="quiet amd_iommu=on" and update-grub
3. Add vfiovfio_iommu_type1, vfio_pci, vfio_virqfd to nano /etc/modules
4. remamp:
   1. echo "options vfio_iommu_type1 allow_unsafe_interrupts=1" > /etc/modprobe.d/iommu_unsafe_interrupts.conf
   2. echo "options kvm ignore_msrs=1" > /etc/modprobe.d/kvm.conf
5. Block drivers:
   1. echo "blacklist nvidia" >> /etc/modprobe.d/blacklist.conf
6. Add the gpu to my VFIO
   1. echo "options vfio-pci ids=10de:1b81,10de:10f0 disable_vga=1"> /etc/modprobe.d/vfio.conf
7. update update-initramfs -u
8. edit nano /etc/pve/qemu-server/<vmid>.conf by adding:
   1. cpu: host,hidden=1,flags=+pcid
   2. args: -cpu 'host,+kvm_pv_unhalt,+kvm_pv_eoi,hv_vendor_id=NV43FIX,kvm=off'
9. Add my gpu to the virtual machine

On my windows session in task manager I can't see my gpu, but gpuz and peripheric manager can see it. However, nividia control panel never pops up and everything that seems to run with gpu (steam games, Davinci resolve) never opens.


In addition, my motherboard accepts sr-iov which I believe would allow me to use the same gpu also for my unbuntu VM. How can I set it up ?

Thanks in advance.

 

[leesteken]

amd_iommu=on does nothing because it is on by default. And you don't need to blacklist nvidia because you should not install NVidia drivers on the Proxmox host. You probably don't need vfio_iommu_type1 allow_unsafe_interrupts=1 nor ignore_msrs=1.
Your GPU does not support SR-IOV and you can only PCIe passthrough it to one VM at a time.
If this is the boot (or only) GPU you need this work-around. Note that not all Proxmox installations use GRUB.
I'm almost sure you don't need hidden=1,flags=+pcid or args: -cpu 'host,+kvm_pv_unhalt,+kvm_pv_eoi,hv_vendor_id=NV43FIX,kvm=off', just enable Primary GPU which already does this.
Try booting the same VM temporarily with a Ubuntu Live installer ISO (without installing it) just to test passthrough: it should show output on the physical display connected to the GPU.

 

[Basosa]

I removedhidden=1,flags=+pcid and args: -cpu 'host,+kvm_pv_unhalt,+kvm_pv_eoi,hv_vendor_id=NV43FIX,kvm=off'.
I added initcall_blacklist=sysfb_ini in /etc/default/grub.
I added Primary GPU
But issue remain

 

[leesteken]

I added initcall_blacklist=sysfb_ini in /etc/default/grub.

Please double check with cat /proc/cmdline. Try booting the same VM temporarily with a Ubuntu Live installer ISO (without installing it) just to test passthrough. Then we know if it's a Windows driver issue or not.

 

[Basosa]

cat /proc/cmdline return BOOT_IMAGE=/boot/vmlinuz-5.15.102-1-pve root=/dev/mapper/pve-root ro quiet amd_iommu=on initcall_blacklist=sysfb_init.

On unbuntu lspci | grep VGA return

 

[leesteken]

cat /proc/cmdline return BOOT_IMAGE=/boot/vmlinuz-5.15.102-1-pve root=/dev/mapper/pve-root ro quiet amd_iommu=on initcall_blacklist=sysfb_init.

That looks good (but you can remove the amd_iommu=on).

On unbuntu lspci | grep VGA return

Looks like you did not enable Primary GPU (because the virtual GPU is still present). Please enable Primary GPU and let me know if output on a physical display looks normal when (temporarily) booting Ubuntu Live installer ISO.

 

[Basosa]

I have to go but will try tomorrow. Thanks for your help!

 

[Basosa]

Hi,

I added primary GPU but the output from unbunt remains the same :

 

[leesteken]

I added primary GPU but the output from unbunt remains the same :

My mistake: it appears that Proxmox no longer automatically disables the virtual display when you enable Primary GPU. Please set the virtual Display to None to force the use of the passed through GPU and try Ubuntu again.

 

[Basosa]

Can I use ssh during boot?

 

[leesteken]

Can I use ssh during boot?

Not until the ssh service has started. I don't understand the question.
Alternatively, add a virtual serial port and set the virtual Display to Serial terminal 0? But it think that requires you to install Linux in the VM to setup the kernel parameters for this.

 

[Basosa]

Without display I can't test unbuntu

 

[leesteken]

Without display I can't test unbuntu

The test is if you see Ubuntu on the physical display connected to the GPU. If you do not see UIbuntu on the physical display connected to the GPU then it's not a Windows driver problem but a passthrough or GPU problem. Do you see Ubuntu starting on the physical display connected to the GPU? What is the current configuration file of the VM?

 

[Basosa]

OK I understand, I have nothing on my display connected to the GPU. My current VM config file is like this:

agent: 0
bios: ovmf
boot: order=ide2;ide0;net0
cores: 8
cpu: host
efidisk0: Windows:vm-101-disk-0,efitype=4m,pre-enrolled-keys=1,size=4M
hostpci0: 0000:0c:00,pcie=1,x-vga=1
ide0: Windows:vm-101-disk-1,size=600G
ide2: local:iso/ubuntu-22.04.2-desktop-amd64.iso,media=cdrom,size=4812096K
machine: pc-q35-7.2
memory: 30000
meta: creation-qemu=7.2.0,ctime=1682587482
name: WindowsVM
net0: virtio=4A:BD:8D:AD:FA:28,bridge=vmbr0,firewall=1
numa: 0
ostype: win11
scsihw: virtio-scsi-single
smbios1: uuid=3fccca64-b363-435f-ac4e-593494acb4eb
sockets: 1
tpmstate0: Windows:vm-101-disk-2,size=4M,version=v2.0
vga: none
vmgenid: 0f6b8b62-33f6-4876-8ff2-c89638e177f4

 

[leesteken]

Then I don't know, sorry. Maybe NVidia 1080 needs something special, like using a romfile and maybe even patching it? I have no experience with that but I there have been threads on this forum about that.

 

[Basosa]

OK I will see if I can find a solution on the forum and will post it if I found one. Thank you very much for your help!