[SOLVED] Unable to Start VM with GPU Pass-through

[ambrose.pve]

Hello,

The problem is if the PVE server start with monitor already attached to a NVIDIA GPU. The VM with that GPU cannot successfully start, it stuck with blinking underscore on boot.

The VM with GPU Pass-through only work if it follow below steps:
1. PVE server start without monitor attached.
2. Attached monitor after PVE successfully boot.
3. Start the VM with GPU Pass-through

How could I let it work with the GPU always attached to a monitor?

Thank you.

Ambrose

 

[leesteken]

The 5.15 kernel versions (default since PVE 7.2) don't release the boot framebuffer before passthrough. Here is a minimal work-around for (non-amdgpu) GPUs.

 

[ambrose.pve]

Thank you.

But this did not work for me. The VM with GPU still stuck with blink underscore on boot.

Below are the steps I did to enable GPU Pass-through:

1. Add intel_iommu=on iommu=pt into /etc/default/grub and run update-grub

GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on iommu=pt"

2. Add intel_iommu=on into /etc/kernel/cmdline and run proxmox-boot-tool refresh

root=ZFS=rpool/ROOT/pve-1 boot=zfs quiet intel_iommu=on

3. Add below lines into /etc/modules-load.d/modules.conf

vfio
vfio_iommu_type1
vfio_pci
vfio_virqfd

4. Add below line into /etc/modprobe.d/iommu_unsafe_interrupts.conf

options vfio_iommu_type1 allow_unsafe_interrupts=1"

5. Add below line into /etc/modprobe.d/kvm.conf

options kvm ignore_msrs=1

6. Add below lines into /etc/modprobe.d/blacklist.conf

blacklist nvidia
blacklist nouveau
blacklist radeon

7. Add below line into /etc/modprobe.d/vfio.conf and run update-initramfs -u

options vfio-pci ids=10de:1eb0,10de:10f8,10de:1ad8,10de:1ad9

And I add initcall_blacklist=sysfb_init into /etc/default/grub as suggested, but this still not work for me.

Thank you.

Ambrose

 

[leesteken]

1. Add intel_iommu=on iommu=pt into /etc/default/grub and run update-grub

GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on iommu=pt"

2. Add intel_iommu=on into /etc/kernel/cmdline and run proxmox-boot-tool refresh

root=ZFS=rpool/ROOT/pve-1 boot=zfs quiet intel_iommu=on

Looks like your system boots in UEFI mode on ZFS root, so it uses systemd-boot and not GRUB.

And I add initcall_blacklist=sysfb_init into /etc/default/grub as suggested, but this still not work for me.

I expect that /etc/default/grub is not even used on your system so please remove intel_iommu=on iommu=pt from /etc/default/grub.
Please add initcall_blacklist=sysfb_init to /etc/kernel/cmdline (everyting on the same single line) and run proxmox-boot-tool refresh and reboot.
Check with cat /proc/cmdline to be sure that the changes to /etc/kernel/cmdline are visible. And do not also add it to /etc/default/grub because then we can't check if /etc/kernel/cmdline is actually the right one.

 

[ambrose.pve]

Looks like your system boots in UEFI mode on ZFS root, so it uses systemd-boot and not GRUB. I expect that /etc/default/grub is not even used on your system so please remove intel_iommu=on iommu=pt from /etc/default/grub. Please add initcall_blacklist=sysfb_init to /etc/kernel/cmdline (everyting on the same single line) and run proxmox-boot-tool refresh and reboot. Check with cat /proc/cmdline to be sure that the changes to /etc/kernel/cmdline are visible. And do not also add it to /etc/default/grub because then we can't check if /etc/kernel/cmdline is actually the right one.

This work, thank you so much.

Ambrose