Issue booting VM after installation with GPU passthrough (Ubuntu, H11DSi, EPYC 7601)

G

grosfaignan

New Member
Aug 26, 2025
5
0
1
Hello,

I’m trying to set up Proxmox on the following hardware:

  • Motherboard: Supermicro H11DSi
  • CPUs: 2 × AMD EPYC 7601
  • RAM: 192 GB ECC
  • GPU: NVIDIA RTX 3050 (passthrough using vfio-pci)

VM config :

Bash: 
root@pve:~# qm config 103
bios: ovmf
boot: order=scsi0
cores: 8
cpu: host,hidden=1
efidisk0: local-lvm:vm-103-disk-0,size=4M
hostpci0: 21:00.0,pcie=1,x-vga=1
hostpci1: 0000:14:00.3,pcie=1,rombar=0
ide2: local:iso/ubuntu-24.04.3-desktop-amd64.iso,media=cdrom,size=6197156K
machine: q35
memory: 16000
meta: creation-qemu=10.0.2,ctime=1756088604
name: Copy-of-VM-GpuPTTemplate
net0: e1000=BC:24:11:63:5D:19,bridge=vmbr0
ostype: l26
scsi0: local-lvm:vm-103-disk-2,discard=on,size=32G
smbios1: uuid=3a63ef58-6bf1-4057-bf90-936c6ab3a1f3
vmgenid: 2d24d441-25f5-403a-b878-8b9e83f0d046


Proxmox and kernel version :

Bash: 
root@pve:~# pveversion
pve-manager/9.0.3/025864202ebb6109 (running kernel: 6.14.8-2-pve)

root@pve:~# uname -a
Linux pve 6.14.8-2-pve #1 SMP PREEMPT_DYNAMIC PMX 6.14.8-2 (2025-07-22T10:04Z) x86_64 GNU/Linux


Context :


  • Proxmox installs fine on the host.
  • I can create a VM and boot from an ISO installer (e.g., Ubuntu) with GPU passthrough (plus keyboard and mouse).
  • The installation itself works without issues.
  • GPU passthrough is done via vfio-pci and an NVIDIA RTX 3050.
  • Problem: after finishing the OS installation and rebooting the VM, it does not boot into the OS. Instead, the VM goes straight into the UEFI shell, as if no bootloader was found.

So, the VM works fine when booting from ISO, but once the OS is installed on disk, it seems like the EFI boot entry is missing or not being recognized.


What I’ve already tried :


  • Created multiple VMs from the same template, issue is reproducible.
  • Tried manually assigning a boot entry name for the ESP in the VM’s UEFI BIOS.
  • Mounted the EFI partition and verified its contents:

Bash: 
ubuntu@ubuntu:/mnt$ ls -R /mnt/EFI
/mnt/EFI:
BOOT  ubuntu

/mnt/EFI/BOOT:
BOOTX64.EFI  fbx64.efi  mmx64.efi

/mnt/EFI/ubuntu:
BOOTX64.CSV  grub.cfg  grubx64.efi  mmx64.efi  shimx64.efi

Attempted to recreate EFI boot entry with efibootmgr:


Bash: 
ubuntu@ubuntu:/mnt$ sudo mount /dev/sda2 /mnt
ubuntu@ubuntu:/mnt$ sudo mount /dev/sda1 /mnt/boot/efi
ubuntu@ubuntu:/mnt$ sudo mount --bind /dev /mnt/dev
ubuntu@ubuntu:/mnt$ sudo mount --bind /proc /mnt/proc
ubuntu@ubuntu:/mnt$ sudo mount --bind /sys /mnt/sys
ubuntu@ubuntu:/mnt$ sudo chroot /mnt
root@ubuntu:/# efibootmgr -c -d /dev/sda -p 1 -L "ubuntu" -l '\EFI\ubuntu\grubx64.efi'
EFI variables are not supported on this system.
root@ubuntu:/# exit


This suggests EFI variables are not writable in this environment.


Questions :


  • Has anyone seen this behavior with GPU passthrough and UEFI/OVMF?
  • Is there something specific I should configure in Proxmox (e.g., EFI disk, boot order, OVMF settings) to make the VM detect the installed bootloader?
  • Could this be related to NVIDIA RTX 3050 passthrough (e.g., OVMF NVRAM not persisting, or a GPU ROM/vBIOS quirk)?

Information I can provide if needed
  • Output of lspci -nnk for the GPU and related devices
  • Host BIOS/UEFI settings (IOMMU, ACS, SR-IOV, Above 4G decoding, etc.)
  • Logs from the VM boot attempt

Thanks in advance for any advice!
 
I think we should isolate whether it is a pass-through influence relationship.

If there is no pass-through, is there a problem with booting after installation?
If there is no problem, what happens if the pass-through is done post-installation?
 
Actually, that’s exactly what I did. I first tried to install the OS with GPU passthrough enabled, but the installation failed — the VM wouldn’t boot properly.

After that, I installed the OS without passthrough, and the installation completed without any issues. Once the OS was installed, I enabled GPU passthrough for the RTX 3050, and graphics output works perfectly.

The only remaining problem is the HDMI audio passthrough (21:00.1). If I try to enable it — either via multifunction=on or assigning it separately — the VM fails to boot or Proxmox reports device '0000:21:00.1' assigned more than once. GPU passthrough alone (21:00.0) works fine.

So in short: the VM boots fine without passthrough, GPU passthrough works post-installation, but HDMI audio passthrough remains the issue.


[Edit] Based on advice I received, I plan to dump the GPU VBIOS and add it to the VM configuration. Is there some good idea ?
 
Last edited:
I don't have an nvidia gpu on hand right now so I can't say for sure, but I never specified vbios when I was using rtx a2000.

We do not know the complete configuration that was used in the past because it is no longer available, but the following was the configuration that would have been used in the past.

Code: 
iommu=pt video=efifb:off pcie_acs_override=downstream,multifunction
 
Last edited:
Thanks for sharing your config.


On my setup I can successfully passthrough the RTX 3050 GPU (21:00.0) to different VMs — for example I can boot both a Windows 10 VM and an Ubuntu VM one after the other with VFIO, and the GPU reset between VM reboots seems to work fine.


The issue only appears when I also passthrough the HDMI audio function of the card (21:00.1, NVIDIA High Definition Audio). As soon as I add this PCI function to the VM, the system starts spamming reset/recovery attempts in the logs and the VM does not boot correctly.


Here’s an excerpt from dmesg | grep -i vfio when the audio device is passed through:

Code: 
[26457.772914] vfio-pci 0000:21:00.0: timed out waiting for pen 
ding transaction; performing function level reset anyway 
[26457.879026] vfio-pci 0000:21:00.0: reset done 
[26457.879299] vfio-pci 0000:14:00.3: resetting 
[26457.903567] vfio-pci 0000:14:00.3: reset done 
[26459.425255] vfio-pci 0000:21:00.0: vfio_bar_restore: reset r 
ecovery - restoring BARs 
[26459.425579] vfio-pci 0000:21:00.1: vfio_bar_restore: reset r 
ecovery - restoring BARs 
[26459.467991] vfio-pci 0000:21:00.0: vfio_bar_restore: reset r 
ecovery - restoring BARs 
[26459.468299] vfio-pci 0000:21:00.1: vfio_bar_restore: reset r 
ecovery - restoring BARs 
[26459.500970] vfio-pci 0000:21:00.0: vfio_bar_restore: reset r 
ecovery - restoring BARs 
[26459.501281] vfio-pci 0000:21:00.1: vfio_bar_restore: reset r 
ecovery - restoring BARs 
[26459.541449] vfio-pci 0000:21:00.0: vfio_bar_restore: reset r 
ecovery - restoring BARs


This loop continues with repeated “reset recovery - restoring BARs” messages for both 21:00.0 and 21:00.1.


So at this point it looks like GPU passthrough on its own works correctly, but adding the HDMI audio function of the GPU triggers the classic NVIDIA audio reset bug that some cards are known to have.


Has anyone managed to get stable passthrough of NVIDIA HDMI audio (RTX 30xx in particular) on Proxmox, or is the recommended workaround still to use an alternative audio device (USB sound card, virtual audio, etc.)?
 
Maybe you already do this, but do you pass through 21:00 in all functions instead of 21:00.0 or 21:00.0?
 
i did all i could, with AllFunction option, or with 21:00.1 on separate pcie device, with or without primary gpu options, with/wo romBARS option, with dumped romfile, both as mapped device, and other initramfs options
 
Last edited:
Dear everyone, in addition to previous test, there is some more inofrmation coming with attacheed logs
Is there any way to:
  • Add more detailed information to the current logs (combined configuration and dmesg output), let me know your idea
  • Test alternative configurations in order to troubleshoot HDMI audio passthrough issues
  • Automate those tests to streamline the process and make comparisons easier

From the logs it seems that adding the option rombar=0 prevents the error

Code: 
Invalid PCI ROM header signature: expecting 0xaa55, got 0xffff


However, this does not solve the BAR reset loop issue which still occurs.
 

Attachments

Last edited:
You must log in or register to reply here.
Top Bottom