Proxmox Stuck on Boot after passthrough

[macnug]

Hi everyone,
I have my PROXMOX server installed on a intel nuc10i5 8gb ram and two ssds
On one Home assistant VM, pihole in a CT and in the second ssd a Ubuntu VM with zoenminder.

I tried to passthrough the GPU but when i restarted proxmox, now it's stuck whitout GUI and terminal.

Here two screenshots:
The shorter code is the normal boot where the boot is stuck on /dev/mapper/pve-root: clean.

Using the recovery mode using the Proxmox ISO it is stuck on vfio-pci XXXX vgaarb:changed VGA decodes: oldecodes=io+mem,decodes=io+memwns=io+mem

Thanks in advance to anyone can help to solve this mess.

Marco

 

[leesteken]

Can you press the e key in the boot menu? You can then change the kernel parameters temporarily and prevent the VM from starting (like removing intel_iommu=on). Do you then get a console to login? Hopefully you can undo your changes after login.

 

[andro0]

Anyone found solution for this? I got a dead Proxmox server because of this too

 

[leesteken]

Anyone found solution for this? I got a dead Proxmox server because of this too

Can you see the boot menu? Can you press the e key and change the kernel parameters? Maybe we can prevent your broken VM from automatically starting.

 

[andro0]

@avw I see grub boot, I pressed "e", I removed iommu and pcie passthrough related stuff after "quiet", still nothing. I can't prevent guest VM from starting automatically so I can go and remove the PCIE passthrough from VM config

 

[leesteken]

@avw I see grub boot, I pressed "e", I removed iommu and pcie passthrough related stuff after "quiet", still nothing. I can't prevent guest VM from starting automatically so I can go and remove the PCIE passthrough from VM config

intel_iommu or amd_iommu (you didn't specify which) could be enable automatically. PLease try intel_iommu=off amd_iommu=off. The iommu parameter does not do what you think it does.

 

[andro0]

@avw I set it to on/off, with/without adding it, it still crashes. the PCIE device I'm passing through a 4x 10GB ethernet card.

It's AMD

 

[leesteken]

@avw I set it to on/off, with/without adding it, it still crashes. the PCIE device I'm passing through a 4x 10GB ethernet card.

It's AMD

Did you set amd_iommu=off or iommu=off? Sorry, but this is not clear from your message. That should disable PCI(e) passthrough. If this does not work, I'm sorry, I don't know another way.
Maybe PCI(e) passthrough is not the problem in your case? Can you describe your problem in more detail? What did you change? What is not working? What do you see on the console? Can you connect to the machine remotely? Can you boot the machine with another Linux Live CD (systemrescue or Ubuntu or other)?

 

[andro0]

@avw It's 100% PCIe passthrough, I set it off specifically, amd_iommu=off, I removed amd_iommu entirely and I left it on. All 3 cases, leads to VM autostarting and crashing the system immediately. How can I enter recover mode and edit the conf file? I did init=/bin/bash as option, dropped to shell, tried to mount lvm storage one by one but couldn't find the conf file. How do I enter proper recovery mode in proxmox?

 

[leesteken]

If this "easy" way does not work for you, you need to boot the system using something else and disable the VM from automatically starting. I don't know how to do that because /etc/pve is not a filesystem but a database. Sorry. Maybe someone else here knows a way without reinstalling and using backups?

 

[leesteken]

Have you tried disabling all virtualization option in the BIOS? Maybe that can prevent the VM from starting and breaking your system.

 

[wowe]

Have you tried disabling all virtualization option in the BIOS? Maybe that can prevent the VM from starting and breaking your system.

I have registered only to thank you for that hint.
I had the same issue as described here and disabling the virtualization on the host cpu did the trick to avoid auto startup of my VMs.
Thank you!

 

[JamborJan]

Have you tried disabling all virtualization option in the BIOS? Maybe that can prevent the VM from starting and breaking your system.

Dear @leesteken you are a live saver! Thanks for that hint. My scenario of total fail:
- moved a VM with GPU requirements from one node in a cluster to another
- there was also a GPU installed but as soon as I passed that through the whole system went down when the VM started
- Having a AMD CPU, I disabled BIOS > Advanced > CPU > SVM Mode
- Reboot, proxmox booted again as expected, VM stayed turned off
- Adjust settings of the VM (disable auto boot, remove failing HW)
- Reboot into BIOS, enable virtualization again
- Test and done

 

[majk]

@leesteken your'e a genius! I almost formatted my whole Proxmox, because I thought that it's lost. By the way, I was experimenting with Passthrough and accidentally enabled PT for SATA controller (so as well host's hard drive). Of course I forgot to disable auto-run for VM. But now lesson is learnt. Editting the GRUB with 'e' button did the trick. Thanks mate for hint!

 

[buriedsunshine]

I'm running into the exact same issue except after I disable my VM's from automatically booting, I re-enable SVM in BIOS and try to boot into Proxmox I'm met with the same problem. Proxmox will only boot when I disable IOMMU and SVM.

Would appreciate being pointed in the right direction as I'm contemplating backing up my VM's (mainly my Truenas core VM) and re-installing Proxmox.

 

[eminempt]

Have you tried disabling all virtualization option in the BIOS? Maybe that can prevent the VM from starting and breaking your system.

lol you are a genious

 

[nougatbyte]

tldr:
It might not be a vfio error but some other problem that prevents booting - but configuring grub in a way that lets you see the console on a headless setup might lead to the vfio error.

Just wanted to add something to this since I also thought I had this problem.
I have a VFIO Setup which ran for years without problems and then faced the same problem as @macnug.

After seeing the vfio error when using recovery mode I was also thinking it has to be passtrhough - but since nothing changed on the Passthrough side it had to be something else:
After I booted into runlevel 3 with amd amd_iommu=off I was able to see that I had a typo in my fstab and it just took forever to skip.

- So the vfio-pci error only appeared for me because I had to disable the kernel parameters that disconnect the framebuffer/console from the GPU in order to be able to see the console. It was fstab all along.

Hope this helps someone