[SOLVED] Trying to recover from dead(?) install

[AeonRemnant]

After redoing my GPU passthrough and updating GRUB I’m now getting stuck at recovering journal whenever I try and boot into the disk.
All I changed in GRUB was ‘amd_iommu=on and iommu=pt’ as instructed in the wiki. After running a GRUB update and rebooting I can’t get back in.
Now theoretically it could hang because the nvidia drivers are blacklisted, but even after waiting far longer than enough I'm getting no network activity, I can't SSH, can't access panel, nothing after recovering journal.

I have no backup for Proxmox (the backup server was supposed to go online in a few days, bad timing for shipped components) and I either need to recover this somehow or scrape the required config files to rebuild my node.
Definitely lesson learned on not having an abundance of healthy backup at all times.

How exactly do I go about fixing this? Undoing the GRUB config when booting up leaves me hanging in the same place.
Using the Proxmox USB media recovery does nothing and simply hangs in the same position.

 

[Moayad]

Hi,

Please post your GRUB config GRUB_CMDLINE_LINUX_DEFAULT

Have you tried to boot from recovery mode and do fsck on the pve-root?

 

[AeonRemnant]

Hi,

Please post your GRUB config GRUB_CMDLINE_LINUX_DEFAULT

Have you tried to boot from recovery mode and do fsck on the pve-root?

Hey, thanks.
So I can't actually boot from recovery mode, the system hangs when I try to.

GRUB: https://pasteboard.co/uWpYLlrIQsHg.png
Recovery log: https://pasteboard.co/kWIW6RprQ1tD.png

 

[Moayad]

Hi,

When you did the update-grub did you get any error?
When the system hangs, did you try to do ctrl+d?

Another question, have you tried to set the GRUB cmd as the default value? If so, does the boot from recovery mode works?

 

[AeonRemnant]

Hi,

When you did the update-grub did you get any error?
When the system hangs, did you try to do ctrl+d?

Another question, have you tried to set the GRUB cmd as the default value? If so, does the boot from recovery mode works?

No errors I remember on update-grub, ctrl+d does nothing, and yes I’ve tried setting GRUB’s config to default when booting.

 

[Moayad]

Is the booting from the old kernel also doesn't help? Basically, the Rescue boot should work, do you use the PVE ISO for rescue boot?

 

[AeonRemnant]

Is the booting from the old kernel also doesn't help? Basically, the Rescue boot should work, do you use the PVE ISO for rescue boot?

Just tried Rescue Boot again, my system hangs and I'm not able to access it in any way, same as before. Also not seeing the IP or MAC on the router.

Screenshot attached.
Rescue Boot: https://pasteboard.co/9d36dSbPoFDN.png

 

[Moayad]

Have you also tried booting from an old PVE-kernel?

I'm not sure if this will help, but you can try to set nomodeset on your GRUB config?

 

[AeonRemnant]

Have you also tried booting from an old PVE-kernel?

I'm not sure if this will help, but you can try to set nomodeset on your GRUB config?

The GPU in the system is a 3070, nomodeset *shouldn't* be required. I did try it regardless and it still hangs, and gets to POSIX File System message a little further down into the log. Tried with different Kernels both normal and recovery mode, also tried cleaning out the GRUB config to check what's happening.
Every time it hangs somewhere around the POSIX message. Sometimes it's POSIX, sometimes it's keyboard language settings, sometimes it's other stuff.

Not sure this is going to work, is there a way to recover the config from a live install or shall we continue trying to fix the boot issue?

 

[AeonRemnant]

Solution ended up being remove the nvidia GPU and replace it with an AMD one.
Not sure what the hell happened, but if some of the random BS in this thread happens maybe switch the GPU hardware so it's a different ID or change your driver blacklist so VFIO doesn't take over?
Some method to prevent driver blacklist I guess.