[SOLVED] Trying to recover from dead(?) install

AeonRemnant

New Member
Mar 4, 2022
7
0
1
24
After redoing my GPU passthrough and updating GRUB I’m now getting stuck at recovering journal whenever I try and boot into the disk.
All I changed in GRUB was ‘amd_iommu=on and iommu=pt’ as instructed in the wiki. After running a GRUB update and rebooting I can’t get back in.
Now theoretically it could hang because the nvidia drivers are blacklisted, but even after waiting far longer than enough I'm getting no network activity, I can't SSH, can't access panel, nothing after recovering journal.

I have no backup for Proxmox (the backup server was supposed to go online in a few days, bad timing for shipped components) and I either need to recover this somehow or scrape the required config files to rebuild my node.
Definitely lesson learned on not having an abundance of healthy backup at all times.

How exactly do I go about fixing this? Undoing the GRUB config when booting up leaves me hanging in the same place.
Using the Proxmox USB media recovery does nothing and simply hangs in the same position.
 

Attachments

  • error.png
    error.png
    390.7 KB · Views: 10

Moayad

Proxmox Staff Member
Staff member
Jan 2, 2020
1,756
143
68
29
Vienna
shop.maurer-it.com
Hi,

Please post your GRUB config GRUB_CMDLINE_LINUX_DEFAULT

Have you tried to boot from recovery mode and do fsck on the pve-root?
 

Moayad

Proxmox Staff Member
Staff member
Jan 2, 2020
1,756
143
68
29
Vienna
shop.maurer-it.com
Hi,

When you did the update-grub did you get any error?
When the system hangs, did you try to do ctrl+d?

Another question, have you tried to set the GRUB cmd as the default value? If so, does the boot from recovery mode works?
 

AeonRemnant

New Member
Mar 4, 2022
7
0
1
24
Hi,

When you did the update-grub did you get any error?
When the system hangs, did you try to do ctrl+d?

Another question, have you tried to set the GRUB cmd as the default value? If so, does the boot from recovery mode works?
No errors I remember on update-grub, ctrl+d does nothing, and yes I’ve tried setting GRUB’s config to default when booting.
 

Moayad

Proxmox Staff Member
Staff member
Jan 2, 2020
1,756
143
68
29
Vienna
shop.maurer-it.com
Have you also tried booting from an old PVE-kernel?

I'm not sure if this will help, but you can try to set nomodeset on your GRUB config?
 

AeonRemnant

New Member
Mar 4, 2022
7
0
1
24
Have you also tried booting from an old PVE-kernel?

I'm not sure if this will help, but you can try to set nomodeset on your GRUB config?
The GPU in the system is a 3070, nomodeset *shouldn't* be required. I did try it regardless and it still hangs, and gets to POSIX File System message a little further down into the log. Tried with different Kernels both normal and recovery mode, also tried cleaning out the GRUB config to check what's happening.
Every time it hangs somewhere around the POSIX message. Sometimes it's POSIX, sometimes it's keyboard language settings, sometimes it's other stuff.

Not sure this is going to work, is there a way to recover the config from a live install or shall we continue trying to fix the boot issue?
 

AeonRemnant

New Member
Mar 4, 2022
7
0
1
24
Solution ended up being remove the nvidia GPU and replace it with an AMD one.
Not sure what the hell happened, but if some of the random BS in this thread happens maybe switch the GPU hardware so it's a different ID or change your driver blacklist so VFIO doesn't take over?
Some method to prevent driver blacklist I guess.
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get your own in 60 seconds.

Buy now!