Not sure how to recover host from this?

[Proximate]

I had a power failure which took this server down. When it came back, it just keeps going in reboot after showing this screen for a second or two.
I've read countless posts on all kinds of recovery but I've not seen anything like this so am not sure what I should be doing.
I see the boot screen, grub and mentions of proxmox, all of that is coming from the drive so I'm not sure what is broken.

If anyone has some ideas, leads, I'd sure appreciate some help.

The first image shows it now no longer booting while the second one shows a similar message but it always booted.
The third image shows what I get if I try to use the debug option of the install cdrom.

 

[leesteken]

If anyone has some ideas, leads, I'd sure appreciate some help.

I don't understand how it can be that you get CPU firmware issues because of a power outage. Maybe it's a result of corrupted software on your drives or maybe some BIOS/UEFI settings have changed (because the battery was empty)? Or maybe those messages were always present and you just only noticed now.

The third image shows what I get if I try to use the debug option of the install cdrom.

Your ZFS pool has a higher version/newer features enabled than the ZFS version of the Proxmox 7.1 installer. Please use the 7.2 installer instead and run a scrub of your rpool.

 

[Proximate]

Hi, thanks for your input.

I don't understand it either but this is what has happened.

I can try to upgrade but before doing that, I wanted to ask here to see how I might be able to fix this without losing the vms on this system.
They didn't get backed up for some reason and losing them will be a big problem.

BIOS settings remained correct and the drives appear to be ok, at least at some level since I can see the proxmox loader starting but then it always reboots so I'm not sure what it is I need to fix and don't want to lose those vms.

So I think you are suggesting I download 7.2 and run what, the debug option?
This host is part of a cluster. Can a cluster run with different versions?

 

[leesteken]

So I think you are suggesting I download 7.2 and run what, the debug option?

I don't know, sorry. It's not clear to me why your installation is not working anymore or how to fix it.

You tried running the 7.1 installer and ran into the problem that it can't handle the newer ZFS. I only wanted to let you know that you need the 7.2 installer when you have updated your ZFS pool(s) in the past. I though you had a plan and just needed a working installer.

I do think you can boot your system with the 7.2 installer using the debug option. You should be able to import your old ZFS pool(s). Maybe then you can copy the virtual disks to another system? You won't be able to look into the /etc/pve/ directory because that is actually a database and requires some Proxmox service(s) to be running.

 

[Proximate]

Using 7.2, Rescue Boot, I get the same. Image #3.
Using the debug mode, it seems to run then reboots again.

I think I am dealing with a hardware problem now. If so, I might have lost those vms since the ZFS is a raid one so not sure how I can mount one of the drives to try and recover the vms.

 

[leesteken]

Using 7.2, Rescue Boot, I get the same. Image #3.

I did not expect that. Thanks for letting me know that even 7.2 installer cannot import the latest ZFS pools.

Using the debug mode, it seems to run then reboots again.

I think I am dealing with a hardware problem now. If so, I might have lost those vms since the ZFS is a raid one so not sure how I can mount one of the drives to try and recover the vms.

You can take one of the drives or even both and connect them to another up to date Proxmox system. Maybe you can import the pool using the numeric ID and by giving it a new name (you cannot have two rpools for example). The are options to import a damaged pool if it is not too broken.
You can then copy the virtual disks but you will have to manual recreate the VM configuration files. Make sure not to overwrite existing VMs that might have the same number (also when copying the disks).

 

[Proximate]

I lucked out. I moved the drives over to another blade and everything fired up. As I suspected, hardware after all.
Thanks for the info above however, I'm going to be sure to make notes of this.

Thanks for your help, it was very appreciated.