Host Dead After Performing VM Backup

[M3Power95]

Hello,

I am running an Intel NUC running Home Assistant in Proxmox. I have been doing periodic backups to the host local storage, but have been typically using the "fast and good" method. This time I used the "good" method and shutdown the HA VM before performing the backup. As far as I could tell from the backup log, all went well. But then the HA vm would not actually start even though Proxmox was telling me that it did start. Eventually I tried rebooting proxmox from the host, and it never came back up. Multiple power cycles, unplugging all peripherals later and still nothing. Took the NUC up to my PC workstation and hooked up the HDMI out to a monitor and nothing is coming up. It looks like the NUC powers up no problem, but I get no HDMI out, ethernet link lights come on but have the slow "I'm not communicating" look to them and I can't ping proxmox or find it on my LAN.

What options do I have for recovering this? I've read I should try to re-load Proxmox VE ISO onto a USB and try to boot from that, and there's some sort of rescue function?

Any help would be appreciated. I'm definitely not looking forward to having to completely start over with my HA setup

 

[M3Power95]

Update. Was able to set up a proxmox USB and do a rescue boot. It found the following issue:

/dev/mapper/pve-root contains a file system with errors, check forced.
Inodes that were part of a corrupted orphan linked list found.

/dev/mapper/pve-root: UNEXPECTED INCONSISTENCY; RUN fsck MANUALLY (i.e., without -a or -p options)
fsck existed with status code 4
done.
Failure: File system check of the root filesystem failed
The root filesystem on /dev/mapper/pve-root requires a manual fsck

I checked the built in commands and don't see a fsck option, and don't want to mess this up any more since I was able to at least get it back this far.

Any suggestions?

Thanks

 

[M3Power95]

Ok so I keep fixing my own problems lol. Executed fsck /dev/mapper/pve-root and it found a bunch of problems, asked me if I wanted them to be fixed. I said yes to all (obviously), and then it took me to initramfs. I just exited and then the boot took off and proxmox is BACK! However, it seems that HA is not starting up. Attaching two screenshots that show the summary and the console. Doesn't look like what I'm used to with HA.

I went back and checked the backup I did in Proxmox and it's not there, so it must have completely bugged out. Has proxmox somehow lost the current HA instance? I do have what appears to be a good backup from a few weeks ago, which sucks but is so much better than not having one. Ideally I'd like to recover the instance from today, but I'll restore this if there is no other option...

Any ideas?

 

[M3Power95]

Update. Still can't get HAOS to run. Even tried spinning up a new one and it doesn't work. I still get the "Guest Agent not running" error. I also cannot stop, shutdown or reboot the VM. I keep getting the following errors. Cannot restore from backup either. Unless anyone has a better idea, I think I'll be forced to do a complete wipe and re-install of Proxmox. I was able to SCP into the VM and grab the backup, so at least I can partially get my HA back up.

Thanks

 

[M3Power95]

Any takers? Will be doing a full wipe and re-install of Proxmox and HA tonight if nobody has any ideas, and will update on here. Thanks

 

[teles1h]

Same exact issue. My host won't book
/dev/mapper/pve-root: clean xxxxx/files, xxxxx/xxxx blocks

 

[M3Power95]

Did yours happen after you did a full backup on a guest VM in Proxmox? I wouldn't have thought simply backing something up could cause such a major issue at the OS level.

 

[khe]

Your HA VM has somehow lost the boot entry. Either the UEFI does not find the boot disk or the boot order is wrong. Check the boot order in the options of the VM configuration.
You might need a rescue disk for HA to restore the boot sector inside the VM if the VM config is correct.

 

[M3Power95]

Your HA VM has somehow lost the boot entry. Either the UEFI does not find the boot disk or the boot order is wrong. Check the boot order in the options of the VM configuration.
You might need a rescue disk for HA to restore the boot sector inside the VM if the VM config is correct.

Didn't know there were rescue disks for HAOS specifically. I will search for one and check my boot order. Thanks

 

[khe]

According to the HAOS installation homepage you need to use Ubuntu for that and use the efibootmgr command.
See the first note below the section START UP YOUR GENERIC X86-64.

 

[M3Power95]

According to the HAOS installation homepage you need to use Ubuntu for that and use the efibootmgr command.
See the first note below the section START UP YOUR GENERIC X86-64.

Thanks. I'm not sure this applies to a Proxmox installation though. I can try to boot the NUC and see if there is anything in there that looks odd, but from the Proxmox perspective it does have the correct boot order. Thanks

 

[M3Power95]

According to the HAOS installation homepage you need to use Ubuntu for that and use the efibootmgr command.
See the first note below the section START UP YOUR GENERIC X86-64.

Went into my BIOS and it looks like Legacy and UEFI boots are both enabled. I could have sworn I had one of them disabled when I first set up the NUC to run Proxmox. Is it correct to have both boot modes enabled?

 

[M3Power95]

Whelp. After multiple tries, somehow I was able to get the old VM to stop. So I gave up and used a successful HAOS save from 4/24 to spin up a new VM and all is well now. Just frustrating that nothing worked, and totally crapped out on me just for doing a backup...