VM suddenly won't boot, can't add boot option in UEFI

[kjo81]

Hello,

A working VM of Home Assistant is suddenly not booting anymore.
I launched an update of of HAOS (from 11.1 to 11.2), and it never restarted. I'm stucked on UEFI Interactive Shell.

Restoring a VM backup changes nothing.
I can't add any boot option, UEFI's file explorer is empty.
In boot manager nothing happens when I try to enter que Qemu harddisk.
I even restarted the entire Proxmox, but still same issue.

Any idea ?

root@littlebox:~# qm config 103
balloon: 2048
bios: ovmf
boot: order=sata0
cores: 2
efidisk0: local-lvm:vm-103-disk-0,efitype=4m,pre-enrolled-keys=1,size=4M
memory: 4096
name: HomeAssistant
net0: virtio=02:11:32:27:a3:6e,bridge=vmbr0,firewall=1
onboot: 1
ostype: other
sata0: local-lvm:vm-103-disk-1,size=32G
scsihw: virtio-scsi-pci
smbios1: uuid=8b26248e-5fb7-4424-9ead-08d4c10b1256
usb0: host=1a86:55d4,usb3=1
vmgenid: 03668eca-387f-472e-8493-9c78da58c5b1

root@littlebox:~# cat /etc/pve/storage.cfg
dir: local
        path /var/lib/vz
        content vztmpl,backup,iso

lvmthin: local-lvm
        thinpool data
        vgname pve
        content rootdir,images

lvmthin: storage
        thinpool storage
        vgname storage
        content rootdir,images
        nodes littlebox

dir: backupOne
        path /mnt/backup1
        content backup
        prune-backups keep-daily=1,keep-weekly=5
        shared 0

root@littlebox:~# pveversion -v
proxmox-ve: 8.0.1 (running kernel: 6.2.16-3-pve)
pve-manager: 8.0.3 (running version: 8.0.3/bbf3993334bfa916)
pve-kernel-6.2: 8.0.2
pve-kernel-6.2.16-3-pve: 6.2.16-3
ceph-fuse: 17.2.6-pve1+3
corosync: 3.1.7-pve3
criu: 3.17.1-2
glusterfs-client: 10.3-5
ifupdown2: 3.2.0-1+pmx2
ksm-control-daemon: 1.4-1
libjs-extjs: 7.0.0-3
libknet1: 1.25-pve1
libproxmox-acme-perl: 1.4.6
libproxmox-backup-qemu0: 1.4.0
libproxmox-rs-perl: 0.3.0
libpve-access-control: 8.0.3
libpve-apiclient-perl: 3.3.1
libpve-common-perl: 8.0.5
libpve-guest-common-perl: 5.0.3
libpve-http-server-perl: 5.0.3
libpve-rs-perl: 0.8.3
libpve-storage-perl: 8.0.1
libspice-server1: 0.15.1-1
lvm2: 2.03.16-2
lxc-pve: 5.0.2-4
lxcfs: 5.0.3-pve3
novnc-pve: 1.4.0-2
proxmox-backup-client: 2.99.0-1
proxmox-backup-file-restore: 2.99.0-1
proxmox-kernel-helper: 8.0.2
proxmox-mail-forward: 0.1.1-1
proxmox-mini-journalreader: 1.4.0
proxmox-widget-toolkit: 4.0.5
pve-cluster: 8.0.1
pve-container: 5.0.3
pve-docs: 8.0.3
pve-edk2-firmware: 3.20230228-4
pve-firewall: 5.0.2
pve-firmware: 3.7-1
pve-ha-manager: 4.0.2
pve-i18n: 3.0.4
pve-qemu-kvm: 8.0.2-3
pve-xtermjs: 4.16.0-3
qemu-server: 8.0.6
smartmontools: 7.3-pve1
spiceterm: 3.3.0
swtpm: 0.8.0+pve1
vncterm: 1.8.0
zfsutils-linux: 2.1.12-pve1

 

[JensF]

I would start with deactivating Secure Boot in the UEFI of the VM if it's activated.

 

[kjo81]

Sorry I didnt mentioned it: it's already disabled (it always was).

 

[JensF]

You checked it again? Some updates might activate it again.
To prevent this, just add a EFI-Disk without "pre-enrolled-keys".

 

[kjo81]

Yes, I even tried to enable it ahah, with no positive result of course

 

[leesteken]

It sounds like the problem was caused by updating something inside the VM (which might not be related to Proxmox). Did you try restoring (a copy of) the VM from a backup? Maybe something went wrong during the update inside the VM and running it again (make sure to check for error messages) might fix it?

 

[kjo81]

Thank you leesteken, you motivated me to try one more backup, and that's good now.

It's quite strange, because the only changes/updates done in HA these last days, was today.

So I tried last night backup, and last-last night backup, without success.
That's why I started to think the issue was somewhere in Proxmox.

Finally the backup from 18th december is good now. I have no explanations...
I can now remove the clone and a fresh install I was playing with to move my disk from one to another ahah.

Thank you !

 

[Timshel]

Just had a similar issue with HAOS; It appears the gpt table was corrupted.

You can check it with gdisk

> gdisk -l  /dev/mapper/pve-vm--200--disk--0

...

Found invalid MBR and corrupt GPT. What do you want to do? (Using the
GPT MAY permit recovery of GPT data.)
 1 - Use current GPT
 2 - Create blank GPT

Your answer: 1

Disk /dev/mapper/pve-vm--200--disk--0: 67108864 sectors, 32.0 GiB
Sector size (logical/physical): 512/512 bytes
Disk identifier (GUID): 08C3B230-4188-4F69-9979-43C03C32A4A8
Partition table holds up to 128 entries
Main partition table begins at sector 2 and ends at sector 33
First usable sector is 34, last usable sector is 67108830
Partitions will be aligned on 2048-sector boundaries
Total free space is 2014 sectors (1007.0 KiB)

Number  Start (sector)    End (sector)  Size       Code  Name
   1            2048           67583   32.0 MiB    EF00  hassos-boot
   2           67584          116735   24.0 MiB    8300  hassos-kernel0
   3          116736          641023   256.0 MiB   8300  hassos-system0
   4          641024          690175   24.0 MiB    8300  hassos-kernel1
   5          690176         1214463   256.0 MiB   8300  hassos-system1
   6         1214464         1230847   8.0 MiB     8300  hassos-bootstate
   7         1230848         1427455   96.0 MiB    8300  hassos-overlay
   8         1427456        67108830   31.3 GiB    8300  hassos-data

In my case the information appeared correct and can then be recreated.

Since I'm lazy opted for the easier solution of exporting the table from a correct disk using sfdisk:

> sfdisk -d /dev/zvol/stock/pve/vm-200-disk-4 > ha.gpt

And then rewrite the table :

> sfdisk /dev/mapper/pve-vm--200--disk--0  < ha.gpt

 

[leesteken]

i believe there was a problem with the first sector of the virtual disk becoming corrupted (maybe by doing a backup) on some specific storage configuration or cache setting. Sorry but I don't remember the details anymore, since that has been fixed for some time now.

EDIT: This was a reaction on a recent post in this thread that just vanished.