BACKUPS nor CLONES do not restore properly and won't boot

DAE51D

New Member
Oct 9, 2024
8
1
3
Seattle, WA
For the past two days I've been struggling to get a few VMs restored from Proxmox backups (the default built in one in VE 8.3.1 community).

They all seem to backup "properly" and restore "properly" in that there are no errors in either case. However the VMs just won't boot !

One is my Home Assistant and another just for sanity testing is a Windows 11 22H2 (logs below)

Code:
INFO: starting new backup job: vzdump 108 --node proxmox --compress zstd --notes-template '{{guestname}}' --notification-mode auto --remove 0 --mode stop --storage local
INFO: Starting Backup of VM 108 (qemu)
INFO: Backup started at 2024-12-14 20:35:19
INFO: status = stopped
INFO: backup mode: stop
INFO: ionice priority: 7
INFO: VM Name: win24h2
INFO: include disk 'scsi0' 'local-nvme:vm-108-disk-1' 100G
INFO: include disk 'efidisk0' 'local-nvme:vm-108-disk-0' 4M
INFO: include disk 'tpmstate0' 'local-nvme:vm-108-disk-2' 4M
INFO: creating vzdump archive '/var/lib/vz/dump/vzdump-qemu-108-2024_12_14-20_35_19.vma.zst'
INFO: starting kvm to execute backup task
swtpm_setup: Not overwriting existing state file.
INFO: attaching TPM drive to QEMU for backup
INFO: started backup task '3fd21898-9433-4b9c-8534-1b0e1ce095f4'
INFO:   0% (532.9 MiB of 100.0 GiB) in 3s, read: 177.6 MiB/s, write: 145.0 MiB/s
INFO:   1% (1.2 GiB of 100.0 GiB) in 6s, read: 227.2 MiB/s, write: 216.1 MiB/s
INFO:   2% (2.2 GiB of 100.0 GiB) in 10s, read: 261.9 MiB/s, write: 258.9 MiB/s
...
...
INFO:  91% (91.3 GiB of 100.0 GiB) in 2m 32s, read: 1.0 GiB/s, write: 0 B/s
INFO:  94% (94.3 GiB of 100.0 GiB) in 2m 35s, read: 1.0 GiB/s, write: 0 B/s
INFO:  97% (97.4 GiB of 100.0 GiB) in 2m 38s, read: 1.0 GiB/s, write: 0 B/s
INFO:  99% (99.7 GiB of 100.0 GiB) in 2m 41s, read: 801.0 MiB/s, write: 74.0 MiB/s
INFO: 100% (100.0 GiB of 100.0 GiB) in 2m 42s, read: 302.8 MiB/s, write: 300.7 MiB/s
INFO: backup is sparse: 81.89 GiB (81%) total zero data
INFO: transferred 100.00 GiB in 162 seconds (632.1 MiB/s)
INFO: stopping kvm after backup task
INFO: archive file size: 18.08GB
INFO: adding notes to backup
INFO: Finished Backup of VM 108 (00:02:48)
INFO: Backup finished at 2024-12-14 20:38:07
INFO: Backup job finished successfully
TASK OK

Then immediately restore the same Win11 backup to another ID
Code:
restore vma archive: zstd -q -d -c /var/lib/vz/dump/vzdump-qemu-108-2024_12_14-20_35_19.vma.zst | vma extract -v -r /var/tmp/vzdumptmp497286.fifo - /var/tmp/vzdumptmp497286
CFG: size: 974 name: qemu-server.conf
DEV: dev_id=1 size: 540672 devname: drive-efidisk0
DEV: dev_id=2 size: 107374182400 devname: drive-scsi0
DEV: dev_id=3 size: 4194304 devname: drive-tpmstate0-backup
CTIME: Sat Dec 14 20:35:23 2024
  Rounding up size to full physical extent 4.00 MiB
  Logical volume "vm-115-disk-0" created.
new volume ID is 'local-nvme:vm-115-disk-0'
  Logical volume "vm-115-disk-1" created.
new volume ID is 'local-nvme:vm-115-disk-1'
  Logical volume "vm-115-disk-2" created.
new volume ID is 'local-nvme:vm-115-disk-2'
map 'drive-efidisk0' to '/dev/pve/vm-115-disk-0' (write zeros = 0)
map 'drive-scsi0' to '/dev/pve/vm-115-disk-1' (write zeros = 0)
map 'drive-tpmstate0-backup' to '/dev/pve/vm-115-disk-2' (write zeros = 0)
progress 1% (read 1073807360 bytes, duration 3 sec)
progress 2% (read 2147614720 bytes, duration 6 sec)
progress 3% (read 3221422080 bytes, duration 10 sec)
...
progress 99% (read 106305159168 bytes, duration 85 sec)
progress 100% (read 107378900992 bytes, duration 87 sec)
total bytes read 107378966528, sparse bytes 87926505472 (81.9%)
space reduction due to 4K zero blocks 0.00876%
rescan volumes...
TASK OK

I've tried to backup the Win11 one to three different storage locations in case that was a factor, but it makes no difference. The restore never works.
my normal local-backup one is BTRFS which I know is experimental so I thought maybe that was it. So then tried to the Synology NAS over CIFS. And lastly I just added "VZDump" to local just to see if that mattered. So as not to destroy the actual working VM, I am restoring to another ID (but that should be fine right?)

1734237647509.png

This is the Win11 error I get...
1734237967150.png

And often see this kind for the Home Assistant one (but have seen for other's too)

1734238091631.png
I check the BIOS and this "looks" right to me?
1734238787572.png

Please help! My Home Assistant STB a few days ago and I can not get any restores up and working so my smart-home is in disarray... But it also freaks me out that I had a false sense of security and have been having regular nightly backups for several months now and I fear that they're all borked and useless if I can't get this sorted out!

Code:
# pveversion -v

proxmox-ve: 8.3.0 (running kernel: 6.8.12-5-pve)
pve-manager: 8.3.1 (running version: 8.3.1/fb48e850ef9dde27)
proxmox-kernel-helper: 8.1.0
proxmox-kernel-6.8: 6.8.12-5
proxmox-kernel-6.8.12-5-pve-signed: 6.8.12-5
proxmox-kernel-6.8.12-4-pve-signed: 6.8.12-4
proxmox-kernel-6.8.4-2-pve-signed: 6.8.4-2
ceph-fuse: 17.2.7-pve3
corosync: 3.1.7-pve3
criu: 3.17.1-2
glusterfs-client: 10.3-5
ifupdown2: 3.2.0-1+pmx11
intel-microcode: 3.20241112.1
ksm-control-daemon: 1.5-1
libjs-extjs: 7.0.0-5
libknet1: 1.28-pve1
libproxmox-acme-perl: 1.5.1
libproxmox-backup-qemu0: 1.4.1
libproxmox-rs-perl: 0.3.4
libpve-access-control: 8.2.0
libpve-apiclient-perl: 3.3.2
libpve-cluster-api-perl: 8.0.10
libpve-cluster-perl: 8.0.10
libpve-common-perl: 8.2.9
libpve-guest-common-perl: 5.1.6
libpve-http-server-perl: 5.1.2
libpve-network-perl: 0.10.0
libpve-rs-perl: 0.9.1
libpve-storage-perl: 8.3.2
libspice-server1: 0.15.1-1
lvm2: 2.03.16-2
lxc-pve: 6.0.0-1
lxcfs: 6.0.0-pve2
novnc-pve: 1.5.0-1
proxmox-backup-client: 3.3.2-1
proxmox-backup-file-restore: 3.3.2-2
proxmox-firewall: 0.6.0
proxmox-kernel-helper: 8.1.0
proxmox-mail-forward: 0.3.1
proxmox-mini-journalreader: 1.4.0
proxmox-offline-mirror-helper: 0.6.7
proxmox-widget-toolkit: 4.3.3
pve-cluster: 8.0.10
pve-container: 5.2.2
pve-docs: 8.3.1
pve-edk2-firmware: 4.2023.08-4
pve-esxi-import-tools: 0.7.2
pve-firewall: 5.1.0
pve-firmware: 3.14-2
pve-ha-manager: 4.0.6
pve-i18n: 3.3.2
pve-qemu-kvm: 9.0.2-4
pve-xtermjs: 5.3.0-3
qemu-server: 8.3.2
smartmontools: 7.3-pve1
spiceterm: 3.3.0
swtpm: 0.8.0+pve1
vncterm: 1.8.0
zfsutils-linux: 2.2.6-pve1
 
Last edited:
Last edited:
For win11 security violation, try turning off Secure Boot in VMBIOS and change backup method from snapshot to "suspend" - then TEST THE RESTORE
The original VM has Secure Boot ENABLED though (this is Windows 11 and AFAIK it's "required" (sans a Rufus patched install, which this is not)

But just for fun, I did try to disable it in the restored VMs BIOS and now I get that other error I showed above
1734298449038.png

As for the BACKUP I have the job set to "stop" for extra safety. AND furthermore for this Win 11 backup test, I definitely have the VM shutdown (clean) before doing the backup.

I don't think these are the solution though? This is not an issue with the OS (Win11 or otherwise). The Proxmox BIOS won't find/boot the drives that it just restored. The virtual disks (all three including TPM and EFI) show as part of the "Hardware" section of the VM and I verified the scsi0 in the Options > Boot Order
1734298626127.png
1734298671465.png
1734299776408.png
Also note it says this can't be found???
1734299922812.png
And comparing to the original VM, these are different right?
1734300555001.png
 
Last edited:
Okay. something is DEFINITELY broken with Proxmox 8.3.1 (is my guess, "it's not me, it's you")

I just simply CLONED the VM and the clone will NOT BOOT (same exact issue as the restore does).

1734301000930.png

Can "someone" (or multiple people) test this on your Proxmox server?
  1. Perform a clone of a VM and see if the clone boots normally
    (ideally Windows11 just to be apples/apples, but I think this is not OS related, but BIOS and is a bug in the clone)
  2. Perform a backup of a stopped VM and restore it to another ID
    (again, don NOT risk hosing up your working one by deleting and restoring to the same ID)
And also verify you're using the same version "8.3.1" AND LIBRARY VERSIONS as listed above?

To add more data here. I then tried to clone a "SeaBIOS" Kali distro and that DID work fine. So perhaps it's related to UEFI only?
1734305559074.png
 
Last edited:

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!