[SOLVED] Corrupt GPT after Restore on PVE 7.0 from Backup on PVE 6.4 | QEMU VM, LVM-Thin Storage, OVMF

quisby

New Member
Aug 19, 2021
3
0
1
Hello everyone

I tested whether I could migrate VMs from PVE 6.4-13 to PVE 7.0-11, using backup and restore, which unfortunately fails. Both PVEs share an equal setup except for their Proxmox version, have an LVM thin pooled storage, were installed from Debian (Buster, Bullseye, respectively).

Independent from the backup archiving method and the guest OS, a restored machine on PVE 7 fails to boot as the disk GPT is corrupt.
Code:
ubuntu@ubuntu:~$  sudo gdisk /dev/sda
GPT fdisk (gdisk) version 1.0.5

Caution! After loading partitions, the CRC doesn't check out!
Warning! Main and backup partition tables differ! Use the 'c' and 'e' options
on the recovery & transformation menu to examine the two tables.

Warning! One or more CRCs don't match. You should repair the disk!
Main header: OK
Backup header: OK
Main partition table: ERROR
Backup partition table: ERROR

Partition table scan:
  MBR: protective
  BSD: not present
  APM: not present
  GPT: damaged

****************************************************************************
Caution: Found protective or hybrid MBR and corrupt GPT. Using GPT, but disk
verification and recovery are STRONGLY recommended.
****************************************************************************

Warning! Secondary partition table overlaps the last partition by
18315034227491254276 blocks!
You will need to delete this partition or resize it in another utility.

Along the PVE backup and using gdisk, I tried to backup and restore the GPT, which let OVMF pick up the disk again. However, e.g. in the case of a Windows guest, the Windows boot manager raises digital signature errors for boot files, error code 0xc0000428, which the Windows Startup-Repair isn't able to fix.

Browsing through the files on disks with recovered GPT from a live OS, it seems that at least the files are in place correctly and not corrupted.

Does anyone have similar experiences, what else can I try, what could be the reason for the stated problems on my migration with backup and restore?

Many thanks, any help appreciated!
Best regards!



PVE 6.4-13

pveversion -v
Code:
proxmox-ve: 6.4-1 (running kernel: 5.4.128-1-pve)
pve-manager: 6.4-13 (running version: 6.4-13/9f411e79)
pve-kernel-5.4: 6.4-5
pve-kernel-helper: 6.4-5
pve-kernel-5.4.128-1-pve: 5.4.128-1
pve-kernel-5.4.124-1-pve: 5.4.124-2
ceph-fuse: 12.2.11+dfsg1-2.1+b1
corosync: 3.1.2-pve1
criu: 3.11-3
glusterfs-client: 5.5-3
ifupdown: residual config
ifupdown2: 3.0.0-1+pve4~bpo10
libjs-extjs: 6.0.1-10
libknet1: 1.20-pve1
libproxmox-acme-perl: 1.1.0
libproxmox-backup-qemu0: 1.1.0-1
libpve-access-control: 6.4-3
libpve-apiclient-perl: 3.1-3
libpve-common-perl: 6.4-3
libpve-guest-common-perl: 3.1-5
libpve-http-server-perl: 3.2-3
libpve-storage-perl: 6.4-1
libqb0: 1.0.5-1
libspice-server1: 0.14.2-4~pve6+1
lvm2: 2.03.02-pve4
lxc-pve: 4.0.6-2
lxcfs: 4.0.6-pve1
novnc-pve: 1.1.0-1
proxmox-backup-client: 1.1.12-1
proxmox-mini-journalreader: 1.1-1
proxmox-widget-toolkit: 2.6-1
pve-cluster: 6.4-1
pve-container: 3.3-6
pve-docs: 6.4-2
pve-edk2-firmware: 2.20200531-1
pve-firewall: 4.1-4
pve-firmware: 3.2-4
pve-ha-manager: 3.1-1
pve-i18n: 2.3-1
pve-qemu-kvm: 5.2.0-6
pve-xtermjs: 4.7.0-3
qemu-server: 6.4-2
smartmontools: 7.2-pve2
spiceterm: 3.1-1
vncterm: 1.6-2
zfsutils-linux: 2.0.5-pve1~bpo10+1



PVE 7.0-11

pveversion -v
Code:
proxmox-ve: 7.0-2 (running kernel: 5.11.22-3-pve)
pve-manager: 7.0-11 (running version: 7.0-11/63d82f4e)
pve-kernel-5.11: 7.0-6
pve-kernel-helper: 7.0-6
pve-kernel-5.11.22-3-pve: 5.11.22-6
ceph-fuse: 14.2.21-1
corosync: 3.1.2-pve2
criu: 3.15-1+pve-1
glusterfs-client: 9.2-1
ifupdown: residual config
ifupdown2: 3.1.0-1+pmx3
libjs-extjs: 7.0.0-1
libknet1: 1.21-pve1
libproxmox-acme-perl: 1.2.0
libproxmox-backup-qemu0: 1.2.0-1
libpve-access-control: 7.0-4
libpve-apiclient-perl: 3.2-1
libpve-common-perl: 7.0-5
libpve-guest-common-perl: 4.0-2
libpve-http-server-perl: 4.0-2
libpve-storage-perl: 7.0-10
libspice-server1: 0.14.3-2.1
lvm2: 2.03.11-2.1
lxc-pve: 4.0.9-4
lxcfs: 4.0.8-pve2
novnc-pve: 1.2.0-3
proxmox-backup-client: 2.0.8-1
proxmox-backup-file-restore: 2.0.8-1
proxmox-mini-journalreader: 1.2-1
proxmox-widget-toolkit: 3.3-6
pve-cluster: 7.0-3
pve-container: 4.0-9
pve-docs: 7.0-5
pve-edk2-firmware: 3.20200531-1
pve-firewall: 4.2-2
pve-firmware: 3.2-4
pve-ha-manager: 3.3-1
pve-i18n: 2.4-1
pve-qemu-kvm: 6.0.0-3
pve-xtermjs: 4.12.0-1
qemu-server: 7.0-13
smartmontools: 7.2-1
spiceterm: 3.2-2
vncterm: 1.7-1
zfsutils-linux: 2.0.5-pve1



Example VM

cat qemu-server/502.conf
Code:
bios: ovmf
boot: order=ide0;ide2;net0
cores: 4
efidisk0: local-lvm:vm-502-disk-1,size=4M
ide0: local-lvm:vm-502-disk-0,size=80G
ide2: local:iso/Win10_21H1_English_x64.iso,media=cdrom
machine: pc-i440fx-5.2
memory: 8192
name: windows2
net0: e1000=16:5B:9D:6A:BA:D4,bridge=vmbr0,firewall=1,tag=2
numa: 0
ostype: win10
scsihw: virtio-scsi-pci
smbios1: uuid=6ff04502-35e1-4b34-aeec-d433de01040e
snaptime: 1627145696
sockets: 1
vga: qxl
vmgenid: b426ca9d-44be-4973-acda-7f2ee82e83d9
 
Last edited:
Something seems to be wrong with my specific installation of PVE 7. The backups coming from the PVE 6 host are working on another installation of PVE 7.

On the problematic host, even cloning machines results in corrupt GPT data and failing booting attempts.
 
The troublemaker has been found: Due to a hint when creating the thinpool, I added flag -Zn so that the first 4k of volumes on their creation are not set to zero. I wanted to do the SSD drives and possibly PVE a favor. However, as in this case, this will not work.

The thinpool is apparently expected to be created without this flag:

Code:
lvcreate -l 100%FREE -T -n data pve



Just FYI

man entry for flag -Z:

Code:
-Z|--zero y|n
       Controls zeroing of the first 4KiB of data in the new LV.  Default is y.  Snapshot COW
       volumes are always zeroed.  For thin pools, this controls zeroing of provisioned
       blocks.  LV is not zeroed if the read only flag is set.  Warning: trying to mount an
       unzeroed LV can cause the system to hang.
 
Last edited:

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!