trouble backing up LVM based KVM guest, known bug?

B

bleomycin

Renowned Member
Mar 20, 2011
49
7
73
I'm just curious if this is a known bug with the latest release? I've performed about 10 vgdump --snapshot and qmrestores today testing backup/restore functionality for LVM based linux KVM guests and about 30% of the time the restored guest will fail to boot citing no bootable device even though the logical volume was re-created. Console output between a failed and successful backup/restore is identical so nothing appears to be going wrong. Is anyone else experiencing this? I REALLY want to get started using proxmox + KVM i've spent probably 20 hours the last 2 days trying to figure out this issue to no avail and its driving me insane. Without a rock solid snapshot backup solution there's no way i can proceed and i don't understand how anyone else does? Thanks for any help!
Code: 
vzdump --dumpdir /mnt/backup/ --snapshot 102
deleted the machine from proxmox, and performed a restore:

Code: 
qmrestore /mnt/backup/vzdump-qemu-102-2011_03_19-23_38_58.tar 102
INFO: restore QemuServer backup '/mnt/backup/vzdump-qemu-102-2011_03_19-23_38_58.tar' using ID 102
INFO: extracting 'qemu-server.conf' from archive
INFO: extracting 'vm-disk-virtio0.raw' from archive
INFO:   Rounding up size to full physical extent 15.00 GB
INFO:   Logical volume "vm-102-disk-1" created
INFO: new volume ID is 'storage:vm-102-disk-1'
INFO: restore data to '/dev/storage/vm-102-disk-1' (16106127360 bytes)
INFO: 19+564713 records in
INFO: 19+564713 records out
INFO: 16106127360 bytes (16 GB) copied, 608.375 s, 26.5 MB/s
INFO: restore QemuServer backup  '/mnt/backup/vzdump-qemu-102-2011_03_19-23_38_58.tar'  successful
lvdisplay shows the logical volume was recreated after the restore:

Code: 
--- Logical volume ---
  LV Name                /dev/storage/vm-102-disk-1
  VG Name                storage
  LV UUID                T0tVOl-e6J5-Kz1B-J8CQ-Fkf8-QbVl-KYquZW
  LV Write Access        read/write
  LV Status              available
  # open                 0
  LV Size                15.00 GB
  Current LE             3841
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           254:3
However the vm boots to the same "No bootable device."

Any help is greatly appreciated!
 
post the output of 'pveversion -v'

what kind of storage system do you use for LVM?
 
Are you sure is related to latest version of Proxmox? Don't know the details of your tests, but this Friday I had a very frightening moment when I restored a XP on LVM storage... I had the same "no bootable device" error.
Fortunately in that VM the important thing was the installation itself, since all data are stored elsewhere, and I had a previous backup which restore went fine.
But the .log of both backup were OK, and I don't want to imagine if I had only the last backup available.

The backup is done through proxmox contab backup itself with something like:
vzdump --quiet --snapshot --compress --storage backup_vm
(backup_vm is a 1TB sata devoted to backups, the VM was on SAS LVM)

$ pveversion -v
pve-manager: 1.7-10 (pve-manager/1.7/5323)
running kernel: 2.6.32-4-pve
proxmox-ve-2.6.32: 1.7-30
pve-kernel-2.6.32-4-pve: 2.6.32-30
pve-kernel-2.6.18-2-pve: 2.6.18-5
qemu-server: 1.1-25
pve-firmware: 1.0-10
libpve-storage-perl: 1.0-16
vncterm: 0.9-2
vzctl: 3.0.24-1pve4
vzdump: 1.2-10
vzprocps: 2.0.11-1dso2
vzquota: 3.0.11-1
pve-qemu-kvm: 0.13.0-3
ksm-control-daemon: 1.0-4
 
Thanks for the replies, the LVM storage is a secondary local disk, the backups are being stored and restored from a remotely mounted server. I have checked all of the obvious settings countless times related to boot device priority/settings in the configss and in proxmox after a restore and everything looks fine.

Code: 
proxmox:/etc/qemu-server# pveversion -v
pve-manager: 1.7-10 (pve-manager/1.7/5323)
running kernel: 2.6.32-4-pve
proxmox-ve-2.6.32: 1.7-28
pve-kernel-2.6.32-4-pve: 2.6.32-28
qemu-server: 1.1-25
pve-firmware: 1.0-9
libpve-storage-perl: 1.0-16
vncterm: 0.9-2
vzctl: 3.0.24-1pve4
vzdump: 1.2-9
vzprocps: 2.0.11-1dso2
vzquota: 3.0.11-1
pve-qemu-kvm: 0.13.0-2
ksm-control-daemon: 1.0-4
Physical Volume:

Code: 
proxmox:/etc/qemu-server# pvdisplay
  --- Physical volume ---
  PV Name               /dev/sdb1
  VG Name               storage
  PV Size               465.76 GB / not usable 1.50 MB
  Allocatable           yes
  PE Size (KByte)       4096
  Total PE              119234
  Free PE               112827
  Allocated PE          6407
  PV UUID               PiUVHB-TD7p-A011-a9qm-IMUe-vJ4b-PjBLv1
VM logical volume:

Code: 
--- Logical volume ---
  LV Name                /dev/storage/vm-102-disk-1
  VG Name                storage
  LV UUID                45Dkn0-ozqs-h2MB-LEVg-eCQO-UGkX-YiUgG3
  LV Write Access        read/write
  LV Status              available
  # open                 1
  LV Size                10.03 GB
  Current LE             2567
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           254:4

Please let me know if there is any more helpful debug information i can provide, thank you!
 
I have to correct myself. My issue seems not due to backup/restore problems, since I've discovered that my co-worked asked me to restore the VM since rebooting he had the VM (XP) con continuously crash at startup. I'm wondering if could be corruption due to the Virtio drivers used for disk access, or just just one more bug from M$
 
bleomycin said:
Code: 
proxmox:/etc/qemu-server# pveversion -v
pve-manager: 1.7-10 (pve-manager/1.7/5323)
running kernel: 2.6.32-4-pve
proxmox-ve-2.6.32: 1.7-28
pve-kernel-2.6.32-4-pve: 2.6.32-28
qemu-server: 1.1-25
pve-firmware: 1.0-9
libpve-storage-perl: 1.0-16
vncterm: 0.9-2
vzctl: 3.0.24-1pve4
vzdump: 1.2-9
vzprocps: 2.0.11-1dso2
vzquota: 3.0.11-1
pve-qemu-kvm: 0.13.0-2
ksm-control-daemon: 1.0-4
Click to expand...

looks like you never updated the 1.7 iso install so I suggest you update to the latest stable (aptitude update && aptitude full-upgrade) and test again.

we did several restore test but til now we do not see this issue in our lab (but we test with the latest packages).

btw, as we plan to release 1.8 soon you can also think of using the pvetest repository, containing already 1.8.
 
Hi,
Is there any info page about 1.8 features.
Here we plan to migrate soon from Xen to KVM (proxmox) and we would like know what features can provide this new 1.8 version.

thks
 
Thank you for the reply's, i went ahead and switched to the pvetesting repo and did a dist-upgrade but the problem persists.

Code: 
proxmox:/var/lib/vz/template/iso# pveversion -v
pve-manager: 1.8-13 (pve-manager/1.8/5696)
running kernel: 2.6.32-4-pve
proxmox-ve-2.6.32: 1.8-31
pve-kernel-2.6.32-4-pve: 2.6.32-31
qemu-server: 1.1-30
pve-firmware: 1.0-10
libpve-storage-perl: 1.0-16
vncterm: 0.9-2
vzctl: 3.0.24-1pve4
vzdump: 1.2-11
vzprocps: 2.0.11-1dso2
vzquota: 3.0.11-1
pve-qemu-kvm: 0.14.0-2
ksm-control-daemon: 1.0-5

I just installed a fresh debian squeeze 64bit guest with ext4 onto the lvm, snapshotted and restored no problem. Installed a few small packages, snapshot/restore again no issue. On the 4th snapshot/restore it happened again "Boot failed: not a bootable disk"

Is there any additional information i can provide to help? The exact install settings used for the guest are: lvm storage, virtio bus/ethernet, debian squeeze base install no gui, ext4 / + swap.
 
bleomycin said:
Thank you for the reply's, i went ahead and switched to the pvetesting repo and did a dist-upgrade but the problem persists.

Code: 
proxmox:/var/lib/vz/template/iso# pveversion -v
pve-manager: 1.8-13 (pve-manager/1.8/5696)
running kernel: 2.6.32-4-pve
proxmox-ve-2.6.32: 1.8-31
pve-kernel-2.6.32-4-pve: 2.6.32-31
qemu-server: 1.1-30
pve-firmware: 1.0-10
libpve-storage-perl: 1.0-16
vncterm: 0.9-2
vzctl: 3.0.24-1pve4
vzdump: 1.2-11
vzprocps: 2.0.11-1dso2
vzquota: 3.0.11-1
pve-qemu-kvm: 0.14.0-2
ksm-control-daemon: 1.0-5
I just installed a fresh debian squeeze 64bit guest with ext4 onto the lvm, snapshotted and restored no problem. Installed a few small packages, snapshot/restore again no issue. On the 4th snapshot/restore it happened again "Boot failed: not a bootable disk"

Is there any additional information i can provide to help? The exact install settings used for the guest are: lvm storage, virtio bus/ethernet, debian squeeze base install no gui, ext4 / + swap.
Click to expand...
Hi,
i'm right that your backup-space is via nfs? Can you try with local backup-space?

Udo
 
Actually, the latest failure was using local space for the backup. I'm beginning to think this is a limitation of LVM snapshot itself and that some critical filesystem data is being overlooked/corrupted during the snapshot. Unless i'm wrong i think i'll just go ahead and stick with vzdump --stop to make sure everything is properly flushed and just treat the VM guests as if they were physical machines and plan my backups accordingly. Thank you all for the help.
 
udo said:
Hi,
i'm right that your backup-space is via nfs? Can you try with local backup-space?

Udo
Click to expand...
I'm beginning to think this is a limitation of LVM snapshot itself and that some critical filesystem data is being overlooked/corrupted during the snapshot. Unless i'm wrong i think i'll just go ahead and stick with vzdump --stop to make sure everything is properly flushed and just treat the VM guests as if they were physical machines and plan my backups accordingly. Thank you all for the help
 
You must log in or register to reply here.
Top Bottom