Snapshot backups going into suspend

[JustaGuy]

Although I have my backups set to use snapshot mode, they're going into suspend & becoming unavailable.

Mar 7 13:09:54 Bascule vzdump[22020]: INFO: Starting Backup of VM 107 (qemu)
Mar 7 13:09:54 Bascule qm[31361]: VM 107 suspend

Please advise.

Bascule:~# pveversion -v
pve-manager: 1.5-7 (pve-manager/1.5/4660)
running kernel: 2.6.32-1-pve
proxmox-ve-2.6.32: 1.5-4
pve-kernel-2.6.32-1-pve: 2.6.32-4
pve-kernel-2.6.18-2-pve: 2.6.18-5
qemu-server: 1.1-11
pve-firmware: 1.0-3
libpve-storage-perl: 1.0-10
vncterm: 0.9-2
vzctl: 3.0.23-1pve8
vzdump: 1.2-5
vzprocps: 2.0.11-1dso2
vzquota: 3.0.11-1
pve-qemu-kvm: 0.11.1-2
ksm-control-daemon: 1.0-3
Bascule:~#

-----EDIT-----

I just received an email detailing the backup jobs, which provides additional information.

106: Mar 07 12:27:39 INFO: Starting Backup of VM 106 (qemu)
106: Mar 07 12:27:39 INFO: running
106: Mar 07 12:27:39 INFO: status = running
106: Mar 07 12:27:39 INFO: mode failure - unable to detect lvm volume group
106: Mar 07 12:27:39 INFO: trying 'suspend' mode instead
106: Mar 07 12:27:39 INFO: backup mode: suspend
106: Mar 07 12:27:39 INFO: bandwidth limit: 10240 KB/s
106: Mar 07 12:27:39 INFO: suspend vm
106: Mar 07 12:27:39 INFO: creating archive '/datastore0/backup/vzdump-qemu-106-2010_03_07-12_27_39.tar'
106: Mar 07 12:27:39 INFO: adding '/datastore0/backup/vzdump-qemu-106-2010_03_07-12_27_39.tmp/qemu-server.conf' to archive ('qemu-server.conf')
106: Mar 07 12:27:39 INFO: adding '/datastore0/backup/images/106/vm-106-disk-1.raw' to archive ('vm-disk-ide0.raw')
106: Mar 07 13:09:53 INFO: Total bytes written: 26570226688 (10.00 MiB/s)
106: Mar 07 13:09:53 INFO: archive file size: 24.75GB
106: Mar 07 13:09:53 INFO: delete old backup '/datastore0/backup/vzdump-qemu-106-2010_03_06-10_54_24.tar'
106: Mar 07 13:09:53 INFO: resume vm
106: Mar 07 13:09:54 INFO: vm is online again after 2535 seconds
106: Mar 07 13:09:54 INFO: Finished Backup of VM 106 (00:42:15)

So it's this 'Unable to detect lvm volume group' causing troubles.

Bascule:~# vgdisplay
  --- Volume group ---
  VG Name               datastore1
  System ID
  Format                lvm2
  Metadata Areas        1
  Metadata Sequence No  57
  VG Access             read/write
  VG Status             resizable
  MAX LV                0
  Cur LV                4
  Open LV               2
  Max PV                0
  Cur PV                1
  Act PV                1
  VG Size               812.57 GB
  PE Size               4.00 MB
  Total PE              208017
  Alloc PE / Size       119040 / 465.00 GB
  Free  PE / Size       88977 / 347.57 GB
  VG UUID               1fqfwQ-emok-kztI-YyYs-PqBh-tWaA-m6M9zO

 

[JustaGuy]

I just noticed that 3 machines have their .raw disk images in the 'Directory' storage pool, rather than the 'LVM' pool where they belong.
Somehow they're in /Datastore0/backup/images. The 'Datastore0' space is designated as vzdump storage.
I think the backups they were restored from specified that they be placed to that path. I seem to remember my old 1.4 installation using that path for Virtual Disk storage.

I only know how to move VDs (from one VM to another) by using the instructions from this post, which makes use of the lvchange command.
And I can't do that in this case because these need to go from xfs to lvm.

When I browse the filesystem via sftp, I see the naming of the xfs-based VDs are quite different than the lvm-based VDs.
So I'm hesitant to simply drag and drop them to a new location.

What's the right way to resolve this?

 

[JustaGuy]

After looking through the forum some I found the following:

Here it says why the VD names are different:

(each disk is a logical volume).

Here it says how to move the xfs VDs to the lv:

In this structure each LV is a disk of a VM, so if you want to copy a disk in raw form into a LV,
of course you must first create such lv

lvcreate -n "VMID"-L "sizeinGigaByte"G pve-iscsi

then you can copy from raw file to LV

dd if="pathrawfile" of =/dev/pve-iscsi/"VMID" bs=1M

Except in my case I'll be substituting datastore1 for pve-iscsi.
Also, I found that this example shows not to use the quotes, and to add a space between the VMID and the -L that follows:

lvcreate -C y -n data -L 1T pve

Here it confirms how to put the contents of the .raw into a lv:

Hi,
in this case you use the second hdd as local lvm-strorage. Add in the VM a disk on this storage (same size as on the local storage).
With

lvdisplay

you will see the disk as a logical volume e.g.

/dev/second-hdd/vm-110-disk-1

If you use already the raw-format for the diskimage you can copy the image directly. If you use qcow2, you must convert it first to raw.
To copy a raw partition:

dd if=/var/lib/vz/images/110/vm-110-disk-1.raw of=/dev/second-hdd/vm-110-disk-1 bs=1024k

After that change the config, that ide0 (or virtio0) is now on the second storage - you have the line due to adding a second HD in the conffile - and delete the old entry.

However, it wasn't immediateley clear as to which config file needed to change until I looked & saw:
Here it says which config file defines to the VM the path of the corresponding VD:

/etc/qemu-server/110.cfg
ide0: local2:110/vm-110-disk-1.raw

Which is fine, except in that example their goal is the opposite of mine.
That example directs the VM to look in a local directory storage location for the .raw VD.
I need to direct the VM to look for a VD in the form of a lv.
So I look at the config file of a working VM whose VD is already in the proper location for an example of the correct syntax & see:

ide0: datastore1:vm-101-disk-1

Which makes sense because that's the path where I saw the VDs whose naming convention differed from the .raw VDs

 

[JustaGuy]

In summary, here are the steps I followed to resolve the issue.
So far each machine boots & I can log on from remote.

The only problem that remains is that the new disks are reported as 0.00 GB in their corresponding Hardware Device List, and as 0KB in the Virtual Machines list.

--Step 1:
Make the target Logical Volume based Virtual Disk:

lvcreate -n vm-102-disk-1 -L 62G datastore1

--Step 2:
Copy the source RAW Image based Virtual Disk to the target Logical Volume based Virtual Disk:

dd if=/datastore0/backup/images/102/vm-102-disk-1.raw of=/dev/datastore1/vm-102-disk-1 bs=1M

--Step 3:
Opened config file in text editor:

nano /etc/qemu-server/102.conf

Changed the line that was:

ide0: datastore0:102/vm-102-disk-1.raw

So that instead it is:

ide0: datastore1:vm-102-disk-1

--Step 4:
Make the target Logical Volume based Virtual Disk:

lvcreate -n vm-106-disk-1 -L 32G datastore1

--Step 5:
Copy the source RAW Image based Virtual Disk to the target Logical Volume based Virtual Disk:

dd if=/datastore0/backup/images/106/vm-106-disk-1.raw of=/dev/datastore1/vm-106-disk-1 bs=1M

--Step 6:
Opened config file in text editor:

nano /etc/qemu-server/106.conf

Changed the line that was:

ide0: datastore0:106/vm-106-disk-1.raw

So that instead it is:

ide0: datastore1:vm-106-disk-1

--Step 7:
Make the target Logical Volume based Virtual Disk:

lvcreate -n vm-107-disk-1 -L 40G datastore1

--Step 8:
Copy the source RAW Image based Virtual Disk to the target Logical Volume based Virtual Disk:

dd if=/datastore0/backup/images/107/vm-107-disk-1.raw of=/dev/datastore1/vm-107-disk-1 bs=1M

--Step 9:
Opened config file in text editor:

nano /etc/qemu-server/107.conf

Changed the line that was:

ide0: datastore0:107/vm-107-disk-1.raw

So that instead it is:

ide0: datastore1:vm-107-disk-1

 

[JustaGuy]

I saw here that to re-sync disk sizes it was advised to remove the disk via the Web Interface's Hardware tab & simply re-add it.

What happened was that it disappeared entirely, rather than showing up as an un-allocated disk still belonging that machine.

Something like this happened when I once moved a VD from one VM to another as per instructions in this post, so I took a look at that for a refresher & remembered it had to do with something called tags.

To see the Logical Volumes' tags I did:

lvs -o +lv_tags

and confirmed the disks in question didn't have tags.

To give them tags that matched the others I did:

lvchange --addtag @pve-vm-102 datastore1/vm-102-disk-1

lvchange --addtag @pve-vm-106 datastore1/vm-106-disk-1

lvchange --addtag @pve-vm-107 datastore1/vm-107-disk-1

And was able to re-add the disk I just removed, and when I did it's size was properly displayed.

Then I removed & re-added the remaining 2 from the other machines and now all is well.