About a week ago I did my usual 'apt-get update && apt-get dist-upgrade' on one of my Proxmox machines and rebooted.
Now the nightly backup is suspending the VM rather than doing a snapshot.
Other than installing the updates, the box hasn't seen changes in several months.
I missed the problem in the logs the first night, so I upgraded the rest of the servers in my cluster. Now they are suffering from the same issue.
All the virtual machines use iscsi for storage.
Version info and backup logs:
kvm3:~# pveversion -v
pve-manager: 1.9-26 (pve-manager/1.9/6567)
running kernel: 2.6.32-6-pve
proxmox-ve-2.6.32: 1.9-50
pve-kernel-2.6.32-4-pve: 2.6.32-33
pve-kernel-2.6.32-6-pve: 2.6.32-50
qemu-server: 1.1-32
pve-firmware: 1.0-14
libpve-storage-perl: 1.0-19
vncterm: 0.9-2
vzctl: 3.0.29-3pve1
vzdump: 1.2-16
vzprocps: 2.0.11-2
vzquota: 3.0.11-1
pve-qemu-kvm: 0.15.0-1
ksm-control-daemon: 1.0-6
kvm3:~#
kvm3:~# uname -a
Linux kvm3 2.6.32-6-pve #1 SMP Fri Nov 4 06:54:05 CET 2011 x86_64 GNU/Linux
kvm3:~#
vzdump --quiet --node 6 --snapshot --compress --storage virtsnaps --maxfiles 3 --mailto email-address-redacted@example.tld 106
106: Nov 30 02:30:02 INFO: Starting Backup of VM 106 (qemu)
106: Nov 30 02:30:02 INFO: running
106: Nov 30 02:30:02 INFO: status = running
106: Nov 30 02:30:02 INFO: mode failure - unable to detect lvm volume group
106: Nov 30 02:30:02 INFO: trying 'suspend' mode instead
106: Nov 30 02:30:02 INFO: backup mode: suspend
106: Nov 30 02:30:02 INFO: ionice priority: 7
106: Nov 30 02:30:02 INFO: suspend vm
106: Nov 30 02:30:03 INFO: creating archive '/mnt/pve/virtsnaps/vzdump-qemu-106-2011_11_30-02_30_02.tgz'
106: Nov 30 02:30:03 INFO: adding '/mnt/pve/virtsnaps/vzdump-qemu-106-2011_11_30-02_30_02.tmp/qemu-server.conf' to archive ('qemu-server.conf')
106: Nov 30 02:30:03 INFO: adding '/dev/disk/by-id/scsi-14945540000000000bc92943370c0328e590cad7cef84c294' to archive ('vm-disk-virtio0.raw')
106: Nov 30 02:52:25 INFO: Total bytes written: 13618599424 (9.68 MiB/s)
106: Nov 30 02:52:26 INFO: archive file size: 6.06GB
106: Nov 30 02:52:27 INFO: resume vm
106: Nov 30 02:52:27 INFO: vm is online again after 1345 seconds
106: Nov 30 02:52:27 INFO: Finished Backup of VM 106 (00:22:25)
[edit: pasted log was formatted poorly]
Now the nightly backup is suspending the VM rather than doing a snapshot.
Other than installing the updates, the box hasn't seen changes in several months.
I missed the problem in the logs the first night, so I upgraded the rest of the servers in my cluster. Now they are suffering from the same issue.
All the virtual machines use iscsi for storage.
Version info and backup logs:
kvm3:~# pveversion -v
pve-manager: 1.9-26 (pve-manager/1.9/6567)
running kernel: 2.6.32-6-pve
proxmox-ve-2.6.32: 1.9-50
pve-kernel-2.6.32-4-pve: 2.6.32-33
pve-kernel-2.6.32-6-pve: 2.6.32-50
qemu-server: 1.1-32
pve-firmware: 1.0-14
libpve-storage-perl: 1.0-19
vncterm: 0.9-2
vzctl: 3.0.29-3pve1
vzdump: 1.2-16
vzprocps: 2.0.11-2
vzquota: 3.0.11-1
pve-qemu-kvm: 0.15.0-1
ksm-control-daemon: 1.0-6
kvm3:~#
kvm3:~# uname -a
Linux kvm3 2.6.32-6-pve #1 SMP Fri Nov 4 06:54:05 CET 2011 x86_64 GNU/Linux
kvm3:~#
vzdump --quiet --node 6 --snapshot --compress --storage virtsnaps --maxfiles 3 --mailto email-address-redacted@example.tld 106
106: Nov 30 02:30:02 INFO: Starting Backup of VM 106 (qemu)
106: Nov 30 02:30:02 INFO: running
106: Nov 30 02:30:02 INFO: status = running
106: Nov 30 02:30:02 INFO: mode failure - unable to detect lvm volume group
106: Nov 30 02:30:02 INFO: trying 'suspend' mode instead
106: Nov 30 02:30:02 INFO: backup mode: suspend
106: Nov 30 02:30:02 INFO: ionice priority: 7
106: Nov 30 02:30:02 INFO: suspend vm
106: Nov 30 02:30:03 INFO: creating archive '/mnt/pve/virtsnaps/vzdump-qemu-106-2011_11_30-02_30_02.tgz'
106: Nov 30 02:30:03 INFO: adding '/mnt/pve/virtsnaps/vzdump-qemu-106-2011_11_30-02_30_02.tmp/qemu-server.conf' to archive ('qemu-server.conf')
106: Nov 30 02:30:03 INFO: adding '/dev/disk/by-id/scsi-14945540000000000bc92943370c0328e590cad7cef84c294' to archive ('vm-disk-virtio0.raw')
106: Nov 30 02:52:25 INFO: Total bytes written: 13618599424 (9.68 MiB/s)
106: Nov 30 02:52:26 INFO: archive file size: 6.06GB
106: Nov 30 02:52:27 INFO: resume vm
106: Nov 30 02:52:27 INFO: vm is online again after 1345 seconds
106: Nov 30 02:52:27 INFO: Finished Backup of VM 106 (00:22:25)
[edit: pasted log was formatted poorly]