[SOLVED] Backup and migration imposible (Proxmox 4)

[Foxhound77]

Hello,

I have a big problem with my 3 nodes cluster (DC01, DC02 and DC03). Each node is in version 4 with a ceph rbd.
Everything works fine however two day ago I cannot run, migrate (live or HA) or backup any vm in DC02 and DC03.

In DC01 I can run and backup my VM (local and ceph storage)
In DC02 and DC03 I cannot do that I receive this message when I migrate a VM:

janv. 28 20:16:41 starting migration of VM 108 to node 'DC02' (10.0.0.2)
janv. 28 20:16:41 copying disk images
janv. 28 20:16:41 starting VM 108 on remote node 'DC02'
janv. 28 20:16:43 start failed: command '/usr/bin/systemd-run --scope --slice qemu --unit 108 -p 'KillMode=none' -p 'CPUShares=1000' /usr/bin/kvm -id 108 -chardev 'socket,id=qmp,path=/var/run/qemu-server/108.qmp,server,nowait' -mon 'chardev=qmp,mode=control' -vnc unix:/var/run/qemu-server/108.vnc,x509,password -pidfile /var/run/qemu-server/108.pid -daemonize -smbios 'type=1,uuid=f52b5d2c-2009-423d-99d8-73ace1542fdb' -name XXX -smp '2,sockets=1,cores=2,maxcpus=2' -nodefaults -boot 'menu=on,strict=on,reboot-timeout=1000' -vga cirrus -cpu kvm64,+lahf_lm,+sep,+kvm_pv_unhalt,+kvm_pv_eoi,enforce -m 4096 -k fr -device 'pci-bridge,id=pci.2,chassis_nr=2,bus=pci.0,addr=0x1f' -device 'pci-bridge,id=pci.1,chassis_nr=1,bus=pci.0,addr=0x1e' -device 'piix3-usb-uhci,id=uhci,bus=pci.0,addr=0x1.0x2' -device 'usb-tablet,id=tablet,bus=uhci.0,port=1' -device 'virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x3' -iscsi 'initiator-name=iqn.1993-08.org.debian:01:4dc4b42f16f6' -drive 'file=rbd:CEPH/vm-108-disk-1:mon_host=DC01 DC02 DC03:id=admin:auth_supported=cephx:keyring=/etc/pve/priv/ceph/CEPH.keyring,if=none,id=drive-virtio0,cache=writeback,format=raw,aio=threads,detect-zeroes=on' -device 'virtio-blk-pci,drive=drive-virtio0,id=virtio0,bus=pci.0,addr=0xa,bootindex=100' -netdev 'type=tap,id=net0,ifname=tap108i0,script=/var/lib/qemu-server/pve-bridge,downscript=/var/lib/qemu-server/pve-bridgedown,vhost=on' -device 'virtio-net-pci,mac=XX:XX:XX:XX:XX:XX,netdev=net0,bus=pci.0,addr=0x12,id=net0,bootindex=300' -incoming tcp:localhost:60000 -S' failed: exit code 1
janv. 28 20:16:43 ERROR: online migrate failure - command '/usr/bin/ssh -o 'BatchMode=yes' root@10.0.0.2 qm start 108 --stateuri tcp --skiplock --migratedfrom DC01' failed: exit code 255
janv. 28 20:16:43 aborting phase 2 - cleanup resources
janv. 28 20:16:43 migrate_cancel
janv. 28 20:16:44 ERROR: migration finished with problems (duration 00:00:03)
TASK ERROR: migration problems

Running as unit 108.scope.
libust[5652/5652]: Error: Error cancelling global ust listener thread: No such process (in lttng_ust_exit() at lttng-ust-comm.c:1592)
libust[5652/5652]: Error: Error cancelling local ust listener thread: No such process (in lttng_ust_exit() at lttng-ust-comm.c:1601)
kvm: -vnc unix:/var/run/qemu-server/108.vnc,x509,password: Failed to start VNC server: Our own certificate /etc/pve/local/pve-ssl.pem failed validation against /etc/pve/pve-root-ca.pem: The certificate hasn't got a known issuer
kvm: ./common/Mutex.h:96: void Mutex::_pre_unlock(): Assertion `nlock > 0' failed.
TASK ERROR: start failed: command '/usr/bin/systemd-run --scope --slice qemu --unit 108 -p 'KillMode=none' -p 'CPUShares=1000' /usr/bin/kvm -id 108 -chardev 'socket,id=qmp,path=/var/run/qemu-server/108.qmp,server,nowait' -mon 'chardev=qmp,mode=control' -vnc unix:/var/run/qemu-server/108.vnc,x509,password -pidfile /var/run/qemu-server/108.pid -daemonize -smbios 'type=1,uuid=f52b5d2c-2009-423d-99d8-73ace1542fdb' -name XXXX -smp '2,sockets=1,cores=2,maxcpus=2' -nodefaults -boot 'menu=on,strict=on,reboot-timeout=1000' -vga cirrus -cpu kvm64,+lahf_lm,+sep,+kvm_pv_unhalt,+kvm_pv_eoi,enforce -m 4096 -k fr -device 'pci-bridge,id=pci.2,chassis_nr=2,bus=pci.0,addr=0x1f' -device 'pci-bridge,id=pci.1,chassis_nr=1,bus=pci.0,addr=0x1e' -device 'piix3-usb-uhci,id=uhci,bus=pci.0,addr=0x1.0x2' -device 'usb-tablet,id=tablet,bus=uhci.0,port=1' -device 'virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x3' -iscsi 'initiator-name=iqn.1993-08.org.debian:01:4dc4b42f16f6' -drive 'file=rbd:CEPH/vm-108-disk-1:mon_host=DC01 DC02 DC03:id=admin:auth_supported=cephx:keyring=/etc/pve/priv/ceph/CEPH.keyring,if=none,id=drive-virtio0,cache=writeback,format=raw,aio=threads,detect-zeroes=on' -device 'virtio-blk-pci,drive=drive-virtio0,id=virtio0,bus=pci.0,addr=0xa,bootindex=100' -netdev 'type=tap,id=net0,ifname=tap108i0,script=/var/lib/qemu-server/pve-bridge,downscript=/var/lib/qemu-server/pve-bridgedown,vhost=on' -device 'virtio-net-pci,mac=XX:XX:XX:XX:XX:XX,netdev=net0,bus=pci.0,addr=0x12,id=net0,bootindex=300' -incoming tcp:localhost:60000 -S' failed: exit code 1

And when I want backup:

INFO: starting new backup job: vzdump 108 --storage backup --mode snapshot --compress lzo --remove 0 --node DC02
INFO: Starting Backup of VM 108 (qemu)
INFO: status = stopped
INFO: update VM 108: -lock backup
INFO: backup mode: stop
INFO: ionice priority: 7
libust[10162/10162]: Warning: HOME environment variable not set. Disabling LTTng-UST per-user tracing. (in setup_local_apps() at lttng-ust-comm.c:375)
INFO: creating archive '/home/dump/vzdump-qemu-108-2016_01_28-20_36_19.vma.lzo'
INFO: starting kvm to execute backup task
Running as unit 108.scope.
libust[10169/10169]: Warning: HOME environment variable not set. Disabling LTTng-UST per-user tracing. (in setup_local_apps() at lttng-ust-comm.c:375)
libust[10172/10172]: Error: Error cancelling global ust listener thread: No such process (in lttng_ust_exit() at lttng-ust-comm.c:1592)
kvm: -vnc unix:/var/run/qemu-server/108.vnc,x509,password: Failed to start VNC server: Our own certificate /etc/pve/local/pve-ssl.pem failed validation against /etc/pve/pve-root-ca.pem: The certificate hasn't got a known issuer
ERROR: Backup of VM 108 failed - start failed: command '/usr/bin/systemd-run --scope --slice qemu --unit 108 -p 'KillMode=none' -p 'CPUShares=1000' /usr/bin/kvm -id 108 -chardev 'socket,id=qmp,path=/var/run/qemu-server/108.qmp,server,nowait' -mon 'chardev=qmp,mode=control' -vnc unix:/var/run/qemu-server/108.vnc,x509,password -pidfile /var/run/qemu-server/108.pid -daemonize -smbios 'type=1,uuid=04ff4252-820c-4187-9b03-039f4e5b0073' -name test -smp '5,sockets=1,cores=5,maxcpus=5' -nodefaults -boot 'menu=on,strict=on,reboot-timeout=1000' -vga cirrus -cpu kvm64,+lahf_lm,+sep,+kvm_pv_unhalt,+kvm_pv_eoi,enforce -m 4096 -k fr -device 'pci-bridge,id=pci.1,chassis_nr=1,bus=pci.0,addr=0x1e' -device 'pci-bridge,id=pci.2,chassis_nr=2,bus=pci.0,addr=0x1f' -device 'piix3-usb-uhci,id=uhci,bus=pci.0,addr=0x1.0x2' -device 'usb-tablet,id=tablet,bus=uhci.0,port=1' -device 'virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x3' -iscsi 'initiator-name=iqn.1993-08.org.debian:01:4dc4b42f16f6' -drive 'file=/home/images/108/vm-108-disk-1.raw,if=none,id=drive-virtio0,cache=writeback,format=raw,aio=threads,detect-zeroes=on' -device 'virtio-blk-pci,drive=drive-virtio0,id=virtio0,bus=pci.0,addr=0xa,bootindex=100' -netdev 'type=tap,id=net0,ifname=tap108i0,script=/var/lib/qemu-server/pve-bridge,downscript=/var/lib/qemu-server/pve-bridgedown' -device 'e1000,mac=32:37:66:33:62:65,netdev=net0,bus=pci.0,addr=0x12,id=net0,bootindex=300' -netdev 'type=tap,id=net1,ifname=tap108i1,script=/var/lib/qemu-server/pve-bridge,downscript=/var/lib/qemu-server/pve-bridgedown' -device 'e1000,mac=62:64:35:63:64:65,netdev=net1,bus=pci.0,addr=0x13,id=net1,bootindex=301' -netdev 'type=tap,id=net2,ifname=tap108i2,script=/var/lib/qemu-server/pve-bridge,downscript=/var/lib/qemu-server/pve-bridgedown' -device 'e1000,mac=62:36:63:66:64:39,netdev=net2,bus=pci.0,addr=0x14,id=net2,bootindex=302' -netdev 'type=tap,id=net3,ifname=tap108i3,script=/var/lib/qemu-server/pve-bridge,downscript=/var/lib/qemu-server/pve-bridgedown' -device 'e1000,mac=32:36:33:33:64:31,netdev=net3,bus=pci.0,addr=0x15,id=net3,bootindex=303' -netdev 'type=tap,id=net4,ifname=tap108i4,script=/var/lib/qemu-server/pve-bridge,downscript=/var/lib/qemu-server/pve-bridgedown' -device 'e1000,mac=32:61:37:65:32:64,netdev=net4,bus=pci.0,addr=0x16,id=net4,bootindex=304' -netdev 'type=tap,id=net5,ifname=tap108i5,script=/var/lib/qemu-server/pve-bridge,downscript=/var/lib/qemu-server/pve-bridgedown' -device 'e1000,mac=62:38:39:66:37:34,netdev=net5,bus=pci.0,addr=0x17,id=net5,bootindex=305' -S' failed: exit code 1
INFO: Backup job finished with errors
TASK ERROR: job errors

Here my pveversion -v :

proxmox-ve: 4.1-34 (running kernel: 4.2.6-1-pve)
pve-manager: 4.1-5 (running version: 4.1-5/f910ef5c)
pve-kernel-4.2.6-1-pve: 4.2.6-34
lvm2: 2.02.116-pve2
corosync-pve: 2.3.5-2
libqb0: 0.17.2-1
pve-cluster: 4.0-31
qemu-server: 4.0-49
pve-firmware: 1.1-7
libpve-common-perl: 4.0-45
libpve-access-control: 4.0-11
libpve-storage-perl: 4.0-38
pve-libspice-server1: 0.12.5-2
vncterm: 1.2-1
pve-qemu-kvm: 2.5-3
pve-container: 1.0-39
pve-firewall: 2.0-15
pve-ha-manager: 1.0-19
ksm-control-daemon: 1.2-1
glusterfs-client: 3.5.2-2+deb8u1
lxc-pve: 1.1.5-6
lxcfs: 0.13-pve3
cgmanager: 0.39-pve1
criu: 1.6.0-1

Many thanks to help me

 

[VollStrecker]

Hi,

I had the same problem after installing the latest package updates. I was not able to start or backup a kvm-based virtual machine.
After downgrading the package "pve-qemu-kvm" from 2.5-3 to 2.4.21 using the following command, everything works fine:

apt-get install pve-qemu-kvm=2.4-21

 

[Foxhound77]

Many thanks guy you save my life !!!

Note for this problem: hold this package to aptitude

 

[niko2]

I had same issue after upgrade. Problem was solved using the good CA certificate /etc/pve/pve-root-ca.pem (from https://www.cacert.org/ in my case)
see : https://pve.proxmox.com/wiki/HTTPSCertificateConfiguration