[SOLVED] Backup and migration imposible (Proxmox 4)

Foxhound77

New Member
Jan 29, 2016
8
0
1
34
Hello,

I have a big problem with my 3 nodes cluster (DC01, DC02 and DC03). Each node is in version 4 with a ceph rbd.
Everything works fine however two day ago I cannot run, migrate (live or HA) or backup any vm in DC02 and DC03.

In DC01 I can run and backup my VM (local and ceph storage)
In DC02 and DC03 I cannot do that I receive this message when I migrate a VM:

Code:
janv. 28 20:16:41 starting migration of VM 108 to node 'DC02' (10.0.0.2)
janv. 28 20:16:41 copying disk images
janv. 28 20:16:41 starting VM 108 on remote node 'DC02'
janv. 28 20:16:43 start failed: command '/usr/bin/systemd-run --scope --slice qemu --unit 108 -p 'KillMode=none' -p 'CPUShares=1000' /usr/bin/kvm -id 108 -chardev 'socket,id=qmp,path=/var/run/qemu-server/108.qmp,server,nowait' -mon 'chardev=qmp,mode=control' -vnc unix:/var/run/qemu-server/108.vnc,x509,password -pidfile /var/run/qemu-server/108.pid -daemonize -smbios 'type=1,uuid=f52b5d2c-2009-423d-99d8-73ace1542fdb' -name XXX -smp '2,sockets=1,cores=2,maxcpus=2' -nodefaults -boot 'menu=on,strict=on,reboot-timeout=1000' -vga cirrus -cpu kvm64,+lahf_lm,+sep,+kvm_pv_unhalt,+kvm_pv_eoi,enforce -m 4096 -k fr -device 'pci-bridge,id=pci.2,chassis_nr=2,bus=pci.0,addr=0x1f' -device 'pci-bridge,id=pci.1,chassis_nr=1,bus=pci.0,addr=0x1e' -device 'piix3-usb-uhci,id=uhci,bus=pci.0,addr=0x1.0x2' -device 'usb-tablet,id=tablet,bus=uhci.0,port=1' -device 'virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x3' -iscsi 'initiator-name=iqn.1993-08.org.debian:01:4dc4b42f16f6' -drive 'file=rbd:CEPH/vm-108-disk-1:mon_host=DC01 DC02 DC03:id=admin:auth_supported=cephx:keyring=/etc/pve/priv/ceph/CEPH.keyring,if=none,id=drive-virtio0,cache=writeback,format=raw,aio=threads,detect-zeroes=on' -device 'virtio-blk-pci,drive=drive-virtio0,id=virtio0,bus=pci.0,addr=0xa,bootindex=100' -netdev 'type=tap,id=net0,ifname=tap108i0,script=/var/lib/qemu-server/pve-bridge,downscript=/var/lib/qemu-server/pve-bridgedown,vhost=on' -device 'virtio-net-pci,mac=XX:XX:XX:XX:XX:XX,netdev=net0,bus=pci.0,addr=0x12,id=net0,bootindex=300' -incoming tcp:localhost:60000 -S' failed: exit code 1
janv. 28 20:16:43 ERROR: online migrate failure - command '/usr/bin/ssh -o 'BatchMode=yes' root@10.0.0.2 qm start 108 --stateuri tcp --skiplock --migratedfrom DC01' failed: exit code 255
janv. 28 20:16:43 aborting phase 2 - cleanup resources
janv. 28 20:16:43 migrate_cancel
janv. 28 20:16:44 ERROR: migration finished with problems (duration 00:00:03)
TASK ERROR: migration problems

Code:
Running as unit 108.scope.
libust[5652/5652]: Error: Error cancelling global ust listener thread: No such process (in lttng_ust_exit() at lttng-ust-comm.c:1592)
libust[5652/5652]: Error: Error cancelling local ust listener thread: No such process (in lttng_ust_exit() at lttng-ust-comm.c:1601)
kvm: -vnc unix:/var/run/qemu-server/108.vnc,x509,password: Failed to start VNC server: Our own certificate /etc/pve/local/pve-ssl.pem failed validation against /etc/pve/pve-root-ca.pem: The certificate hasn't got a known issuer
kvm: ./common/Mutex.h:96: void Mutex::_pre_unlock(): Assertion `nlock > 0' failed.
TASK ERROR: start failed: command '/usr/bin/systemd-run --scope --slice qemu --unit 108 -p 'KillMode=none' -p 'CPUShares=1000' /usr/bin/kvm -id 108 -chardev 'socket,id=qmp,path=/var/run/qemu-server/108.qmp,server,nowait' -mon 'chardev=qmp,mode=control' -vnc unix:/var/run/qemu-server/108.vnc,x509,password -pidfile /var/run/qemu-server/108.pid -daemonize -smbios 'type=1,uuid=f52b5d2c-2009-423d-99d8-73ace1542fdb' -name XXXX -smp '2,sockets=1,cores=2,maxcpus=2' -nodefaults -boot 'menu=on,strict=on,reboot-timeout=1000' -vga cirrus -cpu kvm64,+lahf_lm,+sep,+kvm_pv_unhalt,+kvm_pv_eoi,enforce -m 4096 -k fr -device 'pci-bridge,id=pci.2,chassis_nr=2,bus=pci.0,addr=0x1f' -device 'pci-bridge,id=pci.1,chassis_nr=1,bus=pci.0,addr=0x1e' -device 'piix3-usb-uhci,id=uhci,bus=pci.0,addr=0x1.0x2' -device 'usb-tablet,id=tablet,bus=uhci.0,port=1' -device 'virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x3' -iscsi 'initiator-name=iqn.1993-08.org.debian:01:4dc4b42f16f6' -drive 'file=rbd:CEPH/vm-108-disk-1:mon_host=DC01 DC02 DC03:id=admin:auth_supported=cephx:keyring=/etc/pve/priv/ceph/CEPH.keyring,if=none,id=drive-virtio0,cache=writeback,format=raw,aio=threads,detect-zeroes=on' -device 'virtio-blk-pci,drive=drive-virtio0,id=virtio0,bus=pci.0,addr=0xa,bootindex=100' -netdev 'type=tap,id=net0,ifname=tap108i0,script=/var/lib/qemu-server/pve-bridge,downscript=/var/lib/qemu-server/pve-bridgedown,vhost=on' -device 'virtio-net-pci,mac=XX:XX:XX:XX:XX:XX,netdev=net0,bus=pci.0,addr=0x12,id=net0,bootindex=300' -incoming tcp:localhost:60000 -S' failed: exit code 1

And when I want backup:

Code:
INFO: starting new backup job: vzdump 108 --storage backup --mode snapshot --compress lzo --remove 0 --node DC02
INFO: Starting Backup of VM 108 (qemu)
INFO: status = stopped
INFO: update VM 108: -lock backup
INFO: backup mode: stop
INFO: ionice priority: 7
libust[10162/10162]: Warning: HOME environment variable not set. Disabling LTTng-UST per-user tracing. (in setup_local_apps() at lttng-ust-comm.c:375)
INFO: creating archive '/home/dump/vzdump-qemu-108-2016_01_28-20_36_19.vma.lzo'
INFO: starting kvm to execute backup task
Running as unit 108.scope.
libust[10169/10169]: Warning: HOME environment variable not set. Disabling LTTng-UST per-user tracing. (in setup_local_apps() at lttng-ust-comm.c:375)
libust[10172/10172]: Error: Error cancelling global ust listener thread: No such process (in lttng_ust_exit() at lttng-ust-comm.c:1592)
kvm: -vnc unix:/var/run/qemu-server/108.vnc,x509,password: Failed to start VNC server: Our own certificate /etc/pve/local/pve-ssl.pem failed validation against /etc/pve/pve-root-ca.pem: The certificate hasn't got a known issuer
ERROR: Backup of VM 108 failed - start failed: command '/usr/bin/systemd-run --scope --slice qemu --unit 108 -p 'KillMode=none' -p 'CPUShares=1000' /usr/bin/kvm -id 108 -chardev 'socket,id=qmp,path=/var/run/qemu-server/108.qmp,server,nowait' -mon 'chardev=qmp,mode=control' -vnc unix:/var/run/qemu-server/108.vnc,x509,password -pidfile /var/run/qemu-server/108.pid -daemonize -smbios 'type=1,uuid=04ff4252-820c-4187-9b03-039f4e5b0073' -name test -smp '5,sockets=1,cores=5,maxcpus=5' -nodefaults -boot 'menu=on,strict=on,reboot-timeout=1000' -vga cirrus -cpu kvm64,+lahf_lm,+sep,+kvm_pv_unhalt,+kvm_pv_eoi,enforce -m 4096 -k fr -device 'pci-bridge,id=pci.1,chassis_nr=1,bus=pci.0,addr=0x1e' -device 'pci-bridge,id=pci.2,chassis_nr=2,bus=pci.0,addr=0x1f' -device 'piix3-usb-uhci,id=uhci,bus=pci.0,addr=0x1.0x2' -device 'usb-tablet,id=tablet,bus=uhci.0,port=1' -device 'virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x3' -iscsi 'initiator-name=iqn.1993-08.org.debian:01:4dc4b42f16f6' -drive 'file=/home/images/108/vm-108-disk-1.raw,if=none,id=drive-virtio0,cache=writeback,format=raw,aio=threads,detect-zeroes=on' -device 'virtio-blk-pci,drive=drive-virtio0,id=virtio0,bus=pci.0,addr=0xa,bootindex=100' -netdev 'type=tap,id=net0,ifname=tap108i0,script=/var/lib/qemu-server/pve-bridge,downscript=/var/lib/qemu-server/pve-bridgedown' -device 'e1000,mac=32:37:66:33:62:65,netdev=net0,bus=pci.0,addr=0x12,id=net0,bootindex=300' -netdev 'type=tap,id=net1,ifname=tap108i1,script=/var/lib/qemu-server/pve-bridge,downscript=/var/lib/qemu-server/pve-bridgedown' -device 'e1000,mac=62:64:35:63:64:65,netdev=net1,bus=pci.0,addr=0x13,id=net1,bootindex=301' -netdev 'type=tap,id=net2,ifname=tap108i2,script=/var/lib/qemu-server/pve-bridge,downscript=/var/lib/qemu-server/pve-bridgedown' -device 'e1000,mac=62:36:63:66:64:39,netdev=net2,bus=pci.0,addr=0x14,id=net2,bootindex=302' -netdev 'type=tap,id=net3,ifname=tap108i3,script=/var/lib/qemu-server/pve-bridge,downscript=/var/lib/qemu-server/pve-bridgedown' -device 'e1000,mac=32:36:33:33:64:31,netdev=net3,bus=pci.0,addr=0x15,id=net3,bootindex=303' -netdev 'type=tap,id=net4,ifname=tap108i4,script=/var/lib/qemu-server/pve-bridge,downscript=/var/lib/qemu-server/pve-bridgedown' -device 'e1000,mac=32:61:37:65:32:64,netdev=net4,bus=pci.0,addr=0x16,id=net4,bootindex=304' -netdev 'type=tap,id=net5,ifname=tap108i5,script=/var/lib/qemu-server/pve-bridge,downscript=/var/lib/qemu-server/pve-bridgedown' -device 'e1000,mac=62:38:39:66:37:34,netdev=net5,bus=pci.0,addr=0x17,id=net5,bootindex=305' -S' failed: exit code 1
INFO: Backup job finished with errors
TASK ERROR: job errors


Here my pveversion -v :

Code:
proxmox-ve: 4.1-34 (running kernel: 4.2.6-1-pve)
pve-manager: 4.1-5 (running version: 4.1-5/f910ef5c)
pve-kernel-4.2.6-1-pve: 4.2.6-34
lvm2: 2.02.116-pve2
corosync-pve: 2.3.5-2
libqb0: 0.17.2-1
pve-cluster: 4.0-31
qemu-server: 4.0-49
pve-firmware: 1.1-7
libpve-common-perl: 4.0-45
libpve-access-control: 4.0-11
libpve-storage-perl: 4.0-38
pve-libspice-server1: 0.12.5-2
vncterm: 1.2-1
pve-qemu-kvm: 2.5-3
pve-container: 1.0-39
pve-firewall: 2.0-15
pve-ha-manager: 1.0-19
ksm-control-daemon: 1.2-1
glusterfs-client: 3.5.2-2+deb8u1
lxc-pve: 1.1.5-6
lxcfs: 0.13-pve3
cgmanager: 0.39-pve1
criu: 1.6.0-1


Many thanks to help me
 
Hi,

I had the same problem after installing the latest package updates. I was not able to start or backup a kvm-based virtual machine.
After downgrading the package "pve-qemu-kvm" from 2.5-3 to 2.4.21 using the following command, everything works fine:

apt-get install pve-qemu-kvm=2.4-21
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!