Different PVE versions in cluster, cannot migrate

[check-ict]

Hello,

I have installed 4 nodes. 1 was a bit older (few months) and the 3 others are newer.

I migrated all VM's today from node 1 to the other 3 nodes. That went fine.

Now node 1 is rebooted, I want to migrate some VM's back to balance the load, but Im unable to do so. I'm getting the following error:
Nov 12 23:18:29 copying disk images
Nov 12 23:18:29 starting VM 121 on remote node 'proxmox-cluster-04'
Nov 12 23:18:31 start failed: command '/usr/bin/kvm -id 121 -chardev 'socket,id=qmp,path=/var/run/qemu-server/121.qmp,server,nowait' -mon 'chardev=qmp,mode=control' -vnc unix:/var/run/qemu-server/121.vnc,x509,password -pidfile /var/run/qemu-server/121.pid -daemonize -name router-online-werkplekken -smp 'sockets=1,cores=4' -nodefaults -boot 'menu=on' -vga cirrus -cpu kvm64,+x2apic,+sep -k en-us -m 2048 -device 'piix3-usb-uhci,id=uhci,bus=pci.0,addr=0x1.0x2' -device 'usb-tablet,id=tablet,bus=uhci.0,port=1' -device 'virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x3' -drive 'if=none,id=drive-ide2,media=cdrom,aio=native' -device 'ide-cd,bus=ide.1,unit=0,drive=drive-ide2,id=ide2,bootindex=200' -drive 'file=/mnt/pve/zfs01-online-werkplekken/images/121/vm-121-disk-1.raw,if=none,id=drive-virtio0,format=raw,aio=native,cache=none' -device 'virtio-blk-pci,drive=drive-virtio0,id=virtio0,bus=pci.0,addr=0xa,bootindex=100' -netdev 'type=tap,id=net0,ifname=tap121i0,script=/var/lib/qemu-server/pve-bridge,vhost=on' -device 'virtio-net-pci,mac=46:63:9F:A6:35:A2,netdev=net0,bus=pci.0,addr=0x12,id=net0,bootindex=300' -netdev 'type=tap,id=net1,ifname=tap121i1,script=/var/lib/qemu-server/pve-bridge,vhost=on' -device 'virtio-net-pci,mac=C6:44:0E:EB5:97,netdev=net1,bus=pci.0,addr=0x13,id=net1,bootindex=301' -machine 'type=pc-i440fx-1.7' -incoming tcp:localhost:60000 -S' failed: exit code 1
Nov 12 23:18:31 ERROR: online migrate failure - command '/usr/bin/ssh -o 'BatchMode=yes' root@10.1.1.104 qm start 121 --stateuri tcp --skiplock --migratedfrom proxmox-cluster-01 --machine pc-i440fx-1.7' failed: exit code 255
Nov 12 23:18:31 aborting phase 2 - cleanup resources
Nov 12 23:18:31 migrate_cancel
Nov 12 23:18:31 ERROR: migration finished with problems (duration 00:00:02)

This error is from all 3 nodes to the first node.

When I do pveversion -v I see a difference:
Problem node:
pveversion -v
proxmox-ve-2.6.32: 3.1-114 (running kernel: 2.6.32-26-pve)
pve-manager: 3.1-21 (running version: 3.1-21/93bf03d4)
pve-kernel-2.6.32-25-pve: 2.6.32-113
pve-kernel-2.6.32-22-pve: 2.6.32-107
pve-kernel-2.6.32-26-pve: 2.6.32-114
lvm2: 2.02.98-pve4
clvm: 2.02.98-pve4
corosync-pve: 1.4.5-1
openais-pve: 1.1.4-3
libqb0: 0.11.1-2
redhat-cluster-pve: 3.2.0-2
resource-agents-pve: 3.9.2-4
fence-agents-pve: 4.0.0-2
pve-cluster: 3.0-8
qemu-server: 3.1-8
pve-firmware: 1.0-23
libpve-common-perl: 3.0-8
libpve-access-control: 3.0-7
libpve-storage-perl: 3.0-17
pve-libspice-server1: 0.12.4-2
vncterm: 1.1-4
vzctl: 4.0-1pve4
vzprocps: 2.0.11-2
vzquota: 3.1-2
pve-qemu-kvm: 1.4-17
ksm-control-daemon: 1.1-1
glusterfs-client: 3.4.1-1

Good node
pveversion -v
proxmox-ve-2.6.32: 3.1-114 (running kernel: 2.6.32-26-pve)
pve-manager: 3.1-21 (running version: 3.1-21/93bf03d4)
pve-kernel-2.6.32-26-pve: 2.6.32-114
lvm2: 2.02.98-pve4
clvm: 2.02.98-pve4
corosync-pve: 1.4.5-1
openais-pve: 1.1.4-3
libqb0: 0.11.1-2
redhat-cluster-pve: 3.2.0-2
resource-agents-pve: 3.9.2-4
fence-agents-pve: 4.0.5-1
pve-cluster: 3.0-12
qemu-server: 3.1-15
pve-firmware: 1.1-2
libpve-common-perl: 3.0-14
libpve-access-control: 3.0-11
libpve-storage-perl: 3.0-19
pve-libspice-server1: 0.12.4-3
vncterm: 1.1-6
vzctl: 4.0-1pve5
vzprocps: 2.0.11-2
vzquota: 3.1-2
pve-qemu-kvm: 1.7-6
ksm-control-daemon: 1.1-1
glusterfs-client: 3.4.2-1de:

I tried to update the problem node, but it doesn't get new updates. I tried pveupgrade, apt-get upgrade, apt-get dist-upgrade, aptitude safe-upgrade but there are no new updates.

How can I get the problem node up-to-date? So I can move VM's again to this node.

 

[check-ict]

Ah, I noticed a difference in the apt sources:
deb http://download.proxmox.com/debian wheezy pve-no-subscription

This was:
deb http://download.proxmox.com/debian wheezy pve

So I changed it to the one with -no-subscription

I think I did this once because of the errors about full version.

 

[check-ict]

So it updated now, but it still won't migratie. Software seems newer compared to the 3 good nodes:

Problem node:
pveversion -v
proxmox-ve-2.6.32: 3.1-114 (running kernel: 2.6.32-26-pve)
pve-manager: 3.1-21 (running version: 3.1-21/93bf03d4)
pve-kernel-2.6.32-25-pve: 2.6.32-113
pve-kernel-2.6.32-22-pve: 2.6.32-107
pve-kernel-2.6.32-26-pve: 2.6.32-114
lvm2: 2.02.98-pve4
clvm: 2.02.98-pve4
corosync-pve: 1.4.7-1
openais-pve: 1.1.4-3
libqb0: 0.11.1-2
redhat-cluster-pve: 3.2.0-2
resource-agents-pve: 3.9.2-4
fence-agents-pve: 4.0.0-2
pve-cluster: 3.0-15
qemu-server: 3.1-8
pve-firmware: 1.1-3
libpve-common-perl: 3.0-19
libpve-access-control: 3.0-7
libpve-storage-perl: 3.0-25
pve-libspice-server1: 0.12.4-3
vncterm: 1.1-8
vzctl: 4.0-1pve6
vzprocps: 2.0.11-2
vzquota: 3.1-2
pve-qemu-kvm: 1.4-17
ksm-control-daemon: 1.1-1
glusterfs-client: 3.5.2-1

Any idea what I can do to make this server active again?

 

[check-ict]

I did the following on the bad node (copy all version info from a working node):
apt-get install lvm2=2.02.98-pve4 clvm=2.02.98-pve4 corosync-pve=1.4.5-1 openais-pve=1.1.4-3 libqb0=0.11.1-2 redhat-cluster-pve=3.2.0-2 resource-agents-pve=3.9.2-4 fence-agents-pve=4.0.5-1 pve-cluster=3.0-12 qemu-server=3.1-15 pve-firmware=1.1-2 libpve-common-perl=3.0-14 libpve-access-control=3.0-11 libpve-storage-perl=3.0-19 pve-libspice-server1=0.12.4-3 vncterm=1.1-6 vzctl=4.0-1pve5 vzprocps=2.0.11-2 vzquota=3.1-2 pve-qemu-kvm=1.7-6 ksm-control-daemon=1.1-1 glusterfs-client=3.4.2-1 -y

And it will install all older versions ;-)

And it works now!