Online Migration of OpenVZ container causes target box to kernel panic.
- Thread starter briankwest
- Start date
Apr 12 16:37:08 starting migration of CT 110 to node 'proxmox00' (192.168.1.252)
Apr 12 16:37:08 container is running - using online migration
Apr 12 16:37:08 starting rsync phase 1
Apr 12 16:37:08 # /usr/bin/rsync -aH --delete --numeric-ids --sparse /var/lib/vz/private/110 root@192.168.1.252:/var/lib/vz/private
Apr 12 16:37:33 start live migration - suspending container
Apr 12 16:37:33 dump container state
Apr 12 16:37:33 copy dump file to target node
Apr 12 16:37:35 starting rsync (2nd pass)
Apr 12 16:37:35 # /usr/bin/rsync -aH --delete --numeric-ids /var/lib/vz/private/110 root@192.168.1.252:/var/lib/vz/private
Apr 12 16:37:35 dump 2nd level quota
Apr 12 16:37:35 copy 2nd level quota to target node
Apr 12 16:37:40 initialize container on remote node 'proxmox00'
Apr 12 16:37:40 initializing remote quota
Apr 12 16:37:41 turn on remote quota
Apr 12 16:37:41 load 2nd level quota
Apr 12 16:37:41 starting container on remote node 'proxmox00'
Apr 12 16:37:41 restore container state
Apr 12 16:42:41 # /usr/bin/ssh -c blowfish -o 'BatchMode=yes' root@192.168.1.252 vzctl restore 110 --undump --dumpfile /var/lib/vz/dump/dump.110 --skip_arpdetect
Apr 12 16:37:41 Restoring container ...
Apr 12 16:37:41 Starting container ...
Apr 12 16:37:41 Container is mounted
Apr 12 16:37:41 undump...
Apr 12 16:37:41 Adding IP address(es): 1.2.3.4
Apr 12 16:37:41 Setting CPU units: 1000
Apr 12 16:37:41 Setting CPUs: 1
Apr 12 16:42:41 vzquota : (warning) Quota is running for id 110 already
Apr 12 16:42:41 Write failed: Broken pipe
Apr 12 16:42:41 ERROR: online migrate failure - Failed to restore container: exit code 255
Apr 12 16:42:41 removing container files on local node
Apr 12 16:42:41 start final cleanup
Apr 12 16:42:42 ERROR: migration finished with problems (duration 00:05:35)
TASK ERROR: migration problems
pve-manager: 2.0-57 (pve-manager/2.0/ff6cd700)
running kernel: 2.6.32-11-pve
proxmox-ve-2.6.32: 2.0-65
pve-kernel-2.6.32-10-pve: 2.6.32-63
pve-kernel-2.6.32-11-pve: 2.6.32-65
lvm2: 2.02.88-2pve2
clvm: 2.02.88-2pve2
corosync-pve: 1.4.1-1
openais-pve: 1.1.4-2
libqb: 0.10.1-2
redhat-cluster-pve: 3.1.8-3
resource-agents-pve: 3.9.2-3
fence-agents-pve: 3.1.7-2
pve-cluster: 1.0-26
qemu-server: 2.0-37
pve-firmware: 1.0-15
libpve-common-perl: 1.0-25
libpve-access-control: 1.0-17
libpve-storage-perl: 2.0-17
vncterm: 1.0-2
vzctl: 3.0.30-2pve2
vzprocps: 2.0.11-2
vzquota: 3.0.12-3
pve-qemu-kvm: 1.0-9
ksm-control-daemon: 1.1-1
The source node everything was fine.
/b
Apr 12 16:37:08 container is running - using online migration
Apr 12 16:37:08 starting rsync phase 1
Apr 12 16:37:08 # /usr/bin/rsync -aH --delete --numeric-ids --sparse /var/lib/vz/private/110 root@192.168.1.252:/var/lib/vz/private
Apr 12 16:37:33 start live migration - suspending container
Apr 12 16:37:33 dump container state
Apr 12 16:37:33 copy dump file to target node
Apr 12 16:37:35 starting rsync (2nd pass)
Apr 12 16:37:35 # /usr/bin/rsync -aH --delete --numeric-ids /var/lib/vz/private/110 root@192.168.1.252:/var/lib/vz/private
Apr 12 16:37:35 dump 2nd level quota
Apr 12 16:37:35 copy 2nd level quota to target node
Apr 12 16:37:40 initialize container on remote node 'proxmox00'
Apr 12 16:37:40 initializing remote quota
Apr 12 16:37:41 turn on remote quota
Apr 12 16:37:41 load 2nd level quota
Apr 12 16:37:41 starting container on remote node 'proxmox00'
Apr 12 16:37:41 restore container state
Apr 12 16:42:41 # /usr/bin/ssh -c blowfish -o 'BatchMode=yes' root@192.168.1.252 vzctl restore 110 --undump --dumpfile /var/lib/vz/dump/dump.110 --skip_arpdetect
Apr 12 16:37:41 Restoring container ...
Apr 12 16:37:41 Starting container ...
Apr 12 16:37:41 Container is mounted
Apr 12 16:37:41 undump...
Apr 12 16:37:41 Adding IP address(es): 1.2.3.4
Apr 12 16:37:41 Setting CPU units: 1000
Apr 12 16:37:41 Setting CPUs: 1
Apr 12 16:42:41 vzquota : (warning) Quota is running for id 110 already
Apr 12 16:42:41 Write failed: Broken pipe
Apr 12 16:42:41 ERROR: online migrate failure - Failed to restore container: exit code 255
Apr 12 16:42:41 removing container files on local node
Apr 12 16:42:41 start final cleanup
Apr 12 16:42:42 ERROR: migration finished with problems (duration 00:05:35)
TASK ERROR: migration problems
pve-manager: 2.0-57 (pve-manager/2.0/ff6cd700)
running kernel: 2.6.32-11-pve
proxmox-ve-2.6.32: 2.0-65
pve-kernel-2.6.32-10-pve: 2.6.32-63
pve-kernel-2.6.32-11-pve: 2.6.32-65
lvm2: 2.02.88-2pve2
clvm: 2.02.88-2pve2
corosync-pve: 1.4.1-1
openais-pve: 1.1.4-2
libqb: 0.10.1-2
redhat-cluster-pve: 3.1.8-3
resource-agents-pve: 3.9.2-3
fence-agents-pve: 3.1.7-2
pve-cluster: 1.0-26
qemu-server: 2.0-37
pve-firmware: 1.0-15
libpve-common-perl: 1.0-25
libpve-access-control: 1.0-17
libpve-storage-perl: 2.0-17
vncterm: 1.0-2
vzctl: 3.0.30-2pve2
vzprocps: 2.0.11-2
vzquota: 3.0.12-3
pve-qemu-kvm: 1.0-9
ksm-control-daemon: 1.1-1
The source node everything was fine.
/b
Message from syslogd@proxmox00 at Apr 12 16:48:15 ...
kernel:Oops: 0000 [#1] SMP
Message from syslogd@proxmox00 at Apr 12 16:48:15 ...
kernel:last sysfs file: /sys/kernel/uevent_seqnum
Message from syslogd@proxmox00 at Apr 12 16:48:15 ...
kernel:Stack:
Message from syslogd@proxmox00 at Apr 12 16:48:15 ...
kernel:Call Trace:
Message from syslogd@proxmox00 at Apr 12 16:48:15 ...
kernel:Code: 03 00 00 65 48 8b 04 25 c8 cb 00 00 48 8b 80 38 e0 ff ff a8 08 0f 85 55 fb ff ff 48 81 c4 88 00 00 00 5b 41 5c 41 5d 41 5e 41 5f <c9> c3 66 0f 1f 44 00 00 48 29 d0 48 63 c9 48 89 c2 48 8b 05 30
Message from syslogd@proxmox00 at Apr 12 16:48:15 ...
kernel:CR2: 0000000000000000
Message from syslogd@proxmox00 at Apr 12 16:48:15 ...
kernel:Kernel panic - not syncing: Fatal exception
kernel:Oops: 0000 [#1] SMP
Message from syslogd@proxmox00 at Apr 12 16:48:15 ...
kernel:last sysfs file: /sys/kernel/uevent_seqnum
Message from syslogd@proxmox00 at Apr 12 16:48:15 ...
kernel:Stack:
Message from syslogd@proxmox00 at Apr 12 16:48:15 ...
kernel:Call Trace:
Message from syslogd@proxmox00 at Apr 12 16:48:15 ...
kernel:Code: 03 00 00 65 48 8b 04 25 c8 cb 00 00 48 8b 80 38 e0 ff ff a8 08 0f 85 55 fb ff ff 48 81 c4 88 00 00 00 5b 41 5c 41 5d 41 5e 41 5f <c9> c3 66 0f 1f 44 00 00 48 29 d0 48 63 c9 48 89 c2 48 8b 05 30
Message from syslogd@proxmox00 at Apr 12 16:48:15 ...
kernel:CR2: 0000000000000000
Message from syslogd@proxmox00 at Apr 12 16:48:15 ...
kernel:Kernel panic - not syncing: Fatal exception
Updated both systems and tried again.
pve-manager: 2.0-59 (pve-manager/2.0/18400f07)
running kernel: 2.6.32-11-pve
proxmox-ve-2.6.32: 2.0-66
pve-kernel-2.6.32-10-pve: 2.6.32-63
pve-kernel-2.6.32-11-pve: 2.6.32-66
lvm2: 2.02.88-2pve2
clvm: 2.02.88-2pve2
corosync-pve: 1.4.1-1
openais-pve: 1.1.4-2
libqb: 0.10.1-2
redhat-cluster-pve: 3.1.8-3
resource-agents-pve: 3.9.2-3
fence-agents-pve: 3.1.7-2
pve-cluster: 1.0-26
qemu-server: 2.0-38
pve-firmware: 1.0-15
libpve-common-perl: 1.0-26
libpve-access-control: 1.0-18
libpve-storage-perl: 2.0-17
vncterm: 1.0-2
vzctl: 3.0.30-2pve2
vzprocps: 2.0.11-2
vzquota: 3.0.12-3
pve-qemu-kvm: 1.0-9
ksm-control-daemon: 1.1-1
Same Result!
pve-manager: 2.0-59 (pve-manager/2.0/18400f07)
running kernel: 2.6.32-11-pve
proxmox-ve-2.6.32: 2.0-66
pve-kernel-2.6.32-10-pve: 2.6.32-63
pve-kernel-2.6.32-11-pve: 2.6.32-66
lvm2: 2.02.88-2pve2
clvm: 2.02.88-2pve2
corosync-pve: 1.4.1-1
openais-pve: 1.1.4-2
libqb: 0.10.1-2
redhat-cluster-pve: 3.1.8-3
resource-agents-pve: 3.9.2-3
fence-agents-pve: 3.1.7-2
pve-cluster: 1.0-26
qemu-server: 2.0-38
pve-firmware: 1.0-15
libpve-common-perl: 1.0-26
libpve-access-control: 1.0-18
libpve-storage-perl: 2.0-17
vncterm: 1.0-2
vzctl: 3.0.30-2pve2
vzprocps: 2.0.11-2
vzquota: 3.0.12-3
pve-qemu-kvm: 1.0-9
ksm-control-daemon: 1.1-1
Same Result!
Migration works without problems here. But what kind of container is that (what template?) - how to reproduce the bug?
Source is a : Intel(R) Xeon(R) CPU E31275 @ 3.40GHz
Target machine is: Intel(R) Xeon(R) CPU X3370 @ 3.00GHz
/b
Target machine is: Intel(R) Xeon(R) CPU X3370 @ 3.00GHz
/b
S
stevek
Guest
Yes I got it as well. The thing is I could not replicated with two proxmox run on two KVM - but on real hardnode node it happened. However if on hardware node I use the original kernel form the cd (.10 not .11) then problem did not happen with me. Exact the same kernel OOP message. on the screenshot.
In my case container is centos 6 amd64. For now I just use the old .10 kernel it looks stable in my current setup. I also tried another newer kernel from openvz vmlinuz-2.6.32-042stab054.2 with the same symptom. Not sure where to report the bug though, I am new to the forum
In my case container is centos 6 amd64. For now I just use the old .10 kernel it looks stable in my current setup. I also tried another newer kernel from openvz vmlinuz-2.6.32-042stab054.2 with the same symptom. Not sure where to report the bug though, I am new to the forum
Last edited by a moderator:
I also tried another newer kernel from openvz vmlinuz-2.6.32-042stab054.2 with the same symptom. Not sure where to report the bug though, I am new to the forum
I cannot reproduce the bug here, so maybe it is better if you report directly to bugzilla.openvz.org. But please test with latest available openvz kernel.
S
stevek
Guest
Not sure if it is hardware related. As I said I could not reproduce if two proxmox are running on two kvm machines.
When you mean latest kernel - it is testing kernel as well? looks like now it is 042stab054.3 - the one I tested is 042stab054.2 but the change log does not have any hint abt the oops.
Will test tomorrow ...
Some more info.
It looks weird enough that if the source node is running old (-10) kernel, it does not happen with target node no matter it runs old or new kernel. But if source node is running new kernel - it is 100% reproducible.
Updated: Just test the latest one 042stab054.3 and same thing.
Last edited by a moderator: