Hallo Forum,
seit ein paar Tagen habe ich Probleme bei der Online Migration in unserem Proxmox 4.4 Datacenter. Ich habe mehrere Migrationen probiert. Egal von welchem Proxmox Node zu welchem Proxmox Node, keine Migration läuft durch. Offline Migrationen funktionieren reibungslos. Meiner Meinung nach besteht das Problem seit dem letzten Update von pve-manager/4.4-12 nach pve-manager/4.4-18.
Die Migration habe ich von der Kommandozeile mit folgendem Befehl gestartet: qm migrate 105 kvit-proxmox7 -targetstorage local-storage-ssd -online -with-local-disks
Das Logfile der Migration:
Okt 27 08:37:47 starting migration of VM 105 to node 'kvit-proxmox7' (172.22.23.201)
Okt 27 08:37:47 found local disk 'local-storage-sata:105/vm-105-disk-1.raw' (in current VM config)
Okt 27 08:37:47 found local disk 'local-storage-sata:105/vm-105-disk-2.raw' (in current VM config)
Okt 27 08:37:47 copying disk images
Okt 27 08:37:47 starting VM 105 on remote node 'kvit-proxmox7'
Okt 27 08:37:50 start remote tunnel
Okt 27 08:37:51 starting storage migration
Okt 27 08:37:51 virtio1: start migration to to nbd:172.22.23.201:60000:exportname=drive-virtio1
drive mirror is starting for drive-virtio1
drive-virtio1: transferred: 0 bytes remaining: 26843545600 bytes total: 26843545600 bytes progression: 0.00 % busy: true ready: false
drive-virtio1: transferred: 116391936 bytes remaining: 26727153664 bytes total: 26843545600 bytes progression: 0.43 % busy: true ready: false
drive-virtio1: transferred: 234881024 bytes remaining: 26608664576 bytes total: 26843545600 bytes progression: 0.88 % busy: true ready: false
....
drive-virtio1: transferred: 26843545600 bytes remaining: 0 bytes total: 26843545600 bytes progression: 100.00 % busy: false ready: true
drive-virtio0: transferred: 53688205312 bytes remaining: 0 bytes total: 53688205312 bytes progression: 100.00 % busy: false ready: true
all mirroring jobs are ready
Okt 27 08:44:45 starting online/live migration on unix:/run/qemu-server/105.migrate
Okt 27 08:44:45 migrate_set_speed: 8589934592
Okt 27 08:44:45 migrate_set_downtime: 0.1
Okt 27 08:44:45 set migration_caps
Okt 27 08:44:45 set cachesize: 429496729
Okt 27 08:44:45 start migrate command to unix:/run/qemu-server/105.migrate
Okt 27 08:44:47 migration status: active (transferred 91587247, remaining 4216516608), total 4312604672)
Okt 27 08:44:47 migration xbzrle cachesize: 268435456 transferred 0 pages 0 cachemiss 0 overflow 0
....
Okt 27 08:46:13 migration status: active (transferred 4104240067, remaining 6045696), total 4312604672)
Okt 27 08:46:13 migration xbzrle cachesize: 268435456 transferred 0 pages 0 cachemiss 22352 overflow 0
query migrate failed: VM 105 not running
Okt 27 08:46:13 query migrate failed: VM 105 not running
query migrate failed: VM 105 not running
Okt 27 08:46:14 query migrate failed: VM 105 not running
query migrate failed: VM 105 not running
Okt 27 08:46:16 query migrate failed: VM 105 not running
query migrate failed: VM 105 not running
Okt 27 08:46:17 query migrate failed: VM 105 not running
query migrate failed: VM 105 not running
Okt 27 08:46:18 query migrate failed: VM 105 not running
query migrate failed: VM 105 not running
Okt 27 08:46:19 query migrate failed: VM 105 not running
Okt 27 08:46:19 ERROR: online migrate failure - too many query migrate failures - aborting
Okt 27 08:46:19 aborting phase 2 - cleanup resources
Okt 27 08:46:19 migrate_cancel
Okt 27 08:46:19 migrate_cancel error: VM 105 not running
drive-virtio1: Cancelling block job
drive-virtio0: Cancelling block job
Okt 27 08:46:19 ERROR: VM 105 not running
Okt 27 08:46:23 ERROR: migration finished with problems (duration 00:08:37)
migration problems
Syslog vom Quellnode:
Oct 27 08:37:46 kvit-proxmox6 qm[21083]: <root@pam> starting task UPID:kvit-proxmox6:0000525C:00465BEE:59F2D43A:qmigrate:105:root@pam:
Oct 27 08:37:48 kvit-proxmox6 pmxcfs[1551]: [status] notice: received log
Oct 27 08:37:50 kvit-proxmox6 pmxcfs[1551]: [status] notice: received log
Oct 27 08:40:49 kvit-proxmox6 pmxcfs[1551]: [status] notice: received log
Oct 27 08:46:13 kvit-proxmox6 kernel: [46619.692398] kvm[21582]: segfault at 48 ip 0000560fa639da3a sp 00007fd8efcfb6c0 error 6 in kvm[560fa5dd7000+7c4000]
Oct 27 08:46:13 kvit-proxmox6 kernel: [46619.794083] vmbr0: port 2(tap105i0) entered disabled state
Oct 27 08:46:13 kvit-proxmox6 kernel: [46619.794288] vmbr0: port 2(tap105i0) entered disabled state
Oct 27 08:46:13 kvit-proxmox6 qm[21084]: VM 105 qmp command failed - VM 105 not running
Oct 27 08:46:14 kvit-proxmox6 qm[21084]: VM 105 qmp command failed - VM 105 not running
Oct 27 08:46:16 kvit-proxmox6 qm[21084]: VM 105 qmp command failed - VM 105 not running
Oct 27 08:46:17 kvit-proxmox6 qm[21084]: VM 105 qmp command failed - VM 105 not running
Oct 27 08:46:18 kvit-proxmox6 qm[21084]: VM 105 qmp command failed - VM 105 not running
Oct 27 08:46:19 kvit-proxmox6 qm[21084]: VM 105 qmp command failed - VM 105 not running
Oct 27 08:46:19 kvit-proxmox6 qm[21084]: VM 105 qmp command failed - VM 105 not running
Oct 27 08:46:19 kvit-proxmox6 qm[21084]: VM 105 qmp command failed - VM 105 not running
Oct 27 08:46:19 kvit-proxmox6 qm[21084]: VM 105 qmp command failed - VM 105 not running
Oct 27 08:46:19 kvit-proxmox6 qm[21084]: VM 105 qmp command failed - VM 105 not running
Oct 27 08:46:22 kvit-proxmox6 pmxcfs[1551]: [status] notice: received log
Oct 27 08:46:22 kvit-proxmox6 pmxcfs[1551]: [status] notice: received log
Oct 27 08:46:23 kvit-proxmox6 qm[21084]: migration problems
Oct 27 08:46:23 kvit-proxmox6 qm[21083]: <root@pam> end task UPID:kvit-proxmox6:0000525C:00465BEE:59F2D43A:qmigrate:105:root@pam: migration problems
Oct 27 08:47:05 kvit-proxmox6 pvedaemon[21762]: start VM 105: UPID:kvit-proxmox6:00005502:00473620:59F2D669:qmstart:105:michael.wagenknecht@kv-it:
Oct 27 08:47:05 kvit-proxmox6 pvedaemon[1647]: <michael.wagenknecht@kv-it> starting task UPID:kvit-proxmox6:00005502:00473620:59F2D669:qmstart:105:michael.wagenknecht@kv-it:
Oct 27 08:47:06 kvit-proxmox6 systemd[1]: Starting 105.scope.
Oct 27 08:47:06 kvit-proxmox6 systemd[1]: Started 105.scope.
Oct 27 08:47:06 kvit-proxmox6 kernel: [46673.308455] device tap105i0 entered promiscuous mode
Oct 27 08:47:06 kvit-proxmox6 kernel: [46673.358405] vmbr0: port 2(tap105i0) entered forwarding state
Oct 27 08:47:06 kvit-proxmox6 kernel: [46673.358431] vmbr0: port 2(tap105i0) entered forwarding state
Oct 27 08:47:07 kvit-proxmox6 pvedaemon[1647]: <michael.wagenknecht@kv-it> end task UPID:kvit-proxmox6:00005502:00473620:59F2D669:qmstart:105:michael.wagenknecht@kv-it: OK
Syslog vom Zielnode:
Oct 27 08:37:46 kvit-proxmox7 pmxcfs[1734]: [status] notice: received log
Oct 27 08:37:48 kvit-proxmox7 qm[20642]: <root@pam> starting task UPID:kvit-proxmox7:000050A5:00445726:59F2D43C:qmstart:105:root@pam:
Oct 27 08:37:48 kvit-proxmox7 qm[20645]: start VM 105: UPID:kvit-proxmox7:000050A5:00445726:59F2D43C:qmstart:105:root@pam:
Oct 27 08:37:48 kvit-proxmox7 systemd[1]: Starting 105.scope.
Oct 27 08:37:48 kvit-proxmox7 systemd[1]: Started 105.scope.
Oct 27 08:37:49 kvit-proxmox7 kernel: [44785.697248] device tap105i0 entered promiscuous mode
Oct 27 08:37:49 kvit-proxmox7 kernel: [44785.714580] vmbr0: port 8(tap105i0) entered forwarding state
Oct 27 08:37:49 kvit-proxmox7 kernel: [44785.714599] vmbr0: port 8(tap105i0) entered forwarding state
Oct 27 08:37:50 kvit-proxmox7 qm[20642]: <root@pam> end task UPID:kvit-proxmox7:000050A5:00445726:59F2D43C:qmstart:105:root@pam: OK
Oct 27 08:40:49 kvit-proxmox7 pmxcfs[1734]: [status] notice: received log
Oct 27 08:41:51 kvit-proxmox7 smartd[1603]: Device: /dev/sdc [SAT], SMART Usage Attribute: 194 Temperature_Celsius changed from 74 to 75
Oct 27 08:41:51 kvit-proxmox7 smartd[1603]: Device: /dev/sdd [SAT], SMART Usage Attribute: 194 Temperature_Celsius changed from 75 to 69
Oct 27 08:42:08 kvit-proxmox7 systemd-timesyncd[1038]: interval/delta/delay/jitter/drift 2048s/-0.000s/0.013s/0.004s/-50ppm
Oct 27 08:46:13 kvit-proxmox7 kernel: [45289.760101] vmbr0: port 8(tap105i0) entered disabled state
Oct 27 08:46:22 kvit-proxmox7 qm[21390]: <root@pam> starting task UPID:kvit-proxmox7:00005391:00451FD7:59F2D63E:qmstop:105:root@pam:
Oct 27 08:46:22 kvit-proxmox7 qm[21393]: stop VM 105: UPID:kvit-proxmox7:00005391:00451FD7:59F2D63E:qmstop:105:root@pam:
Oct 27 08:46:22 kvit-proxmox7 qm[21390]: <root@pam> end task UPID:kvit-proxmox7:00005391:00451FD7:59F2D63E:qmstop:105:root@pam: OK
pveversion -v
proxmox-ve: 4.4-96 (running kernel: 4.4.83-1-pve)
pve-manager: 4.4-18 (running version: 4.4-18/ef2610e8)
pve-kernel-4.4.67-1-pve: 4.4.67-92
pve-kernel-4.4.19-1-pve: 4.4.19-66
pve-kernel-4.4.83-1-pve: 4.4.83-96
lvm2: 2.02.116-pve3
corosync-pve: 2.4.2-2~pve4+1
libqb0: 1.0.1-1
pve-cluster: 4.0-53
qemu-server: 4.0-113
pve-firmware: 1.1-11
libpve-common-perl: 4.0-96
libpve-access-control: 4.0-23
libpve-storage-perl: 4.0-76
pve-libspice-server1: 0.12.8-2
vncterm: 1.3-2
pve-docs: 4.4-4
pve-qemu-kvm: 2.9.0-5~pve4
pve-container: 1.0-101
pve-firewall: 2.0-33
pve-ha-manager: 1.0-41
ksm-control-daemon: 1.2-1
glusterfs-client: 3.5.2-2+deb8u3
lxc-pve: 2.0.7-4
lxcfs: 2.0.6-pve1
criu: 1.6.0-1
novnc-pve: 0.5-9
smartmontools: 6.5+svn4324-1~pve80
zfsutils: 0.6.5.9-pve15~bpo80
Hat jemand eine Idee woran das liegen kann?
Danke,
Michael
seit ein paar Tagen habe ich Probleme bei der Online Migration in unserem Proxmox 4.4 Datacenter. Ich habe mehrere Migrationen probiert. Egal von welchem Proxmox Node zu welchem Proxmox Node, keine Migration läuft durch. Offline Migrationen funktionieren reibungslos. Meiner Meinung nach besteht das Problem seit dem letzten Update von pve-manager/4.4-12 nach pve-manager/4.4-18.
Die Migration habe ich von der Kommandozeile mit folgendem Befehl gestartet: qm migrate 105 kvit-proxmox7 -targetstorage local-storage-ssd -online -with-local-disks
Das Logfile der Migration:
Okt 27 08:37:47 starting migration of VM 105 to node 'kvit-proxmox7' (172.22.23.201)
Okt 27 08:37:47 found local disk 'local-storage-sata:105/vm-105-disk-1.raw' (in current VM config)
Okt 27 08:37:47 found local disk 'local-storage-sata:105/vm-105-disk-2.raw' (in current VM config)
Okt 27 08:37:47 copying disk images
Okt 27 08:37:47 starting VM 105 on remote node 'kvit-proxmox7'
Okt 27 08:37:50 start remote tunnel
Okt 27 08:37:51 starting storage migration
Okt 27 08:37:51 virtio1: start migration to to nbd:172.22.23.201:60000:exportname=drive-virtio1
drive mirror is starting for drive-virtio1
drive-virtio1: transferred: 0 bytes remaining: 26843545600 bytes total: 26843545600 bytes progression: 0.00 % busy: true ready: false
drive-virtio1: transferred: 116391936 bytes remaining: 26727153664 bytes total: 26843545600 bytes progression: 0.43 % busy: true ready: false
drive-virtio1: transferred: 234881024 bytes remaining: 26608664576 bytes total: 26843545600 bytes progression: 0.88 % busy: true ready: false
....
drive-virtio1: transferred: 26843545600 bytes remaining: 0 bytes total: 26843545600 bytes progression: 100.00 % busy: false ready: true
drive-virtio0: transferred: 53688205312 bytes remaining: 0 bytes total: 53688205312 bytes progression: 100.00 % busy: false ready: true
all mirroring jobs are ready
Okt 27 08:44:45 starting online/live migration on unix:/run/qemu-server/105.migrate
Okt 27 08:44:45 migrate_set_speed: 8589934592
Okt 27 08:44:45 migrate_set_downtime: 0.1
Okt 27 08:44:45 set migration_caps
Okt 27 08:44:45 set cachesize: 429496729
Okt 27 08:44:45 start migrate command to unix:/run/qemu-server/105.migrate
Okt 27 08:44:47 migration status: active (transferred 91587247, remaining 4216516608), total 4312604672)
Okt 27 08:44:47 migration xbzrle cachesize: 268435456 transferred 0 pages 0 cachemiss 0 overflow 0
....
Okt 27 08:46:13 migration status: active (transferred 4104240067, remaining 6045696), total 4312604672)
Okt 27 08:46:13 migration xbzrle cachesize: 268435456 transferred 0 pages 0 cachemiss 22352 overflow 0
query migrate failed: VM 105 not running
Okt 27 08:46:13 query migrate failed: VM 105 not running
query migrate failed: VM 105 not running
Okt 27 08:46:14 query migrate failed: VM 105 not running
query migrate failed: VM 105 not running
Okt 27 08:46:16 query migrate failed: VM 105 not running
query migrate failed: VM 105 not running
Okt 27 08:46:17 query migrate failed: VM 105 not running
query migrate failed: VM 105 not running
Okt 27 08:46:18 query migrate failed: VM 105 not running
query migrate failed: VM 105 not running
Okt 27 08:46:19 query migrate failed: VM 105 not running
Okt 27 08:46:19 ERROR: online migrate failure - too many query migrate failures - aborting
Okt 27 08:46:19 aborting phase 2 - cleanup resources
Okt 27 08:46:19 migrate_cancel
Okt 27 08:46:19 migrate_cancel error: VM 105 not running
drive-virtio1: Cancelling block job
drive-virtio0: Cancelling block job
Okt 27 08:46:19 ERROR: VM 105 not running
Okt 27 08:46:23 ERROR: migration finished with problems (duration 00:08:37)
migration problems
Syslog vom Quellnode:
Oct 27 08:37:46 kvit-proxmox6 qm[21083]: <root@pam> starting task UPID:kvit-proxmox6:0000525C:00465BEE:59F2D43A:qmigrate:105:root@pam:
Oct 27 08:37:48 kvit-proxmox6 pmxcfs[1551]: [status] notice: received log
Oct 27 08:37:50 kvit-proxmox6 pmxcfs[1551]: [status] notice: received log
Oct 27 08:40:49 kvit-proxmox6 pmxcfs[1551]: [status] notice: received log
Oct 27 08:46:13 kvit-proxmox6 kernel: [46619.692398] kvm[21582]: segfault at 48 ip 0000560fa639da3a sp 00007fd8efcfb6c0 error 6 in kvm[560fa5dd7000+7c4000]
Oct 27 08:46:13 kvit-proxmox6 kernel: [46619.794083] vmbr0: port 2(tap105i0) entered disabled state
Oct 27 08:46:13 kvit-proxmox6 kernel: [46619.794288] vmbr0: port 2(tap105i0) entered disabled state
Oct 27 08:46:13 kvit-proxmox6 qm[21084]: VM 105 qmp command failed - VM 105 not running
Oct 27 08:46:14 kvit-proxmox6 qm[21084]: VM 105 qmp command failed - VM 105 not running
Oct 27 08:46:16 kvit-proxmox6 qm[21084]: VM 105 qmp command failed - VM 105 not running
Oct 27 08:46:17 kvit-proxmox6 qm[21084]: VM 105 qmp command failed - VM 105 not running
Oct 27 08:46:18 kvit-proxmox6 qm[21084]: VM 105 qmp command failed - VM 105 not running
Oct 27 08:46:19 kvit-proxmox6 qm[21084]: VM 105 qmp command failed - VM 105 not running
Oct 27 08:46:19 kvit-proxmox6 qm[21084]: VM 105 qmp command failed - VM 105 not running
Oct 27 08:46:19 kvit-proxmox6 qm[21084]: VM 105 qmp command failed - VM 105 not running
Oct 27 08:46:19 kvit-proxmox6 qm[21084]: VM 105 qmp command failed - VM 105 not running
Oct 27 08:46:19 kvit-proxmox6 qm[21084]: VM 105 qmp command failed - VM 105 not running
Oct 27 08:46:22 kvit-proxmox6 pmxcfs[1551]: [status] notice: received log
Oct 27 08:46:22 kvit-proxmox6 pmxcfs[1551]: [status] notice: received log
Oct 27 08:46:23 kvit-proxmox6 qm[21084]: migration problems
Oct 27 08:46:23 kvit-proxmox6 qm[21083]: <root@pam> end task UPID:kvit-proxmox6:0000525C:00465BEE:59F2D43A:qmigrate:105:root@pam: migration problems
Oct 27 08:47:05 kvit-proxmox6 pvedaemon[21762]: start VM 105: UPID:kvit-proxmox6:00005502:00473620:59F2D669:qmstart:105:michael.wagenknecht@kv-it:
Oct 27 08:47:05 kvit-proxmox6 pvedaemon[1647]: <michael.wagenknecht@kv-it> starting task UPID:kvit-proxmox6:00005502:00473620:59F2D669:qmstart:105:michael.wagenknecht@kv-it:
Oct 27 08:47:06 kvit-proxmox6 systemd[1]: Starting 105.scope.
Oct 27 08:47:06 kvit-proxmox6 systemd[1]: Started 105.scope.
Oct 27 08:47:06 kvit-proxmox6 kernel: [46673.308455] device tap105i0 entered promiscuous mode
Oct 27 08:47:06 kvit-proxmox6 kernel: [46673.358405] vmbr0: port 2(tap105i0) entered forwarding state
Oct 27 08:47:06 kvit-proxmox6 kernel: [46673.358431] vmbr0: port 2(tap105i0) entered forwarding state
Oct 27 08:47:07 kvit-proxmox6 pvedaemon[1647]: <michael.wagenknecht@kv-it> end task UPID:kvit-proxmox6:00005502:00473620:59F2D669:qmstart:105:michael.wagenknecht@kv-it: OK
Syslog vom Zielnode:
Oct 27 08:37:46 kvit-proxmox7 pmxcfs[1734]: [status] notice: received log
Oct 27 08:37:48 kvit-proxmox7 qm[20642]: <root@pam> starting task UPID:kvit-proxmox7:000050A5:00445726:59F2D43C:qmstart:105:root@pam:
Oct 27 08:37:48 kvit-proxmox7 qm[20645]: start VM 105: UPID:kvit-proxmox7:000050A5:00445726:59F2D43C:qmstart:105:root@pam:
Oct 27 08:37:48 kvit-proxmox7 systemd[1]: Starting 105.scope.
Oct 27 08:37:48 kvit-proxmox7 systemd[1]: Started 105.scope.
Oct 27 08:37:49 kvit-proxmox7 kernel: [44785.697248] device tap105i0 entered promiscuous mode
Oct 27 08:37:49 kvit-proxmox7 kernel: [44785.714580] vmbr0: port 8(tap105i0) entered forwarding state
Oct 27 08:37:49 kvit-proxmox7 kernel: [44785.714599] vmbr0: port 8(tap105i0) entered forwarding state
Oct 27 08:37:50 kvit-proxmox7 qm[20642]: <root@pam> end task UPID:kvit-proxmox7:000050A5:00445726:59F2D43C:qmstart:105:root@pam: OK
Oct 27 08:40:49 kvit-proxmox7 pmxcfs[1734]: [status] notice: received log
Oct 27 08:41:51 kvit-proxmox7 smartd[1603]: Device: /dev/sdc [SAT], SMART Usage Attribute: 194 Temperature_Celsius changed from 74 to 75
Oct 27 08:41:51 kvit-proxmox7 smartd[1603]: Device: /dev/sdd [SAT], SMART Usage Attribute: 194 Temperature_Celsius changed from 75 to 69
Oct 27 08:42:08 kvit-proxmox7 systemd-timesyncd[1038]: interval/delta/delay/jitter/drift 2048s/-0.000s/0.013s/0.004s/-50ppm
Oct 27 08:46:13 kvit-proxmox7 kernel: [45289.760101] vmbr0: port 8(tap105i0) entered disabled state
Oct 27 08:46:22 kvit-proxmox7 qm[21390]: <root@pam> starting task UPID:kvit-proxmox7:00005391:00451FD7:59F2D63E:qmstop:105:root@pam:
Oct 27 08:46:22 kvit-proxmox7 qm[21393]: stop VM 105: UPID:kvit-proxmox7:00005391:00451FD7:59F2D63E:qmstop:105:root@pam:
Oct 27 08:46:22 kvit-proxmox7 qm[21390]: <root@pam> end task UPID:kvit-proxmox7:00005391:00451FD7:59F2D63E:qmstop:105:root@pam: OK
pveversion -v
proxmox-ve: 4.4-96 (running kernel: 4.4.83-1-pve)
pve-manager: 4.4-18 (running version: 4.4-18/ef2610e8)
pve-kernel-4.4.67-1-pve: 4.4.67-92
pve-kernel-4.4.19-1-pve: 4.4.19-66
pve-kernel-4.4.83-1-pve: 4.4.83-96
lvm2: 2.02.116-pve3
corosync-pve: 2.4.2-2~pve4+1
libqb0: 1.0.1-1
pve-cluster: 4.0-53
qemu-server: 4.0-113
pve-firmware: 1.1-11
libpve-common-perl: 4.0-96
libpve-access-control: 4.0-23
libpve-storage-perl: 4.0-76
pve-libspice-server1: 0.12.8-2
vncterm: 1.3-2
pve-docs: 4.4-4
pve-qemu-kvm: 2.9.0-5~pve4
pve-container: 1.0-101
pve-firewall: 2.0-33
pve-ha-manager: 1.0-41
ksm-control-daemon: 1.2-1
glusterfs-client: 3.5.2-2+deb8u3
lxc-pve: 2.0.7-4
lxcfs: 2.0.6-pve1
criu: 1.6.0-1
novnc-pve: 0.5-9
smartmontools: 6.5+svn4324-1~pve80
zfsutils: 0.6.5.9-pve15~bpo80
Hat jemand eine Idee woran das liegen kann?
Danke,
Michael