Currently we are trying to live migrate a VM to another server within the same cluster.
The first migration successfully migrated all the attached disks and got a hangup at the "VM-state" migration step.
After 15 minutes of no progress I pressed the "Stop" button to abort the migration.
Now when we try to start a new migration the disks also successfully migrate to the new hypervisor but it still fails on the "VM-state" migration step.
But now the error states there is already migration process in progress.
I already view the "ps" output on the hypervisors but I cannot seem to find a any running process that has a reference to the VM migration.
The first migration successfully migrated all the attached disks and got a hangup at the "VM-state" migration step.
After 15 minutes of no progress I pressed the "Stop" button to abort the migration.
Code:
2023-12-18 12:01:38 starting online/live migration on tcp:10.40.4.72:60000
2023-12-18 12:01:38 set migration capabilities
2023-12-18 12:01:38 migration downtime limit: 100 ms
2023-12-18 12:01:38 migration cachesize: 2.0 GiB
2023-12-18 12:01:38 set migration parameters
2023-12-18 12:01:38 start migrate command to tcp:10.40.4.72:60000
2023-12-18 12:19:07 ERROR: online migrate failure - interrupted by signal
2023-12-18 12:19:07 aborting phase 2 - cleanup resources
2023-12-18 12:19:07 migrate_cancel
drive-scsi0: Cancelling block job
drive-scsi5: Cancelling block job
drive-scsi4: Cancelling block job
drive-scsi2: Cancelling block job
drive-scsi6: Cancelling block job
drive-scsi1: Cancelling block job
drive-scsi3: Cancelling block job
drive-scsi0: Done.
drive-scsi5: Done.
drive-scsi4: Done.
drive-scsi2: Done.
drive-scsi6: Done.
drive-scsi1: Done.
drive-scsi3: Done.Now when we try to start a new migration the disks also successfully migrate to the new hypervisor but it still fails on the "VM-state" migration step.
But now the error states there is already migration process in progress.
Code:
2023-12-18 12:30:49 starting online/live migration on tcp:10.40.4.72:60000
2023-12-18 12:30:49 set migration capabilities
VM 162 qmp command 'migrate-set-capabilities' failed - There's a migration process in progress
2023-12-18 12:30:49 migration downtime limit: 100 ms
2023-12-18 12:30:49 migration cachesize: 2.0 GiB
2023-12-18 12:30:49 set migration parameters
2023-12-18 12:30:49 start migrate command to tcp:10.40.4.72:60000
2023-12-18 12:30:49 migrate uri => tcp:10.40.4.72:60000 failed: VM 162 qmp command 'migrate' failed - There's a migration process in progress
2023-12-18 12:30:50 ERROR: online migrate failure - VM 162 qmp command 'migrate' failed - There's a migration process in progress
2023-12-18 12:30:50 aborting phase 2 - cleanup resources
2023-12-18 12:30:50 migrate_cancel
drive-scsi0: Cancelling block job
drive-scsi4: Cancelling block job
drive-scsi2: Cancelling block job
drive-scsi5: Cancelling block job
drive-scsi6: Cancelling block job
drive-scsi1: Cancelling block job
drive-scsi3: Cancelling block job
drive-scsi0: Done.
drive-scsi4: Done.
drive-scsi2: Done.
drive-scsi5: Done.
drive-scsi6: Done.
drive-scsi1: Done.
drive-scsi3: Done.
2023-12-18 12:31:00 ERROR: migration finished with problems (duration 00:03:50)
TASK ERROR: migration problemsSource hypervisor
Destination hypervisor
Code:
proxmox-ve: 8.0.2 (running kernel: 6.2.16-15-pve)
pve-manager: 8.0.4 (running version: 8.0.4/d258a813cfa6b390)
pve-kernel-6.2: 8.0.5
proxmox-kernel-helper: 8.0.3
proxmox-kernel-6.2.16-15-pve: 6.2.16-15
proxmox-kernel-6.2: 6.2.16-15
proxmox-kernel-6.2.16-12-pve: 6.2.16-12
proxmox-kernel-6.2.16-6-pve: 6.2.16-7
pve-kernel-6.2.16-3-pve: 6.2.16-3
ceph-fuse: 17.2.6-pve1+3
corosync: 3.1.7-pve3
criu: 3.17.1-2
glusterfs-client: 10.3-5
ifupdown2: 3.2.0-1+pmx4
ksm-control-daemon: 1.4-1
libjs-extjs: 7.0.0-4
libknet1: 1.25-pve1
libproxmox-acme-perl: 1.4.6
libproxmox-backup-qemu0: 1.4.0
libproxmox-rs-perl: 0.3.0
libpve-access-control: 8.0.4
libpve-apiclient-perl: 3.3.1
libpve-common-perl: 8.0.7
libpve-guest-common-perl: 5.0.3
libpve-http-server-perl: 5.0.4
libpve-rs-perl: 0.8.4
libpve-storage-perl: 8.0.2
libspice-server1: 0.15.1-1
lvm2: 2.03.16-2
lxc-pve: 5.0.2-4
lxcfs: 5.0.3-pve3
novnc-pve: 1.4.0-2
proxmox-backup-client: 3.0.2-1
proxmox-backup-file-restore: 3.0.2-1
proxmox-kernel-helper: 8.0.3
proxmox-mail-forward: 0.2.0
proxmox-mini-journalreader: 1.4.0
proxmox-widget-toolkit: 4.0.9
pve-cluster: 8.0.2
pve-container: 5.0.4
pve-docs: 8.0.5
pve-edk2-firmware: 3.20230228-4
pve-firewall: 5.0.3
pve-firmware: 3.8-2
pve-ha-manager: 4.0.2
pve-i18n: 3.0.7
pve-qemu-kvm: 8.0.2-6
pve-xtermjs: 4.16.0-3
qemu-server: 8.0.7
smartmontools: 7.3-pve1
spiceterm: 3.3.0
swtpm: 0.8.0+pve1
vncterm: 1.8.0
zfsutils-linux: 2.1.13-pve1
Code:
proxmox-ve: 8.1.0 (running kernel: 6.5.11-7-pve)
pve-manager: 8.1.3 (running version: 8.1.3/b46aac3b42da5d15)
proxmox-kernel-helper: 8.1.0
proxmox-kernel-6.5: 6.5.11-7
proxmox-kernel-6.5.11-7-pve-signed: 6.5.11-7
proxmox-kernel-6.5.11-4-pve-signed: 6.5.11-4
ceph-fuse: 17.2.7-pve1
corosync: 3.1.7-pve3
criu: 3.17.1-2
glusterfs-client: 10.3-5
ifupdown2: 3.2.0-1+pmx7
ksm-control-daemon: 1.4-1
libjs-extjs: 7.0.0-4
libknet1: 1.28-pve1
libproxmox-acme-perl: 1.5.0
libproxmox-backup-qemu0: 1.4.1
libproxmox-rs-perl: 0.3.1
libpve-access-control: 8.0.7
libpve-apiclient-perl: 3.3.1
libpve-common-perl: 8.1.0
libpve-guest-common-perl: 5.0.6
libpve-http-server-perl: 5.0.5
libpve-network-perl: 0.9.5
libpve-rs-perl: 0.8.7
libpve-storage-perl: 8.0.5
libspice-server1: 0.15.1-1
lvm2: 2.03.16-2
lxc-pve: 5.0.2-4
lxcfs: 5.0.3-pve4
novnc-pve: 1.4.0-3
proxmox-backup-client: 3.1.2-1
proxmox-backup-file-restore: 3.1.2-1
proxmox-kernel-helper: 8.1.0
proxmox-mail-forward: 0.2.2
proxmox-mini-journalreader: 1.4.0
proxmox-offline-mirror-helper: 0.6.3
proxmox-widget-toolkit: 4.1.3
pve-cluster: 8.0.5
pve-container: 5.0.8
pve-docs: 8.1.3
pve-edk2-firmware: 4.2023.08-2
pve-firewall: 5.0.3
pve-firmware: 3.9-1
pve-ha-manager: 4.0.3
pve-i18n: 3.1.4
pve-qemu-kvm: 8.1.2-4
pve-xtermjs: 5.3.0-2
qemu-server: 8.0.10
smartmontools: 7.3-pve1
spiceterm: 3.3.0
swtpm: 0.8.0+pve1
vncterm: 1.8.0
zfsutils-linux: 2.2.2-pve1