can not migrate VMs

[zunder1990]

I have a 3 node cluster all running pve-manager/7.3-6/723bb6ec (running kernel: 5.15.85-1-pve). All nodes use local non shared storage. When I try to migrate vms from two of the nodes to node called rackserver it all fails at step right after, drive mirror is starting for drive-scsi0. I can migrate between the other two nodes with no errors. The started about the same time I updated the whole cluster to 7.3 from 7.1 and removed a node on a really old version and added it back to cluster after a wipe a fresh install.

2023-03-10 13:59:11 starting migration of VM 108 to node 'rackserver' (10.10.0.13)
2023-03-10 13:59:11 found local disk 'ssd-vmhost2:108/vm-108-disk-0.raw' (in current VM config)
2023-03-10 13:59:11 starting VM 108 on remote node 'rackserver'
2023-03-10 13:59:12 [rackserver] Use of uninitialized value $PVE::JSONSchema::PVE_TAG_RE in concatenation (.) or string at /usr/share/perl5/PVE/DataCenterConfig.pm line 161.
2023-03-10 13:59:12 [rackserver] Use of uninitialized value $PVE::JSONSchema::PVE_TAG_RE in concatenation (.) or string at /usr/share/perl5/PVE/DataCenterConfig.pm line 196.
2023-03-10 13:59:12 [rackserver] Use of uninitialized value $PVE::JSONSchema::PVE_TAG_RE in concatenation (.) or string at /usr/share/perl5/PVE/DataCenterConfig.pm line 196.
2023-03-10 13:59:12 [rackserver] Use of uninitialized value $PVE::JSONSchema::PVE_TAG_RE in concatenation (.) or string at /usr/share/perl5/PVE/DataCenterConfig.pm line 222.
2023-03-10 13:59:12 [rackserver] Use of uninitialized value $PVE::JSONSchema::PVE_TAG_RE in concatenation (.) or string at /usr/share/perl5/PVE/DataCenterConfig.pm line 222.
2023-03-10 13:59:15 [rackserver] Use of uninitialized value $PVE::JSONSchema::PVE_TAG_RE in concatenation (.) or string at /usr/share/perl5/PVE/DataCenterConfig.pm line 161.
2023-03-10 13:59:15 [rackserver] Use of uninitialized value $PVE::JSONSchema::PVE_TAG_RE in concatenation (.) or string at /usr/share/perl5/PVE/DataCenterConfig.pm line 196.
2023-03-10 13:59:15 [rackserver] Use of uninitialized value $PVE::JSONSchema::PVE_TAG_RE in concatenation (.) or string at /usr/share/perl5/PVE/DataCenterConfig.pm line 196.
2023-03-10 13:59:15 [rackserver] Use of uninitialized value $PVE::JSONSchema::PVE_TAG_RE in concatenation (.) or string at /usr/share/perl5/PVE/DataCenterConfig.pm line 222.
2023-03-10 13:59:15 [rackserver] Use of uninitialized value $PVE::JSONSchema::PVE_TAG_RE in concatenation (.) or string at /usr/share/perl5/PVE/DataCenterConfig.pm line 222.
2023-03-10 13:59:15 volume 'ssd-vmhost2:108/vm-108-disk-0.raw' is 'ssd1:vm-108-disk-0' on the target
2023-03-10 13:59:15 start remote tunnel
Use of uninitialized value $PVE::JSONSchema::PVE_TAG_RE in concatenation (.) or string at /usr/share/perl5/PVE/DataCenterConfig.pm line 161.
Use of uninitialized value $PVE::JSONSchema::PVE_TAG_RE in concatenation (.) or string at /usr/share/perl5/PVE/DataCenterConfig.pm line 196.
Use of uninitialized value $PVE::JSONSchema::PVE_TAG_RE in concatenation (.) or string at /usr/share/perl5/PVE/DataCenterConfig.pm line 196.
Use of uninitialized value $PVE::JSONSchema::PVE_TAG_RE in concatenation (.) or string at /usr/share/perl5/PVE/DataCenterConfig.pm line 222.
Use of uninitialized value $PVE::JSONSchema::PVE_TAG_RE in concatenation (.) or string at /usr/share/perl5/PVE/DataCenterConfig.pm line 222.
2023-03-10 13:59:17 ssh tunnel ver 1
2023-03-10 13:59:17 starting storage migration
2023-03-10 13:59:17 scsi0: start migration to nbd:unix:/run/qemu-server/108_nbd.migrate:exportname=drive-scsi0
drive mirror is starting for drive-scsi0
drive-scsi0: Cancelling block job
drive-scsi0: Done.
2023-03-10 14:00:17 ERROR: online migrate failure - block job (mirror) error: interrupted by signal
2023-03-10 14:00:17 aborting phase 2 - cleanup resources
2023-03-10 14:00:17 migrate_cancel

same error

root@rackserver:/etc/pve# pvesm status
Use of uninitialized value $PVE::JSONSchema::PVE_TAG_RE in concatenation (.) or string at /usr/share/perl5/PVE/DataCenterConfig.pm line 161, <DATA> line 960.
Use of uninitialized value $PVE::JSONSchema::PVE_TAG_RE in concatenation (.) or string at /usr/share/perl5/PVE/DataCenterConfig.pm line 196, <DATA> line 960.
Use of uninitialized value $PVE::JSONSchema::PVE_TAG_RE in concatenation (.) or string at /usr/share/perl5/PVE/DataCenterConfig.pm line 196, <DATA> line 960.
Use of uninitialized value $PVE::JSONSchema::PVE_TAG_RE in concatenation (.) or string at /usr/share/perl5/PVE/DataCenterConfig.pm line 222, <DATA> line 960.
Use of uninitialized value $PVE::JSONSchema::PVE_TAG_RE in concatenation (.) or string at /usr/share/perl5/PVE/DataCenterConfig.pm line 222, <DATA> line 960.
Name                   Type     Status           Total            Used       Available        %
bigdata-backup          dir     active    129872072896     78615405440     51256667456   60.53%
hdd-vmhost2             dir   disabled               0               0               0      N/A
local                   dir     active        98497780        17755408        75692824   18.03%
local-lvm           lvmthin     active       594092032               0       594092032    0.00%
ssd-mirror          zfspool     active      1804599296       194697718      1609901577   10.79%
ssd-mirror-dell     zfspool   disabled               0               0               0      N/A
ssd-vmhost2             dir   disabled               0               0               0      N/A
ssd1                zfspool     active       377946112       170849508       207096604   45.20%

same error

root@rackserver:/etc/pve# pveversion
Use of uninitialized value $PVE::JSONSchema::PVE_TAG_RE in concatenation (.) or string at /usr/share/perl5/PVE/DataCenterConfig.pm line 161.
Use of uninitialized value $PVE::JSONSchema::PVE_TAG_RE in concatenation (.) or string at /usr/share/perl5/PVE/DataCenterConfig.pm line 196.
Use of uninitialized value $PVE::JSONSchema::PVE_TAG_RE in concatenation (.) or string at /usr/share/perl5/PVE/DataCenterConfig.pm line 196.
Use of uninitialized value $PVE::JSONSchema::PVE_TAG_RE in concatenation (.) or string at /usr/share/perl5/PVE/DataCenterConfig.pm line 222.
Use of uninitialized value $PVE::JSONSchema::PVE_TAG_RE in concatenation (.) or string at /usr/share/perl5/PVE/DataCenterConfig.pm line 222.
pve-manager/7.2-3/c743d6c1 (running kernel: 5.15.30-2-pve)

And in syslog

Mar 10 14:17:40 rackserver qmeventd[3876]: read: Connection reset by peer
Mar 10 14:17:40 rackserver systemd[1]: 108.scope: Succeeded.
Mar 10 14:17:40 rackserver systemd[1]: 108.scope: Consumed 3.706s CPU time.
Mar 10 14:17:40 rackserver qmeventd[943189]: Use of uninitialized value $PVE::JSONSchema::PVE_TAG_RE in concatenation (.) or string at /usr/share/perl5/PVE/DataCenterConfig.pm line 161.
Mar 10 14:17:40 rackserver qmeventd[943189]: Use of uninitialized value $PVE::JSONSchema::PVE_TAG_RE in concatenation (.) or string at /usr/share/perl5/PVE/DataCenterConfig.pm line 196.
Mar 10 14:17:40 rackserver qmeventd[943189]: Use of uninitialized value $PVE::JSONSchema::PVE_TAG_RE in concatenation (.) or string at /usr/share/perl5/PVE/DataCenterConfig.pm line 196.
Mar 10 14:17:40 rackserver qmeventd[943189]: Use of uninitialized value $PVE::JSONSchema::PVE_TAG_RE in concatenation (.) or string at /usr/share/perl5/PVE/DataCenterConfig.pm line 222.
Mar 10 14:17:40 rackserver qmeventd[943189]: Use of uninitialized value $PVE::JSONSchema::PVE_TAG_RE in concatenation (.) or string at /usr/share/perl5/PVE/DataCenterConfig.pm line 222.

 

[Neobin]

I have a 3 node cluster all running pve-manager/7.3-6/723bb6ec (running kernel: 5.15.85-1-pve).

Looks not so:

pve-manager/7.2-3/c743d6c1 (running kernel: 5.15.30-2-pve)

Properly update all nodes and reboot them one after the other afterwards; so that they are all on the same versions and booted kernel.

If your nodes do not have active subscriptions, see here how to:
https://forum.proxmox.com/threads/reboot-occurs-randomly.119840/post-520320

 

[zunder1990]

I thought that the host was update to already but I was wrong. most have gotten mixed up since the host was also doing other stuff too. Host is fully patched and rebooted. errors have gone away during the live migration