[SOLVED] after node failure, local storage VM cannot be migrated

[kifeo]

Hi everyone,

I had a node with a VM that had a local storage, which had a failure and shut down for some time.
Due to the HA, it was migrated to another node, and ended up in failed state due to the local storage not migrated.

Now the node is up again.

I've put the HA state to disabled for this node in order to workaround the failed state.
I would like to migrate the VM back again to that not-failed-anymore node, but I have this issue :

task started by HA resource agent
2021-01-27 09:05:12 starting migration of VM 4010 to node 'proxmox5' (10.0.5.15)
2021-01-27 09:05:13 found local disk 'local:4010/vm-4010-disk-0.qcow2' (in current VM config)
2021-01-27 09:05:13 copying local disk images
failed to stat '/var/lib/vz/images/4010/vm-4010-disk-0.qcow2'
Use of uninitialized value $format in string eq at /usr/share/perl5/PVE/Storage/Plugin.pm line 1376.
2021-01-27 09:05:13 ERROR: Failed to sync data - storage migration for 'local:4010/vm-4010-disk-0.qcow2' to storage 'local' failed - volume 'local:4010/vm-4010-disk-0.qcow2' does not exist
2021-01-27 09:05:13 aborting phase 1 - cleanup resources
2021-01-27 09:05:13 ERROR: found stale volume copy 'local:4010/vm-4010-disk-0.qcow2' on node 'proxmox5'
2021-01-27 09:05:13 ERROR: migration aborted (duration 00:00:01): Failed to sync data - storage migration for 'local:4010/vm-4010-disk-0.qcow2' to storage 'local' failed - volume 'local:4010/vm-4010-disk-0.qcow2' does not exist
TASK ERROR: migration aborted

Does someone have an idea on how to migrate back this vm ?
Also it seems it needs a fix due to the uninitialized value message.

Thanks !
Thomas.

 

[fiona]

Hi,
note that this is exactly the reason why HA shouldn't be used with local storage, see HA requirements.

If all the disks are on the old node, you can manually move the configuration file from /etc/pve/nodes/<source_node>/qemu-server/<ID>.conf to /etc/pve/nodes/<target_node>/qemu-server/<ID>.conf. An alternative is to remove the lines for the disks in the configuration file and migrate back the VM. Then do a qm rescan --vmid <ID> and attach the disks again.

If you could share your pveversion -v, I'll be able to find the reason for the uninitialized warning more easily, thanks!

 

[kifeo]

Hello Fabian,

I'm sorry, indeed it was on a shared storage, and I moved it to local and forgot to remove the HA.

here is the pveversion -v :

proxmox-ve: 6.3-1 (running kernel: 5.4.78-2-pve)
pve-manager: 6.3-3 (running version: 6.3-3/eee5f901)
pve-kernel-5.4: 6.3-3
pve-kernel-helper: 6.3-3
pve-kernel-5.3: 6.1-6
pve-kernel-5.4.78-2-pve: 5.4.78-2
pve-kernel-5.4.78-1-pve: 5.4.78-1
pve-kernel-5.4.65-1-pve: 5.4.65-1
pve-kernel-5.3.18-3-pve: 5.3.18-3
pve-kernel-5.3.10-1-pve: 5.3.10-1
ceph: 15.2.8-pve2
ceph-fuse: 15.2.8-pve2
corosync: 3.0.4-pve1
criu: 3.11-3
glusterfs-client: 5.5-3
ifupdown: residual config
ifupdown2: 3.0.0-1+pve3
ksm-control-daemon: 1.3-1
libjs-extjs: 6.0.1-10
libknet1: 1.16-pve1
libproxmox-acme-perl: 1.0.7
libproxmox-backup-qemu0: 1.0.2-1
libpve-access-control: 6.1-3
libpve-apiclient-perl: 3.1-3
libpve-common-perl: 6.3-2
libpve-guest-common-perl: 3.1-4
libpve-http-server-perl: 3.1-1
libpve-storage-perl: 6.3-4
libqb0: 1.0.5-1
libspice-server1: 0.14.2-4~pve6+1
lvm2: 2.03.02-pve4
lxc-pve: 4.0.3-1
lxcfs: 4.0.6-pve1
novnc-pve: 1.1.0-1
proxmox-backup-client: 1.0.6-1
proxmox-mini-journalreader: 1.1-1
proxmox-widget-toolkit: 2.4-3
pve-cluster: 6.2-1
pve-container: 3.3-2
pve-docs: 6.3-1
pve-edk2-firmware: 2.20200531-1
pve-firewall: 4.1-3
pve-firmware: 3.1-3
pve-ha-manager: 3.1-1
pve-i18n: 2.2-2
pve-qemu-kvm: 5.1.0-8
pve-xtermjs: 4.7.0-3
qemu-server: 6.3-3
smartmontools: 7.1-pve2
spiceterm: 3.1-1
vncterm: 1.6-2
zfsutils-linux: 0.8.5-pve1

Best Regards

 

[kifeo]

Many thanks, the move operation did resolve the situation.

 

[fiona]

Many thanks, the move operation did resolve the situation.

Glad to hear it worked! I did find the cause of the uninitialized value problem, and luckily it's only cosmetic (cannot lead to anything bad), but I might be able to fix it as part of refactoring of that code I'm working on.

 

[kifeo]

Hi Fabian,

It happened again, too bad the GUI does not allow it.
Thanks again for the help.
Regards,
Kifeo