[SOLVED] Unable to move VM back to failed node after recovery because local storage not available on current node

[proxale]

I have a PVE 7.2-7 HA cluster with Ceph. Some VMs were originally on local ZFS storage on Node A which was our first node before Ceph was installed.

They were migrated to Ceph storage eventually but unfortunately I overlooked that there were some snapshots and unused disks still on Node A local storage.

Node A was down and fenced. VMs were auto migrated to other nodes on the cluster.
After recovery of Node A, we are unable to migrate the VM from their current nodes back to Node A.

When attempting to do so, Proxmox complains that it cannot find the mount point in Node B/C/D/E whichever the VM happens to be on.
But this is expected, the local storage mount point never existed on those nodes.

I cannot delete the snapshots nor remove the Unused disk for the same reason.

I've tried adding the local storage via DataCenter -> Storage -> Edit, to the nodes that don't actually have them but this didn't help.

How can we move those VMs back to Node A? Do we have to trigger a fenced event by powering off Node B/C/D/E one by one so that Proxmox will ignore such constraints like when Node A went down?

 

[fabian]

did you try ha-manager crm-command relocate <SID> <OLD_NODE> (where SID is the HA ID of your guest)? you might want to shut it the guest down cleanly before

 

[proxale]

did you try ha-manager crm-command relocate <SID> <OLD_NODE> (where SID is the HA ID of your guest)? you might want to shut it the guest down cleanly before

I shutdown the VM cleanly. Executed the command on the old node as well as on the new node.
No output on the command line but from web UI, I can see the task appear in the log and fails with the same reason.
Can I simply move a conf file or something from <CURRENT_NODE> to <OLD_NODE>?

Tried to do this by disabling HA for the VM but obviously not allowed.
Also set the priority node to OLD_NODE but same error.

 

[fabian]

yes, after you shut down the guest and ensuring no other actions like backups/.. are running, you can move the config file manually and should then be able to cleanup the local references/snapshots/..

 

[proxale]

@fabian Thanks for the confirmation. I was able to move the VM back to OLD_NODE then delete the snapshots/disks.

To verify that everything is now OK, I use the GUI to migrate the shutdown VM to NEW_NODE and then tried to migrate it back to OLD_NODE. Since there are no longer any snapshot or Unused disk, I expected this to work. However, Proxmox UI still throws

Error: migration aborted
in output log:

zfs error: cannot open 'local_xxx_zfs': no such pool

ERROR: Problem found while scanning volumes - could not activate storage 'local_xxx_zfs', zfs error: cannot import 'local_xxx_zfs': no such pool available

aborting phase 1 - cleanup resources

ERROR: migration aborted (duration 00:00:00): Problem found while scanning volumes - could not activate storage 'local_xxx_zfs', zfs error: cannot import 'local_xxx_zfs': no such pool available
TASK ERROR: migration aborted

I've verified on NEW_NODE:/etc/pve/qemu-server/<VM_ID>.conf that there are no reference to that storage.
Is the information cached somewhere or is there another conf file I need to deal with?

 

[fabian]

that sounds like the storage.cfg file is wrong - local storages that are not available on all nodes need to be restricted accordingly (and local storages that are not available at all should be disabled or removed).

 

[proxale]

Overlooked reporting back on this issue. The root cause was actually in the message but misunderstood due to the context of me using ZFS only on that node. ProxMox complaining about unable to activate local_zfs is because the disk for ZFS was dead. Once the disk issue was fixed, the migration issue also went away.