ignore crm command - service already on target node:

[Adam Carheden]

How do I recover from a failed migration?

-= Current status =-

# qm migrate 100 node2 --online
(from daemon.log)
Mar 22 17:07:00 stout pve-ha-crm[2187]: ignore crm command - service already on target node: migrate vm:100 node2

The VM is still running fine. `ps` confirms kvm is running on node1 but not on node2.

# ha-manager status
quorum OK
master node1 (active, Wed Mar 22 17:14:11 2017)
lrm node2 (active, Wed Mar 22 17:14:11 2017)
lrm node3 (idle, Wed Mar 22 17:14:16 2017)
lrm node4 (active, Wed Mar 22 17:14:07 2017)
lrm node1 (idle, Wed Mar 22 17:14:13 2017)
service vm:100 (node2, error)

-= How I got here =-
I attempted a migration via the web GUI and it threw an error about something in /var being full. Attempting to migrating to other nodes also failed immediately after that. This is a test cluster, so I unfortunately didn't bother getting the exact error messages. /var is now no longer full. Migration from the GUI appears to start normally but never completes. Migration via the command line results in the above. Trying to migrate to nodes 3 or 4 also fails, but without logging 'ignore crm command - service already on target node...'. In all cases qm/ha-manager exit with a 0 exit status, but migration simply never occurs.

I suspect killing and restarting the VM would recover things, but I'm evaluating proxmox, so I'd like to know that I can recover from such states without rebotting the VM before I use it for production stuff.

Thanks

 

[wolfgang]

Hi,

remove the resource(VM) from HA.
Wait 1 sec and then add it in again.

 

[Adam Carheden]

Worked like a champ. Thanks

 

[Adam Carheden]

For the uninitiated (like me) who arrive here via Google, the commands were:
# ha-manager remove vm:100
# ha-manager add vm:100 -group cluster1group

Remove 'vm:100' and 'cluster1group1' with values for your environment, of course. That can also be done via the WebUI.