Clear HA migration queue?

[konaya]

Let's say you have a cluster with HA enabled. If you have an “oh, crap” moment and accidentally trigger a mass migrate of a large number of virtual machines, placing them all in HA state migrate and queued for migration, is there a way to clear this queue so they don't get migrated?

If there's no official way to do it, is there a way that might work in theory that I could experiment with?

 

[fiona]

Hi,
no this is not possible. Once the state of a service is set to migrate it can only be (and will be) changed after the migration finished or failed. So you could stop/kill each migration task, but I really wouldn't recommend that. Why not simply migrate the services back afterwards? With HA managed VMs you always need shared storage anyways (otherwise it's not officialy supported by PVE), so migration shouldn't even take that long.

 

[konaya]

Is there a way to view this migration queue, then? Besides fetching a list of all the services with state migrate, I mean, which doesn't show any additional information such as destination. Where is the queue stored internally?

Why not simply migrate the services back afterwards?

Well … the last time it happened, we decided to do just that, and HA kept migrating and migrating until a node OOMed and crashed. HA doesn't appear to have very many sanity checks in place, which is why I would feel more comfortable being able to clear the queue in case something like that would happen. In any case, isn't it a reasonable feature to be able to cancel a queued migration?

 

[fiona]

Is there a way to view this migration queue, then? Besides fetching a list of all the services with state migrate, I mean, which doesn't show any additional information such as destination. Where is the queue stored internally?

Well … the last time it happened, we decided to do just that, and HA kept migrating and migrating until a node OOMed and crashed. HA doesn't appear to have very many sanity checks in place, which is why I would feel more comfortable being able to clear the queue in case something like that would happen. In any case, isn't it a reasonable feature to be able to cancel a queued migration?

To get a feeling, how many VMs are we talking about here?

There is no internal queue in the HA manager for migrations. The HA manager uses a state machine for each service and once the state is set to migrate, it can only change after the migration either finished or failed, because the migration task is already running. I agree that showing the destination is an improvement.

But there are some good news. When using Bulk migrate, it takes a bit of time for all the service states to become migrate, so when you abort the Bulk migrate task, only those machines where the migration has already begun will be migrated (and if you really want to, you can abort the migration tasks for the individual machines). So there is a way to abort bulk migration, although not completely, but I feel like bulk actions should be utilized carefully in any case.

But the number of migrations wasn't the real problem here, was it? It was that the other node wasn't prepared to have so many running services. To play the devil's advocate for a bit, what if the services wouldn't have been relocated by bulk migration, but because of a hardware failure? Then an other node would've potentially also ended up with more services than it can handle, no? If you have multiple remainng nodes, the services will be more evenly distributed of course, so maybe it would've been fine.

 

[konaya]

To get a feeling, how many VMs are we talking about here?

About 500 or so. The cluster contains seven nodes, but most of the VMs are kept on five. We can go down to four in an emergency without disruption of service, if things are kept balanced. Something I've spent quite a few code-hours trying to automate, and it mostly works fine, but there are a few things which could allow me to make my code even better, such as being able to unset the migration state. Hooks for pre- and post-migration would be nice, too.

There is no internal queue in the HA manager for migrations. The HA manager uses a state machine for each service and once the state is set to migrate, it can only change after the migration either finished or failed, because the migration task is already running.

And there is no way to make it fail before the migration starts? That's a shame. Is there a way to make migrations auto-fail, then? Some hooks would have been a welcome feature here – adding /bin/false to a pre-migration hook would have sidestepped the problem.

To play the devil's advocate for a bit, what if the services wouldn't have been relocated by bulk migration, but because of a hardware failure?

That's a very different scenario. I'm assuming HA is clever enough not to send all the VMs to one and the same node in the event of a hardware failure, but to use the priority numbers in the HA groups and spread evenly across nodes with the same priority number. With a bulk migration, however, someone has explicitly told HA to migrate to one specific node. This obviously shouldn't happen, but when we're training new crew things happen. It would be nice to be able to add an arrow to that state machine of yours.

The potential of a node crashing is just one of the issues behind a bulk migration, however. We have a few Cisco ASAv VMs which network interfaces straight up and die when they are live migrated, so those we can't move at all – but if some newbie has managed to initiate a bulk migration, we'd have to babysit Proxmox and press cancel when a migration of such a VM comes up. Again, some hooks would have come in handy here.

(One would think that HA groups would come in handy here, but HA apparently has no objections at all migrating a VM outside its HA group, although it will migrate it back a few moments later.)

 

[fiona]

About 500 or so. The cluster contains seven nodes, but most of the VMs are kept on five. We can go down to four in an emergency without disruption of service, if things are kept balanced. Something I've spent quite a few code-hours trying to automate, and it mostly works fine, but there are a few things which could allow me to make my code even better, such as being able to unset the migration state. Hooks for pre- and post-migration would be nice, too.

And there is no way to make it fail before the migration starts? That's a shame. Is there a way to make migrations auto-fail, then? Some hooks would have been a welcome feature here – adding /bin/false to a pre-migration hook would have sidestepped the problem.

I reopened the enhancement request for this.

That's a very different scenario. I'm assuming HA is clever enough not to send all the VMs to one and the same node in the event of a hardware failure, but to use the priority numbers in the HA groups and spread evenly across nodes with the same priority number. With a bulk migration, however, someone has explicitly told HA to migrate to one specific node. This obviously shouldn't happen, but when we're training new crew things happen. It would be nice to be able to add an arrow to that state machine of yours.

Yes, HA is clever enough, see the very last sentence I wrote in my previous reply.

It's not recommended to cancel a running migration, but that is the arrow you mentioned and you can trigger it manually if you really want to.

But I think I found the real problem now. There is a maxworkers setting to limit the number of simultaneous migration workers and this is 1 when starting a bulk migration from the GUI. So why is it spawning multiple workers? Because in the HA managed case, the workers only request migration from HA and therefore finish much faster than the migration itself. I opened a bug report for this. The bulk migration uses a queue, but with HA managed VMs it is wrongly side-stepped.

The potential of a node crashing is just one of the issues behind a bulk migration, however. We have a few Cisco ASAv VMs which network interfaces straight up and die when they are live migrated, so those we can't move at all – but if some newbie has managed to initiate a bulk migration, we'd have to babysit Proxmox and press cancel when a migration of such a VM comes up. Again, some hooks would have come in handy here.

(One would think that HA groups would come in handy here, but HA apparently has no objections at all migrating a VM outside its HA group, although it will migrate it back a few moments later.)

This is a known issue.

 

[konaya]

I reopened the enhancement request for this.

Thank you.

But I think I found the real problem now. There is a maxworkers setting to limit the number of simultaneous migration workers and this is 1 when starting a bulk migration from the GUI. So why is it spawning multiple workers? Because in the HA managed case, the workers only request migration from HA and therefore finish much faster than the migration itself. I opened a bug report for this. The bulk migration uses a queue, but with HA managed VMs it is wrongly side-stepped.

To be honest, I sort of assumed that this was known and intentional for some reason. Yes, if this behaviour were changed it would change a lot.

Again, thank you.