Migrate doesn't check target host memory

[avn]

When target host doesn't have enough free memory, migration process doesn't stop. It tries to allocate memory on the target host, which result in OOM killing the largest VM, and another, and another. In earlier versions of Proxmox VE it wasn't like that. Migration was giving an error "cannot allocate memory" or something like that. Now I need to sit and monitor when the target host reach ~90% of memory and stop the migration manually. Don't know when it changed, maybe in 5th version, not sure. Just wonder, why, because it certainly not supposed to work like that.

 

[itNGO]

Same here.... we have a 3 node Cluster.... All nodes using about 55 to 60% Memory.
Sometimes when we start Mass-Migrate it works, cause other process and swap compensate missing memory, sometimes it just kills random VMs on target node. Coming from HyperV I would expect some sort of check, if migration is possible before just filling up target hosts RAM.

 

[ness1602]

This is okay,since migration can last hours,and you can start-stop machines during it.

 

[itNGO]

This is okay,since migration can last hours,and you can start-stop machines during it.

It is OK that VMs get killed on target host when other VMs are migrated towards it? NO! It is not.... Consider your important Database-Server gets turned off during production-hours....

 

[ness1602]

Then don't do migrations in production-hours, proxmox leaves this up to you. Don't undeprovision your servers,etc etc.

 

[itNGO]

Then don't do migrations in production-hours, proxmox leaves this up to you. Don't undeprovision your servers,etc etc.

Sorry, but it does not work like that in real live.... ever considered a node-crash where 20VMs get HA-Moved to other nodes and startet there?
Consider a 3 Node Cluster used with about 66%-Memory... loosing a node would in "theory" mean, all VMs can start on other nodes... but some won't cause of OOM..... nope the behavior is absolutely not optimal in PVE..... And when this happens, you not only loose some VMs from the crashed node, worst case VMs which are running on the other Nodes might also crash and things might get even worse then.....

Or do you want to say having a 3 Node-Cluster means you have to have spare-resources on every node in capacity of "EVERY" other node... what's the usage of HA-Cluster then? Get it?

 

[spirit]

Maybe something had change on qemu side.

I'm 100% sure that qemu couldn't start before if no enough memory was available to handle the vm max memory size ("cannot allocate memory" ), before the transfert of the memory by the live migration job.


About HA, I'm currently working to add ressource aware HA manager. (looking at available cpu/ram/load/storage/...) before restart vms.
(and the idea is to add loadbalacing too, migrating vm from too much loaded hosts, add support for "migrate all",....).
I have sent beta patch on pve-deve mailinglist.

Currently the HA only try to loadbalance by number of vm by host.

 

[itNGO]

Maybe something had change on qemu side.

I'm 100% sure that qemu couldn't start before if no enough memory was available to handle the vm max memory size ("cannot allocate memory" ), before the transfert of the memory by the live migration job.


About HA, I'm currently working to add ressource aware HA manager. (looking at available cpu/ram/load/storage/...) before restart vms.
(and the idea is to add loadbalacing too, migrating vm from too much loaded hosts, add support for "migrate all",....).
I have sent beta patch on pve-deve mailinglist.

Currently the HA only try to loadbalance by number of vm by host.

I read the messages in the Mailing-List about this, and for me it sounds like an amazing approach for the described problems.
Comes more close to how Failove-Cluster-Manager in HyperV does it. Not that they do it perfect, but the idea to keep the cluster stable even on a node failure is good. That's what we are looking forward to... maybe 7.2? ;-)