Bulk migration leads to oom killer killing VMs

[123pve!]

Hi there

I am running a 3 node Proxmox Cluster with CEPH and about 60 VMs with HA on it.

All three nodes are equaly sized with one Epyc CPU and 512GB of RAM and some OSDs each.

All VMs have fixed RAM size.

My Problem ist with bulk migration of VMs / evacuation of nodes.

I like to evacuate a node to apply patches and reboot for newer kernel.

This is how it looks before:
node 1 - 30 VMs
node 2 - 30 VMs
node 3 - 0 VMs

If I do a bulk migration of all VMs from node1 to node3 everything works as expected.

But as the preselectet node in the migration dialogue ist the one with the lowest number not including itself ist ist node2 in this case there is a risk to migrate to the already populated node.

I made sure to always double check, but totay was one of those mondays where I simply forgot.
Vms were migrated until about maybe 10Gigs of free RAM on node2 but that was not enough free space so the oom killer killed two ram intensive VMs

I do not want this to happen again any time soon and I do not think it is enough to be cautious while doing admin stuff. What I would like is a technical solution to my problem and there may be one that I am not aware of.

Things I could think of:
1. Proxmox preselects the node with the lowest memory load

2. Proxmox Checks that the max Ram all VMs to be migrated will less than the free RAM on the destination host + a security margin for CEPH and waht else may need RAM

3. Something like VMwares DRS would have prevented this by balancing the nodes in time. If I am not mistaken there is work going on to implement somethin similar.

Im looking forward to hear your ideas and how you deal with this.

All the best
123pve!

 

[bbgeek17]

Proxmox does not have DRS type functionality today. It may be on roadmap, check the bugzilla for an enhancement request and add your vote, or if its not there - log it.
Other than that, check the github there are a few projects there that build on top of PVE to add this functionality:
https://github.com/cvk98/Proxmox-load-balancer

I am not endorsing this one ^, just the first one in the list.

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[123pve!]

Thank you for that link!

I will look into that, but it is not that I need something fancy like DRS, just a way to prevent me oom killing VMs with a thoughtless bulk migration. Nevertheless this may help.

All the best
123pve!

 

[123pve!]

Hi its me again

I looked into the different load balancer solutions and I do not think it will be a good solution to my issue.

If i accidentally migrate more VMs to a already saturated node the balancer script needs to react very fast and keep up at least the same migration performance as the accidental migrate so it will not run into memory issues. Even if that works there will be a lot of uneccessary migration stuff going on.

What would be much more helpful would be a defined amount of ram to keep free so that the migration process stops the migration of more vms.
Is there any way to achieve such a behaviour?

I tried to find solutions for this online but it seems that I am the only person with this problem? But in my opinion ever cluster with more than 50% memory load per node is affected by this?

All the best
123pve!