I know this is an old thread, but I've been dealing with this issue too for a few years.
In my case, I have about a dozen VM's actively running in production, but the elephant in the room amoung them is a Windows VM in which its allocated 64GB RAM (PVE has 512GB), (most other guests are Debian...