I have two Proxmox hosts, 196 Gb and 256 Gb RAM, dual Xeon, Supermicro, very similar to your configuration. Very standard servers. VMs with and without ballooning, Windows and Linux, on both nodes and I have never face with your problem.
What is the difference?..
Is your RAM fast enough?