ProxMox uses 50% ram without ZFS and won't...

While it might be the hugepages change which fixed things, could it have also been just the reboot cleared out those VMs?

If you're open to doing a bit of experimentation, maybe try changing it back and seeing if the problem returns after a reboot? (if it does, then change hugepages again + reboot to fix it)
 
well I rebooted several times with the hugepages=32 set, and it did not work. So I guess the hugepages were reserved by the kernel but kvm was not able to use them ?