I came across this thread: https://forum.proxmox.com/threads/memory-usage-not-same-as-in-vm.82640/
which is the exact same issue I have, however the guy didn't end up replying back.
I'm running Ubuntu 20.04.3 with PVE 6.4-4, Ballooning device is enabled in the Memory Settings:
Doing info...