You have allocated 32 GB to the VM, it will use that amount...it allocates caches, etc., and does not know how to release it to the host. If you need it to use less you can allocate less to the VM.
If you enable ballooning (and have the driver installed) Proxmox will remove memory from the VM...