thanks!!4 GB for PVE should be enough in general [1].
Are you using ZFS by any chance? The ARC (Adaptive Replacement Cache) used by ZFS will use up to half of your system's RAM by default. In theory it should shrink under high memory pressure, but it can be too slow in some situtations. You can limit the maximum size of the ARC if you want [2].
You can check the current size of the ARC by using the arcstat command.
Hope this helps.
https://pve.proxmox.com/pve-docs/pve-admin-guide.html#_system_requirements
https://pve.proxmox.com/wiki/ZFS_on_Linux#sysadmin_zfs_limit_memory_usage