Thanks for your reply! I see, really didn't paid much attention to buff/cache on the guest, and was focused on the graph displayed in Proxmox, which gets red and we all know red is bad :P
So, based on your suggestion to another user (above), I shouldn't give the guest more memory than it...