Okay, I heard that somewhere, that Linux kernel tends to preoccupy almost all available memory for performance reasons.
I don't think I have any sleeping processes because the server is literally empty. Nothing running on it, definitely not 4GBs. But I can imagine that the memory is prepared...