Memory utilisation when VM is TrueNAS

[Scotty]

I'm just trying to get my head around the memory utilisation on my machine as I appear to be running out.

My machine has 128GB installed and I have 2 VM's running.

• TrueNAS:
• 
  • HDD's passed through using qm set, then ZFS volume built by TrueNAS
• Linux-Mint:

Over a period of a few days, I'll get to this point:



You can see the rate of memory consumption, with the dip being a reboot:





'top'appears to show the momery is being used but doesn't show the process using it:



How do I go about finding what is consuming all the memory to the point that swapping is occurring?

Thanks.

 

[Scotty]

Just to add...

After shutting down both of the VM's, it's still consuming 40GB:

 

[Johannes S]

This is expected since the Linux kernel ( and ProxmoxVE is basically a Debian Linux with an Ubuntu Kernel) uses the system RAM as cache to speed up things. Additionally ZFS ( if you use it on the Proxmix VE host) uses RAM for it's own cache.

This is a good thing, I would be more worried if the RAM wouldn't be used thus wasted. See https://linuxatemyram.com for more information.
To monitor the ram usage inside the vms you need a dedicated monitoring Tool like Zabbix, Icinga2, prometheus, checkmk or whatever you prefer.

Btw: You don't use a dedicated disc controller as PCI passthrough with TrueNAS don't you? Please reconsider this approach, you might loose your data:
https://www.truenas.com/community/r...guide-to-not-completely-losing-your-data.212/

 

[Scotty]

I think the bit I was missing was my logic of 'why would an OS increase its cache to the point that it needs to swap'. That seemed counter-intuitive. However, the link you provided mentions ‘Linux can push application memory into swap if that memory is accessed less often than the filesystem cache’. That makes sense.

Regarding passing through the HBA card…I’m aware. The issue is the HP ML350 I have only has the one HBA, which also has the boot disks, so couldn’t be handed over entirely. I’m actually in the process of backing everything up to a PBS (first time trying PBS). Once I also have everything over onto LTO, I’ll add another HBA and rebuild it with two distinct cards (Proxmox and TrueNAS). Seems a shame, I love Proxmox for its VM management but it lacks the finesse when it comes to fileserving, where TrueNAS is awful (personal taste) regarding VM’s but good on the fileserver capabilities.

 

[gfngfn256]

On top cycling through with the "m" key will actually show you buff/cache in use. (I believe that is also the initial screen top displays).

 

[Scotty]

On top cycling through with the "m" key will actually show you buff/cache in use. (I believe that is also the initial screen top displays).

It is indeed.

Sorry, I should have qualified at the start that this is a home server, so no 'production' issue here.

I rebooted the machine last night and I've had it performing a backup all day to get it stretching its leggs. The Linux-Mint VM is only there to mount the SMB shares from the TrueNas VM and run the PBS client to a seperate PBS machine. I know it's an odd way to do it but it keeps the TrueNas install nice and clean. The logic being it's clean and the traffic is at the hypervisor level until it goes out on the 10g to the PBS machine.

After a day of stretching it's leggs...



Or in a more readable format:



So yes, there is almost 1GB of cache/buffer, and 25GB free.

Although I'm guessing that by the end of the week...without any further activity other than the backups from clients (1x Macbook and 1x small Qnap-rsync) to the TrueNas SMB shares...I start to run out again and it starts swapping. It will be interesting (to me at least) what top/free is showing this time next week.