Out of memory

[Matteo Calorio]

Hi,

we experience problems about RAM, expecially on a node that appears to not have very high usage:



Despite this we note that some machines go "out of memory":



Any ideas?

Regards,
Matteo

 

[B.Otto]

Hello,

are you using ZFS? By default ZFS uses up to 50% of your RAM for the ARC cache, so it is recommended to limit it appropriately: https://pve.proxmox.com/wiki/ZFS_on_Linux#sysadmin_zfs_limit_memory_usage

Kind regards,
Benedikt

 

[Matteo Calorio]

Thanks, I'm not a ZFS expert, but as far as I can see ARC usage is low:

# arc_summary -g

ARC: 4.3 GiB (0.9 %)  MFU: 3.9 GiB  MRU: 196.7 MiB  META: 440.8 MiB (377.9 GiB) DNODE 107.5 MiB (37.8 GiB)
+----------------------------------------------------------+
|                                                          |
+----------------------------------------------------------+

Also "smem" tells we have a lot of free memory:

# smem -tw
Area                           Used      Cache   Noncache
firmware/hardware                 0          0          0
kernel image                      0          0          0
kernel dynamic memory     207801072  195526700   12274372
userspace memory          576656820     183780  576473040
free memory               272279204  272279204          0
----------------------------------------------------------
                         1056737096  467989684  588747412

Did I miss something?

Matteo

 

[Matteo Calorio]

Still having the problem: why ballooning occurs so early and so heavily?

 

[Dunuin]

Ballooning starts at 80% RAM usage of the host and will only stop after hosts RAM drops below 80% again or all guests reached their minimum RAM limit.

 

[Matteo Calorio]

Ballooning starts at 80% RAM usage of the hostz and will only stop after hosts RAM drops below 80% again or all guests reached their minimum RAM limit.

In fact, we opened the ticket because it seems to us that it already occurs at 50% use and not at 80%.

 

[dcsapak]

Despite this we note that some machines go "out of memory":

do you mean this is inside the vm (at least it looks like it, since those processes don't exist on a vanilla proxmox installation)?

can you post the config /summary from such a vm

 

[Matteo Calorio]

Yes, "out of memory" inside the machine.

Also, with "info balloon" in monitor we see that "actual" memory is low and processes on guests do not start or get killed with OoM error. Also, on windows machines we have the same situation in which we do not have all configured RAM available.

This is from the original configuration of the machine from which I took the screenshot above:

agent: 1
balloon: 512
memory: 45056
cores: 11
sockets: 1

Rising the minimum RAM solves the problem on that machine:

balloon: 16384

But we don't understand why ballooing occours on a node with RAM at 50-60%:

 

[dcsapak]

my guess is that the processes in the vm request memory much faster than it is allocated to the vm and thus the vm runs out of memory

 

[Matteo Calorio]

Yes, I thought it, too, but the problem is that it happens with almost all the VMs turned on from a certain point on (so with those turned on before the problem occurrd, at least it seems).

However, it also happens when the minimum RAM is raised to 8 or 16 GB out of 32: when processes that require RAM beyond that threshold start, they go in OoM.

Is there any debug command to see if the ballooning is "active" before 80% of host RAM usage, i.e. if it is intervening and with which parameters or thresholds?

 

[dcsapak]

Is there any debug command to see if the ballooning is "active" before 80% of host RAM usage, i.e. if it is intervening and with which parameters or thresholds?

no not really, but you could run 'pvestatd' in debug mode by stopping it first and then starting it with

pvestatd start -debug

then it will print debug messages for ballooning..
(note that pvestatd also collects rrd data and sends metrics data)