Hello,
I am currently trying to solve a problem with OOM/memory exhaustion on 2 of our proxmox hosts. The problem is extensive shared and buff/cache memory usage, which none of our other hosts have. The 2 problematic nodes are in 3-node cluster running
The problem here is obvious:
but the total sum of all memory assigned to VMs is only around 200GiB. All our other nodes in different clusters have memory consumption roughly equal to the sum of memory assigned to VMs.
For reference, here is info from a machine with the same HW configuration and VM memory sum of 238GiB.
All our nodes are installed via ansible with same configuration and are running the mentioned PVE version and kernel. This is not a ZFS problem since all have
Here is few selected metrics from /proc/meminfo
So on the bad host Shmem is ~191 GiB, AnonPages is also ~191 GiB. On the good host AnonPages is even higher, but Shmem stays small.
Did some more investigative work and came up with this:
We tried to turn off balooning on the biggest VMs on the problematic hosts. There was around 80GiB of free mem after we restarted the VMs but it came back to almost nothing again. The workload is very mixed: haproxy, dns, traefik, k8s (biggest VMs memorywise), appcloud, keycloack, ldap, pfsense. I suspected the workload to be the problem but the restart of the biggest VMs freed only 80GiB not the 170GiB there should be. I assume all VMs on this node add to the Shmem their own part.
The only differnece from the other clusters is that the bad node VMs have disks on local storage SSDs with VirtIO SCSI controller and not network storage.
I am trying to solve this for a few weeks now and did not find this particular problem anywhere. The only lead is that the Shared memory and the Anon pages are almost same size and I suspect the VMs memory is somehow cached/duplicated there, but I donť know why or how.
I would very much appreciate any insight or help with this problem. I can provide more outputs if needed.
I am currently trying to solve a problem with OOM/memory exhaustion on 2 of our proxmox hosts. The problem is extensive shared and buff/cache memory usage, which none of our other hosts have. The 2 problematic nodes are in 3-node cluster running
pve-manager/9.0.3/025864202ebb6109 (running kernel: 6.14.8-2-pve) on 2 sockets with Intel Xeon Gold 6330N.The problem here is obvious:
Bash:
free -h
total used free shared buff/cache available
Mem: 377Gi 371Gi 5,1Gi 182Gi 185Gi 5,4Gi
Swap: 6,1Gi 6,1Gi 36KiFor reference, here is info from a machine with the same HW configuration and VM memory sum of 238GiB.
Bash:
free -h
total used free shared buff/cache available
Mem: 377Gi 247Gi 128Gi 5,5Gi 8,8Gi 129Gi
Swap: 6,1Gi 0B 6,1Gizfs_arc_max 17179869184 and arcstats is virtually zero. I will provide data for only one of the bad nodes since the other is virtually in the same situation.Here is few selected metrics from /proc/meminfo
| Metric | Bad node | Good node | Unit |
|---|---|---|---|
| MemTotal | 395389696 | 395390644 | kB |
| MemFree | 5426088 | 134788944 | kB |
| MemAvailable | 5761788 | 136034968 | kB |
| Cached | 193723388 | 8878728 | kB |
| SwapCached | 79876 | 0 | kB |
| Unevictable | 206348 | 288816 | kB |
| SwapTotal | 6422524 | 6422524 | kB |
| SwapFree | 36 | 6422524 | kB |
| AnonPages | 191609476 | 247222708 | kB |
| Mapped | 217564 | 250364 | kB |
| Shmem | 191610732 | 5751292 | kB |
| Slab | 1979440 | 1422216 | kB |
| SReclaimable | 598088 | 222448 | kB |
| KernelStack | 29552 | 23424 | kB |
| PageTables | 406380 | 507012 | kB |
| CommitLimit | 204117372 | 204117844 | kB |
| Committed_AS | 423805608 | 261909520 | kB |
| Percpu | 390656 | 427840 | kB |
| AnonHugePages | 171546624 | 245547008 | kB |
| ShmemHugePages | 0 | 0 | kB |
| ShmemPmdMapped | 0 | 0 | kB |
So on the bad host Shmem is ~191 GiB, AnonPages is also ~191 GiB. On the good host AnonPages is even higher, but Shmem stays small.
Did some more investigative work and came up with this:
- /dev/shm is almost empty
- ipcs -m shows nothing useful
- RssShmem / Pss_Shmem for processes is small
- kvm processes mostly show:
- high Pss_Anon
- high Anonymous
- high AnonHugePages
- almost zero Pss_Shmem
- ruled out CPU problems since the machines with the same CPU are running good
Bash:
PID 3075114:
Rss: 50242852 kB
Pss: 50231151 kB
Pss_Anon: 50228792 kB
Pss_File: 2315 kB
Pss_Shmem: 44 kB
Anonymous: 50228792 kB
AnonHugePages: 35942400 kB
Shared_Hugetlb: 0 kB
Private_Hugetlb: 0 kB
Swap: 0 kBWe tried to turn off balooning on the biggest VMs on the problematic hosts. There was around 80GiB of free mem after we restarted the VMs but it came back to almost nothing again. The workload is very mixed: haproxy, dns, traefik, k8s (biggest VMs memorywise), appcloud, keycloack, ldap, pfsense. I suspected the workload to be the problem but the restart of the biggest VMs freed only 80GiB not the 170GiB there should be. I assume all VMs on this node add to the Shmem their own part.
The only differnece from the other clusters is that the bad node VMs have disks on local storage SSDs with VirtIO SCSI controller and not network storage.
I am trying to solve this for a few weeks now and did not find this particular problem anywhere. The only lead is that the Shared memory and the Anon pages are almost same size and I suspect the VMs memory is somehow cached/duplicated there, but I donť know why or how.
I would very much appreciate any insight or help with this problem. I can provide more outputs if needed.