[PVE 8.3 | Kernel 6.8.12-4] KVM VM memory usage exceeds assigned limit, causes host OOM and process kill

E

Evanshenf

New Member
Mar 17, 2025
4
0
1

Post Content:​

Host Environment:

  • Proxmox VE 8.3, kernel 6.8.12-4-pve
  • CPU: Intel Core i7-11700 (8C/16T)
  • RAM: 16GB, no swap configured
  • Storage: Full LVM-Thin (no ZFS)
  • 2 running VMs, each assigned 7GB of memory
  • Ballooning is disabled
During a build task inside a VM, the host system ran out of memory and invoked the OOM killer. One of the kvm processes was killed, causing the VM to shut down.

Key log entries:

Out of memory: Killed process 2856832 (kvm) total-vm:12770796kB, anon-rss:7364480kB
[2856832] ... rss: 1841536 pages ≈ 7.02 GB
[2856739] ... rss: 1731340 pages ≈ 6.60 GB
Although each VM was configured with 7GB RAM, the corresponding KVM processes were using 7.02GB and 6.60GB of actual physical memory, totaling ~13.6GB, which is unexpected.

At the time of the OOM event, the host had only ~130MB free memory left:

Apr 02 01:10:57 pve kernel: free:33495 # ≈ 130MB
This led the system to kill the kvm process of VM 100.

❓Questions:​

  1. Is this a known issue where KVM memory limits are not properly enforced or memory is leaked?
  2. Could this be a regression in the 6.8.12-4 kernel?
  3. Are there any recommended workarounds? Should I consider reverting to kernel 6.5?
I can provide full
 
Last edited:
pr 02 00:57:59 pve pveproxy[3198419]: parse error in '/etc/pve/datacenter.cfg' - 'sdn': property is not defined in schema and the schema does not allow additional properties
Apr 02 01:10:57 pve kernel: proxmox-firewal invoked oom-killer: gfp_mask=0x140cca(GFP_HIGHUSER_MOVABLE|__GFP_COMP), order=0, oom_score_adj=0
Apr 02 01:10:57 pve kernel: CPU: 6 PID: 1198 Comm: proxmox-firewal Tainted: P O 6.8.12-4-pve #1
Apr 02 01:10:57 pve kernel: Hardware name: Dell Inc. OptiPlex 7090/0GNVHC, BIOS 1.12.0 09/12/2022
pr 02 01:10:57 pve kernel: active_anon:3641081 inactive_anon:193252 isolated_anon:0 active_file:37 inactive_file:156 isolated_file:0 unevictable:8736 dirty:14 writeback:0 slab_reclaimable:6116 slab_unreclaimable:73775 mapped:11138 shmem:12539 pagetables:9579 sec_pagetables:1043 bounce:0 kernel_misc_reclaimable:0 free:33495 free_pcp:6147 free_cma:0
Apr 02 01:10:57 pve kernel: Node 0 active_anon:14564324kB inactive_anon:773008kB active_file:148kB inactive_file:624kB unevictable:34944kB isolated(anon):0kB isolated(file):0kB mapped:44552kB dirty:56kB writeback:0kB shmem:50156kB shmem_thp:0kB shmem_pmdmapped:0kB anon_thp:12175360kB writeback_tmp:0kB kernel_stack:5888kB pagetables:38316kB sec_pagetables:4172kB all_unreclaimable? no
Apr 02 01:10:57 pve kernel: Node 0 DMA free:13312kB boost:0kB min:64kB low:80kB high:96kB reserved_highatomic:0KB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:15992kB managed:15360kB mlocked:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
Apr 02 01:10:57 pve kernel: lowmem_reserve[]: 0 1382 15633 15633 15633
Apr 02 01:10:57 pve kernel: Node 0 DMA32 free:62616kB boost:0kB min:5968kB low:7460kB high:8952kB reserved_highatomic:0KB active_anon:1405892kB inactive_anon:400kB active_file:44kB inactive_file:188kB unevictable:0kB writepending:4kB present:1541580kB managed:1475316kB mlocked:0kB bounce:0kB free_pcp:1552kB local_pcp:500kB free_cma:0kB
Apr 02 01:10:57 pve kernel: lowmem_reserve[]: 0 0 14251 14251 14251
Apr 02 01:10:57 pve kernel: Node 0 Normal free:58052kB boost:0kB min:61548kB low:76932kB high:92316kB reserved_highatomic:0KB active_anon:13120208kB inactive_anon:810832kB active_file:52kB inactive_file:184kB unevictable:34944kB writepending:52kB present:14934016kB managed:14601268kB mlocked:34800kB bounce:0kB free_pcp:23036kB local_pcp:4444kB free_cma:0kB
Apr 02 01:10:57 pve kernel: lowmem_reserve[]: 0 0 0 0 0
Apr 02 01:10:57 pve kernel: Node 0 DMA: 0*4kB 0*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 1*1024kB (U) 2*2048kB (UM) 2*4096kB (M) = 13312kB
Apr 02 01:10:57 pve kernel: Node 0 DMA32: 4*4kB (UM) 3*8kB (U) 5*16kB (UM) 3*32kB (U) 3*64kB (U) 2*128kB (U) 32*256kB (UM) 39*512kB (UM) 33*1024kB (UM) 0*2048kB 0*4096kB = 62616kB
Apr 02 01:10:57 pve kernel: Node 0 Normal: 340*4kB (UME) 788*8kB (UME) 1125*16kB (UME) 542*32kB (UME) 185*64kB (UME) 27*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 58304kB
Apr 02 01:10:57 pve kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
Apr 02 01:10:57 pve kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Apr 02 01:10:57 pve kernel: 15505 total pagecache pages
Apr 02 01:10:57 pve kernel: 0 pages in swap cache
Apr 02 01:10:57 pve kernel: Free swap = 0kB
Apr 02 01:10:57 pve kernel: Total swap = 0kB
Apr 02 01:10:57 pve kernel: 4122897 pages RAM
Apr 02 01:10:57 pve kernel: 0 pages HighMem/MovableOnly
Apr 02 01:10:57 pve kernel: 99911 pages reserved
Apr 02 01:10:57 pve kernel: 0 pages hwpoisoned
Apr 02 01:10:57 pve kernel: Tasks state (memory values in pages):



Apr 02 01:10:57 pve kernel: oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=proxmox-firewall.service,mems_allowed=0,global_oom,task_memcg=/qemu.slice/100.scope,task=kvm,pid=2856832,uid=0
Apr 02 01:10:57 pve kernel: Out of memory: Killed process 2856832 (kvm) total-vm:12770796kB, anon-rss:7364480kB, file-rss:1664kB, shmem-rss:0kB, UID:0 pgtables:15496kB oom_score_adj:0
Apr 02 01:10:57 pve kernel: fwbr100i0: port 2(tap100i0) entered disabled state
Apr 02 01:10:57 pve kernel: tap100i0 (unregistering): left allmulticast mode
Apr 02 01:10:57 pve kernel: fwbr100i0: port 2(tap100i0) entered disabled state
Apr 02 01:10:57 pve systemd[1]: 100.scope: A process of this unit has been killed by the OOM killer.
Apr 02 01:10:57 pve systemd[1]: 100.scope: Failed with result 'oom-kill'.
Apr 02 01:10:57 pve systemd[1]: 100.scope: Consumed 17min 8.357s CPU time.
Apr 02 01:10:58 pve qmeventd[3219876]: Starting cleanup for 100
Apr 02 01:10:58 pve kernel: fwbr100i0: port 1(fwln100i0) entered disabled state
Apr 02 01:10:58 pve kernel: vmbr0: port 3(fwpr100p0) entered disabled state
Apr 02 01:10:58 pve kernel: fwln100i0 (unregistering): left allmulticast mode
Apr 02 01:10:58 pve kernel: fwln100i0 (unregistering): left promiscuous mode
Apr 02 01:10:58 pve kernel: fwbr100i0: port 1(fwln100i0) entered disabled state
Apr 02 01:10:58 pve kernel: fwpr100p0 (unregistering): left allmulticast mode
Apr 02 01:10:58 pve kernel: fwpr100p0 (unregistering): left promiscuous mode
Apr 02 01:10:58 pve kernel: vmbr0: port 3(fwpr100p0) entered disabled state
Apr 02 01:10:58 pve qmeventd[3219876]: Finished cleanup for 100
 
You never have enough RAM. And giving only 2 GiB to the main OS is... brave. (I've upgraded some nodes in my Homelab(!) to 128 GiB...)

Try to utilize "zram" - it may give you a little bit more free headroom.
Code: 
~# apt show zram-tools
Description: utilities for working with zram
 zram is a Linux kernel module that allows you to set up compressed
 filesystems in RAM.
 .
 zram-tools uses this module to set up compressed swap space.
 This is useful on systems with low memory or servers
 running a large amount of services with data that's easily swappable
 but that you may wish to swap back fast without sacrificing disk
 bandwidth.

In any case the correct solution is to upgrade for more physical RAM...
 
You must log in or register to reply here.
Top Bottom