Proxmox 6.1 shutdown VMs (Out of memory: Killed process)

I

IT Admin

Member
Jul 25, 2019
4
0
21
51
Hi,

In the past few weeks we have noticed a lot of VMs shutdown randomly on different hosts, we have 7 hosts in cluster.
after reading the logs of one of the hosts, we found:

[615743.603922] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0-1,global_oom,task_memcg=/qemu.slice/193.scope,task=kvm,pid=7078,uid=0
[615743.603961] Out of memory: Killed process 7078 (kvm) total-vm:34461560kB, anon-rss:33597640kB, file-rss:728kB, shmem-rss:4kB, UID:0 pgtables:66992kB oom_score_adj:0

Host's memory:

free -g
total used free shared buff/cache available
Mem: 503 480 21 0 0 19
Swap: 3 3 0

it doesn't look like the VM hadn't enough memory.
is there a way to overcome this issue?


thanks!
 
Hallo,
ich habe ein ähnliches Problem. Der Server hat 256 GByte Ram. 200 Gbyte ram sind den Maschinen zugewiesen. Doch der Server hat plötzlich 100% Speicherauslastung. Warum? Gibt es ein Memory Leak Problem mit ZFS?



Feb 28 15:47:40 px18 kernel: [793409.702407] zfs invoked oom-killer: gfp_mask=0x42dc0(GFP_KERNEL|__GFP_NOWARN|__GFP_COMP|__GFP_ZERO), order=2, oom_score_adj=0
Feb 28 15:47:40 px18 kernel: [793409.702564] CPU: 59 PID: 3369244 Comm: zfs Tainted: P O 5.11.22-3-pve #2
Feb 28 15:47:40 px18 kernel: [793409.702688] Hardware name: Micro-Star International Co., Ltd. MS-7C60/TRX40 PRO WIFI (MS-7C60), BIOS 2.81 **********
Feb 28 15:47:40 px18 kernel: [793409.709894] Node 0 active_anon:107474156kB inactive_anon:8502144kB active_file:101464kB inactive_file:10939328kB unevictable:14232808kB isolated(anon):0kB isolated(file):0kB mapped:96564kB dirty:13912kB writeback:680kB shmem:97916kB shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 1316864kB writeback_tmp:0kB kernel_stack:26816kB pagetables:318456kB all_unreclaimable? no
Feb 28 15:47:40 px18 kernel: [793409.710980] Node 0 DMA free:11812kB min:4kB low:16kB high:28kB reserved_highatomic:0KB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:15996kB managed:15908kB mlocked:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
Feb 28 15:47:40 px18 kernel: [793409.712276] lowmem_reserve[]: 0 2843 257540 257540 257540
Feb 28 15:47:40 px18 kernel: [793409.712721] Node 0 DMA32 free:1021204kB min:744kB low:3652kB high:6560kB reserved_highatomic:2048KB active_anon:1209896kB inactive_anon:44404kB active_file:8kB inactive_file:12kB unevictable:3396kB writepending:4kB present:3035988kB managed:3034860kB mlocked:3396kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
Feb 28 15:47:40 px18 kernel: [793409.714160] lowmem_reserve[]: 0 0 254696 254696 254696
Feb 28 15:47:40 px18 kernel: [793409.714669] Node 0 Normal free:675728kB min:66832kB low:327640kB high:588448kB reserved_highatomic:2048KB active_anon:106264260kB inactive_anon:8457740kB active_file:101456kB inactive_file:10939436kB unevictable:14229412kB writepending:14588kB present:265275392kB managed:260816932kB mlocked:14229412kB bounce:0kB free_pcp:3608kB local_pcp:0kB free_cma:0kB
Feb 28 15:47:40 px18 kernel: [793409.716208] lowmem_reserve[]: 0 0 0 0 0
Feb 28 15:47:40 px18 kernel: [793409.716715] Node 0 DMA: 1*4kB (U) 2*8kB (U) 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 2*4096kB (M) = 11812kB
Feb 28 15:47:40 px18 kernel: [793409.717769] Node 0 DMA32: 28453*4kB (UME) 18038*8kB (UME) 6811*16kB (MEH) 2988*32kB (UMEH) 672*64kB (UMEH) 603*128kB (UMEH) 316*256kB (UMEH) 196*512kB (UMEH) 147*1024kB (UME) 40*2048kB (UM) 6*4096kB (UM) = 1021172kB
Feb 28 15:47:40 px18 kernel: [793409.718870] Node 0 Normal: 168704*4kB (UH) 4*8kB (H) 9*16kB (H) 1*32kB (H) 1*64kB (H) 3*128kB (H) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 675472kB
Feb 28 15:47:40 px18 kernel: [793409.720055] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
Feb 28 15:47:40 px18 kernel: [793409.720659] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Feb 28 15:47:40 px18 kernel: [793409.721249] 2789918 total pagecache pages
Feb 28 15:47:40 px18 kernel: [793409.721839] 0 pages in swap cache
Feb 28 15:47:40 px18 kernel: [793409.722412] Swap cache stats: add 0, delete 0, find 0/0
Feb 28 15:47:40 px18 kernel: [793409.723011] Free swap = 0kB
Feb 28 15:47:40 px18 kernel: [793409.723616] Total swap = 0kB
Feb 28 15:47:40 px18 kernel: [793409.724206] 67081844 pages RAM
Feb 28 15:47:40 px18 kernel: [793409.724777] 0 pages HighMem/MovableOnly
Feb 28 15:47:40 px18 kernel: [793409.725369] 1114919 pages reserved
Feb 28 15:47:40 px18 kernel: [793409.725943] 0 pages hwpoisoned
Feb 28 15:47:40 px18 kernel: [793409.726531] Tasks state (memory values in pages):
Feb 28 15:47:40 px18 kernel: [793409.727123] [ pid ] uid tgid total_vm rss pgtables_bytes swapents oom_score_adj name
Feb 28 15:47:40 px18 kernel: [793409.727743] [ 1729] 0 1729 90065 3418 761856 0 -250 systemd-journal
Feb 28 15:47:40 px18 kernel: [793409.728264] [ 1748] 0 1748 20094 6243 94208 0 -1000 dmeventd
Feb 28 15:47:40 px18 kernel: [793409.728826] [ 1810] 0 1810 5665 1120 57344 0 -1000 systemd-udevd
Feb 28 15:47:40 px18 kernel: [793409.729401] [ 2263] 103 2263 1960 759 57344 0 0 rpcbind
Feb 28 15:47:40 px18 kernel: [793409.730003] [ 2281] 102 2281 2047 1007 57344 0 -900 dbus-daemon
Feb 28 15:47:40 px18 kernel: [793409.730598] [ 2284] 0 2284 37728 428 53248 0 0 lxcfs
Feb 28 15:47:40 px18 kernel: [793409.731172] [ 2292] 0 2292 1742 510 57344 0 0 ksmtuned
Feb 28 15:47:40 px18 kernel: [793409.731745] [ 2294] 0 2294 55185 1187 69632 0 0 rsyslogd
Feb 28 15:47:40 px18 kernel: [793409.732310] [ 2301] 0 2301 1051 338 45056 0 0 qmeventd
Feb 28 15:47:40 px18 kernel: [793409.732881] [ 2303] 0 2303 2833 1113 65536 0 0 smartd
Feb 28 15:47:40 px18 kernel: [793409.733433] [ 2368] 0 2368 77211 1382 114688 0 0 systemd-logind
Feb 28 15:47:40 px18 kernel: [793409.733971] [ 2373] 0 2373 543 240 40960 0 -1000 watchdog-mux
Feb 28 15:47:40 px18 kernel: [793409.734538] [ 2619] 0 2619 1137 360 53248 0 0 lxc-monitord
Feb 28 15:47:40 px18 kernel: [793409.735100] [ 2653] 0 2653 2873 132 61440 0 0 iscsid
Feb 28 15:47:40 px18 kernel: [793409.735666] [ 2655] 0 2655 2999 2937 65536 0 -17 iscsid
Feb 28 15:47:40 px18 kernel: [793409.736204] [ 2674] 0 2674 3323 1520 65536 0 -1000 sshd
Feb 28 15:47:40 px18 kernel: [793409.736734] [ 2814] 0 2814 22930 480 69632 0 0 hptsvr
Feb 28 15:47:40 px18 kernel: [793409.737234] [ 2857] 0 2857 304797 1469 233472 0 0 rrdcached
Feb 28 15:47:40 px18 kernel: [793409.737739] [ 3252] 0 3252 9997 655 77824 0 0 master
Feb 28 15:47:40 px18 kernel: [793409.738223] [ 3263] 0 3263 1671 550 53248 0 0 cron
Feb 28 15:47:40 px18 kernel: [793409.738716] [ 3284] 0 3284 70267 22588 311296 0 0 pve-firewall
Feb 28 15:47:40 px18 kernel: [793409.739209] [ 3303] 0 3303 576 127 45056 0 0 bpfilter_umh
Feb 28 15:47:40 px18 kernel: [793409.739667] [ 3313] 0 3313 81097 25560 368640 0 0 pvescheduler
Feb 28 15:47:40 px18 kernel: [793409.740096] [ 3363] 33 3363 18524 13272 192512 0 0 spiceproxy
Feb 28 15:47:40 px18 kernel: [793409.740557] [ 3549] 0 3549 9781836 8215905 67039232 0 0 kvm
Feb 28 15:47:40 px18 kernel: [793409.740978] [ 3754] 109 3754 1164 421 49152 0 0 rpc.statd
Feb 28 15:47:40 px18 kernel: [793409.741431] [ 8850] 0 8850 1446 403 49152 0 0 agetty
Feb 28 15:47:40 px18 kernel: [793409.741820] [ 146953] 0 146953 9249193 8210883 67432448 0 0 kvm
Feb 28 15:47:40 px18 kernel: [793409.742234] [4087683] 0 4087683 8877604 271038 3350528 0 0 kvm
Feb 28 15:47:40 px18 kernel: [793409.742620] [4087888] 0 4087888 8875317 270585 3346432 0 0 kvm
Feb 28 15:47:40 px18 kernel: [793409.742975] [2599489] 0 2599489 8895410 271540 3346432 0 0 kvm
Feb 28 15:47:40 px18 kernel: [793409.743315] [2014090] 106 2014090 10076 683 69632 0 0 qmgr
Feb 28 15:47:40 px18 kernel: [793409.743665] [3098914] 0 3098914 1196937 797335 7909376 0 0 kvm
Feb 28 15:47:40 px18 kernel: [793409.743984] [1689966] 0 1689966 3655385 3557015 28585984 0 0 corosync
Feb 28 15:47:40 px18 kernel: [793409.744331] [3045764] 101 3045764 4743 750 61440 0 0 chronyd
Feb 28 15:47:40 px18 kernel: [793409.744675] [3045765] 101 3045765 2695 549 61440 0 0 chronyd
Feb 28 15:47:40 px18 kernel: [793409.745013] [3055383] 0 3055383 1083607 881 602112 0 0 pve-lxc-syscall
Feb 28 15:47:40 px18 kernel: [793409.745353] [3055669] 0 3055669 40996 938 73728 0 0 zed
Feb 28 15:47:40 px18 kernel: [793409.745705] [3065944] 0 3065944 82453 24582 352256 0 0 pve-ha-lrm
Feb 28 15:47:40 px18 kernel: [793409.746050] [3065953] 0 3065953 82540 24606 352256 0 0 pve-ha-crm
Feb 28 15:47:40 px18 kernel: [793409.746429] [3067214] 0 3067214 18491090 16812176 136097792 0 0 kvm
Feb 28 15:47:40 px18 kernel: [793409.746744] [3595616] 33 3595616 18590 12652 180224 0 0 spiceproxy work
Feb 28 15:47:40 px18 kernel: [793409.747077] [3595626] 0 3595626 20035 332 57344 0 0 pvefw-logger
Feb 28 15:47:40 px18 kernel: [793409.747436] [2815461] 0 2815461 3701 1699 73728 0 0 sshd
Feb 28 15:47:40 px18 kernel: [793409.747790] [2815700] 0 2815700 3615 1483 69632 0 0 sshd
Feb 28 15:47:40 px18 kernel: [793409.748148] [2815711] 0 2815711 3969 1258 69632 0 0 systemd
Feb 28 15:47:40 px18 kernel: [793409.748532] [2815712] 0 2815712 42253 1017 98304 0 0 (sd-pam)
Feb 28 15:47:40 px18 kernel: [793409.748883] [2816739] 0 2816739 1993 772 53248 0 0 bash
Feb 28 15:47:40 px18 kernel: [793409.749216] [2816752] 0 2816752 1454 1046 57344 0 0 sftp-server
Feb 28 15:47:40 px18 kernel: [793409.749575] [2857948] 0 2857948 151297 13928 393216 0 0 pmxcfs
Feb 28 15:47:40 px18 kernel: [793409.749921] [2860000] 33 2860000 86378 30687 397312 0 0 pveproxy
Feb 28 15:47:40 px18 kernel: [793409.750268] [2860001] 33 2860001 88838 32741 417792 0 0 pveproxy worker
Feb 28 15:47:40 px18 kernel: [793409.750606] [2860002] 33 2860002 88804 32733 417792 0 0 pveproxy worker
Feb 28 15:47:40 px18 kernel: [793409.750974] [2860003] 33 2860003 88847 32616 417792 0 0 pveproxy worker
Feb 28 15:47:40 px18 kernel: [793409.751299] [2862028] 0 2862028 86033 30296 389120 0 0 pvedaemon
Feb 28 15:47:40 px18 kernel: [793409.751647] [2862029] 0 2862029 88434 32015 413696 0 0 pvedaemon worke
Feb 28 15:47:40 px18 kernel: [793409.751992] [2862030] 0 2862030 88435 32017 413696 0 0 pvedaemon worke
Feb 28 15:47:40 px18 kernel: [793409.752361] [2862031] 0 2862031 88557 32191 413696 0 0 pvedaemon worke
Feb 28 15:47:40 px18 kernel: [793409.752664] [2862269] 0 2862269 67481 21946 282624 0 0 pvestatd
Feb 28 15:47:40 px18 kernel: [793409.753005] [3354585] 106 3354585 10064 1282 73728 0 0 pickup
Feb 28 15:47:40 px18 kernel: [793409.753355] [3361158] 0 3361158 1326 143 49152 0 0 sleep
Feb 28 15:47:40 px18 kernel: [793409.753683] [3369244] 0 3369244 2157 557 53248 0 0 zfs
Feb 28 15:47:40 px18 kernel: [793409.754015] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=pvestatd.service,mems_allowed=0,global_oom,task_memcg=/qemu.slice/249.scope,task=kvm,pid=3067214,uid=0
Feb 28 15:47:40 px18 kernel: [793409.754746] Out of memory: Killed process 3067214 (kvm) total-vm:73964360kB, anon-rss:67242580kB, file-rss:6124kB, shmem-rss:0kB, UID:0 pgtables:132908kB oom_score_adj:0
Feb 28 15:47:40 px18 systemd[1]: 249.scope: A process of this unit has been killed by the OOM killer.
Feb 28 15:47:42 px18 kernel: [793412.694251] zd0: p1 p2 p3
Feb 28 15:47:43 px18 systemd[1]: Starting LVM event activation on device 230:3...
Feb 28 15:47:43 px18 lvm[3369864]: pvscan[3369864] /dev/zd0p3 excluded by filters: device is rejected by filter config.
Feb 28 15:47:44 px18 systemd[1]: Finished LVM event activation on device 230:3.
Feb 28 15:47:45 px18 kernel: [793415.627991] oom_reaper: reaped process 3067214 (kvm), now anon-rss:0kB, file-rss:196kB, shmem-rss:0kB
Feb 28 15:47:46 px18 kernel: [793415.780457] vmbr0: port 8(tap249i0) entered disabled state
Feb 28 15:47:46 px18 kernel: [793415.786690] vmbr0: port 8(tap249i0) entered disabled state
Feb 28 15:47:46 px18 systemd[1]: 249.scope: Succeeded.
Feb 28 15:47:46 px18 systemd[1]: 249.scope: Consumed 4d 15h 19min 52.366s CPU time.
 
chrigiboy said:
ich habe ein ähnliches Problem. Der Server hat 256 GByte Ram. 200 Gbyte ram sind den Maschinen zugewiesen. Doch der Server hat plötzlich 100% Speicherauslastung. Warum?
Click to expand...
Das hängt von vielem ab:
- Wie groß ist der ARC (standard 50% des Arbeitsspeichers)
- Wieviel ist mit KSM geteilt
- Wieviel Swap gibt es
- Wie ist die Swapiness eingestellt.
 
Ich habe da eigentlich die Standard Installation von Proxmox genommen und nicht viel umgestellt.

KSM Sharing ist 11 GByte,
Swap steht N/A. (ich weiss nicht warum Proxmox kein SWAP eingerichtet hat), bei einem anderen Server gibt es ein SWAP Drive

ARC:
# cat /sys/module/zfs/parameters/zfs_arc_min
0
# cat /sys/module/zfs/parameters/zfs_arc_max
0

Swapiness:
# cat /proc/sys/vm/swappiness
60
 
chrigiboy said:
Swap steht N/A. (ich weiss nicht warum Proxmox kein SWAP eingerichtet hat), bei einem anderen Server gibt es ein SWAP Drive
Click to expand...
Swap auf ZFS wird nicht standardmäßig angelegt, da es zu Problemen kommen kann. Es wird immer empfohlen, dass auf ein nicht ZFS-basiertes Gerät zu legen.

chrigiboy said:
ARC:
# cat /sys/module/zfs/parameters/zfs_arc_min
0
# cat /sys/module/zfs/parameters/zfs_arc_max
0
Click to expand...
Also ZFS verwendet maximal die häfte Arbeitsspeichers, bei dir somit 128 GB.

chrigiboy said:
Swapiness:
# cat /proc/sys/vm/swappiness
60
Click to expand...
Würde ich auf 1 oder sogar 0 stellen. Da scheiden sich die Geister was da sinnvoller ist - ohne Swap wahrscheinlich 0 sinnvoller.

Aber allgemein würd ich sagen ja, dein System war zu 100% speichertechnisch ausgelastet und dann hat der OOM angefangen zu arbeiten. Also alles so wie er es machen soll. Wichtig ist auch zu wissen, dass der OOM auch aufschlägt wenn in Summe noch genug Platz frei ist, er jedoch keinen zusammenhängen Speicherbereich mehr finden kann wenn ein Prozess einen größeren zusammenhängenden Block anfordert.

Um deine Situtation ohne Swap etwas zu entlasten, würde ich vorschlagen du installierst dir mal noch zram-tools und legst dir ein komprimiertes SWAP im Arbeitsspeicher an. Somit bekommst du ein paar MB "frei". Auch solltest du kontrollieren, ob alle VMs eingestellt haben, dass kein Caching verwendet wird, da es dadurch zu doppeltem Caching kommen kann (im ARC und im verbleibenden RAM nochmal). Sonst hilft es natürlich auch noch ein entsprechendes Swap-Gerät zu verwenden.
 
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom