How to fix this issue? My VM100 every morning will be killed by OOM.

[dickson lee]

Oct 01 05:02:42 kng kernel: [ pid ] uid tgid total_vm rss pgtables_bytes swapents oom_score_adj name
Oct 01 05:02:42 kng kernel: [ 1157] 0 1157 5682 844 65536 0 -1000 systemd-udevd
Oct 01 05:02:42 kng kernel: [ 1548] 103 1548 1939 470 57344 0 0 rpcbind
Oct 01 05:02:42 kng kernel: [ 1658] 102 1658 2042 513 57344 0 -900 dbus-daemon
Oct 01 05:02:42 kng kernel: [ 1661] 0 1661 37745 298 61440 0 -1000 lxcfs
Oct 01 05:02:42 kng kernel: [ 1665] 0 1665 1758 392 45056 0 0 ksmtuned
Oct 01 05:02:42 kng kernel: [ 1668] 0 1668 69482 398 81920 0 0 pve-lxc-syscall
Oct 01 05:02:42 kng kernel: [ 1670] 0 1670 55177 708 81920 0 0 rsyslogd
Oct 01 05:02:42 kng kernel: [ 1671] 0 1671 2920 804 61440 0 0 smartd
Oct 01 05:02:42 kng kernel: [ 1672] 0 1672 11639 743 90112 0 0 systemd-logind
Oct 01 05:02:42 kng kernel: [ 1673] 0 1673 582 223 40960 0 -1000 watchdog-mux
Oct 01 05:02:42 kng kernel: [ 1675] 0 1675 59332 704 77824 0 0 zed
Oct 01 05:02:42 kng kernel: [ 1702] 0 1702 1059 221 49152 0 0 qmeventd
Oct 01 05:02:42 kng kernel: [ 1815] 0 1815 2172 599 57344 0 0 lxc-monitord
Oct 01 05:02:42 kng kernel: [ 1868] 0 1868 2885 140 57344 0 0 iscsid
Oct 01 05:02:42 kng kernel: [ 1869] 0 1869 3011 2975 61440 0 -17 iscsid
Oct 01 05:02:42 kng kernel: [ 1872] 0 1872 3303 964 65536 0 -1000 sshd
Oct 01 05:02:42 kng kernel: [ 1873] 0 1873 5875 359 61440 0 0 apcupsd
Oct 01 05:02:42 kng kernel: [ 1917] 0 1917 1467 223 49152 0 0 agetty
Oct 01 05:02:42 kng kernel: [ 1944] 101 1944 4656 610 57344 0 0 chronyd
Oct 01 05:02:42 kng kernel: [ 1950] 101 1950 2574 464 57344 0 0 chronyd
Oct 01 05:02:42 kng kernel: [ 2050] 0 2050 9952 609 73728 0 0 master
Oct 01 05:02:42 kng kernel: [ 2052] 106 2052 10063 639 69632 0 0 qmgr
Oct 01 05:02:42 kng kernel: [ 2056] 0 2056 200121 718 184320 0 0 rrdcached
Oct 01 05:02:42 kng kernel: [ 2064] 0 2064 166572 14747 389120 0 0 pmxcfs
Oct 01 05:02:42 kng kernel: [ 2210] 0 2210 1658 567 49152 0 0 cron
Oct 01 05:02:42 kng kernel: [ 2227] 0 2227 70901 22866 299008 0 0 pve-firewall
Oct 01 05:02:42 kng kernel: [ 2229] 0 2229 68602 23175 327680 0 0 pvestatd
Oct 01 05:02:42 kng kernel: [ 2232] 0 2232 615 227 45056 0 0 bpfilter_umh
Oct 01 05:02:42 kng kernel: [ 2255] 0 2255 89143 31617 405504 0 0 pvedaemon
Oct 01 05:02:42 kng kernel: [ 2265] 0 2265 86185 26275 352256 0 0 pve-ha-crm
Oct 01 05:02:42 kng kernel: [ 2266] 33 2266 89472 32762 438272 0 0 pveproxy
Oct 01 05:02:42 kng kernel: [ 2274] 33 2274 19097 13438 200704 0 0 spiceproxy
Oct 01 05:02:42 kng kernel: [ 2276] 0 2276 86035 26049 372736 0 0 pve-ha-lrm
Oct 01 05:02:42 kng kernel: [ 2391] 0 2391 7949833 7612302 61988864 0 0 kvm
Oct 01 05:02:42 kng kernel: [ 2556] 0 2556 85563 26876 356352 0 0 pvescheduler
Oct 01 05:02:42 kng kernel: [3371437] 0 3371437 91379 32070 421888 0 0 task UPID:kng:0
Oct 01 05:02:42 kng kernel: [3371479] 0 3371479 4298 796 69632 0 0 termproxy
Oct 01 05:02:42 kng kernel: [3371499] 0 3371499 19214 11472 192512 0 0 pveupgrade
Oct 01 05:02:42 kng kernel: [3371505] 0 3371505 17896 5123 180224 0 0 apt-get
Oct 01 05:02:42 kng kernel: [3546568] 33 3546568 92996 33878 434176 0 0 pveproxy worker
Oct 01 05:02:42 kng kernel: [3675617] 0 3675617 1977 1074 53248 0 0 dpkg
Oct 01 05:02:42 kng kernel: [3930432] 0 3930432 91275 32565 421888 0 0 pvedaemon worke
Oct 01 05:02:42 kng kernel: [3993843] 0 3993843 91258 32545 421888 0 0 pvedaemon worke
Oct 01 05:02:42 kng kernel: [ 6403] 0 6403 91218 32297 421888 0 0 pvedaemon worke
Oct 01 05:02:42 kng kernel: [2900578] 33 2900578 19162 12712 180224 0 0 spiceproxy work
Oct 01 05:02:42 kng kernel: [2900587] 0 2900587 20041 316 53248 0 0 pvefw-logger
Oct 01 05:02:42 kng kernel: [2900593] 33 2900593 89572 32442 397312 0 0 pveproxy worker
Oct 01 05:02:42 kng kernel: [2900594] 33 2900594 89572 32428 397312 0 0 pveproxy worker
Oct 01 05:02:42 kng kernel: [2900595] 33 2900595 89572 32400 397312 0 0 pveproxy worker
Oct 01 05:02:42 kng kernel: [ 693224] 106 693224 10016 613 73728 0 0 pickup
Oct 01 05:02:42 kng kernel: [ 938186] 0 938186 8002 1107 73728 0 -250 systemd-journal
Oct 01 05:02:42 kng kernel: [1226807] 0 1226807 1367 202 49152 0 0 sleep
Oct 01 05:02:42 kng kernel: oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=qemu.slice,mems_allowed=0,global_oom,task_memcg=/qemu.slice/100.scope,task=kvm,pid=2391,uid=0
Oct 01 05:02:42 kng kernel: Out of memory: Killed process 2391 (kvm) total-vm:31799332kB, anon-rss:30447304kB, file-rss:2032kB, shmem-rss:0kB, UID:0 pgtables:60536kB oom_score_adj:0
Oct 01 05:02:42 kng kernel: oom_reaper: reaped process 2391 (kvm), now anon-rss:0kB, file-rss:36kB, shmem-rss:0kB

 

[esi_y]

Well, unless it has some memory leak issue which you have to solve within that VM, you can only increase RAM or SWAP, try using KSM [1]?

[1] https://pve.proxmox.com/wiki/Kernel_Samepage_Merging_(KSM)

 

[dickson lee]

Well, unless it has some memory leak issue which you have to solve within that VM, you can only increase RAM or SWAP, try using KSM [1]?

[1] https://pve.proxmox.com/wiki/Kernel_Samepage_Merging_(KSM)

Disabling KSM​

To see if KSM is active, you can check the output of:
# systemctl status ksmtuned
If it is, it can be disabled immediately with:
# systemctl disable --now ksmtuned
Finally, to unmerge all the currently merged pages, run:
# echo 2 > /sys/kernel/mm/ksm/run

so i follow this method?

disable it?

 

[esi_y]

Disabling KSM​

To see if KSM is active, you can check the output of:
# systemctl status ksmtuned
If it is, it can be disabled immediately with:
# systemctl disable --now ksmtuned
Finally, to unmerge all the currently merged pages, run:
# echo 2 > /sys/kernel/mm/ksm/run

so i follow this method?

disable it?

No, no, it should be enabled, but do you want to provide more details on what kind of VM is doing this, what is the system configuration etc? You are basically running out of RAM, it's NOT a bug to have OOM reaper step in, question is, why you are running out of it.

 

[dickson lee]

No, no, it should be enabled, but do you want to provide more details on what kind of VM is doing this, what is the system configuration etc? You are basically running out of RAM, it's a bug to have OOM reaper step in, question is, why you are running out of it.

i running backup vm every 4 am

 

[iwik]

Add more memory to server or power off some virtual machines. Not good idea to swap virtual machines, they lags very much. And you are wasting your SSDs when you swap on them.
EDIT: If you are using ZFS it may be related to zfs eating too much memory. In older PVE setups, there was default zfs limit - 50% of ram. See this wiki how to set some proper limit https://pve.proxmox.com/wiki/ZFS_on_Linux#sysadmin_zfs_limit_memory_usage

 

[esi_y]

i running backup vm every 4 am

My bad, that was a typo. It is NOT a bug for OOM reaper to take out the most RAM hungry process when it is running out of memory.

Is your system configuration secret?

 

[dickson lee]

My bad, that was a typo. It is NOT a bug for OOM reaper to take out the most RAM hungry process when it is running out of memory.

Is your system configuration secret?

nope like norma configure only

 

[esi_y]

I can only repeat, as there's nothing to catch on:

do you want to provide more details on what kind of VM is doing this

And:

what is the system configuration etc?

Literally the hardware specs.

You are basically running out of RAM, it's NOT a bug to have OOM reaper step in, question is, why you are running out of it.