Proxmox keep killed my vm , anyone can help ? please

D

dickson lee

Member
Jan 30, 2023
31
1
8
Jan 07 05:21:47 kng kernel: kthreadd invoked oom-killer: gfp_mask=0x2dc2(GFP_KERNEL|__GFP_HIGHMEM|__GFP_NOWARN|__GFP_ZERO), order=0, oom_score_adj=0
Jan 07 05:21:50 kng kernel: CPU: 6 PID: 2 Comm: kthreadd Tainted: P O 5.15.102-1-pve #1
Jan 07 05:21:50 kng kernel: Hardware name: ASUS System Product Name/PRIME B550M-K, BIOS 0805 07/01/2020
Jan 07 05:21:50 kng kernel: Call Trace:
Jan 07 05:21:50 kng kernel: <TASK>
Jan 07 05:21:50 kng kernel: dump_stack_lvl+0x4a/0x63
Jan 07 05:21:50 kng kernel: dump_stack+0x10/0x16
Jan 07 05:21:50 kng kernel: dump_header+0x53/0x225
Jan 07 05:21:50 kng kernel: oom_kill_process.cold+0xb/0x10
Jan 07 05:21:50 kng kernel: out_of_memory+0x1dc/0x530
Jan 07 05:21:50 kng kernel: __alloc_pages_slowpath.constprop.0+0xd68/0xe80
Jan 07 05:21:50 kng kernel: __alloc_pages+0x30c/0x320
Jan 07 05:21:50 kng kernel: alloc_pages+0x9e/0x1e0
Jan 07 05:21:50 kng kernel: __vmalloc_node_range+0x161/0x320
Jan 07 05:21:50 kng kernel: copy_process+0x6c9/0x1cc0
Jan 07 05:21:50 kng kernel: ? kernel_clone+0x9d/0x3f0
Jan 07 05:21:50 kng kernel: kernel_clone+0x9d/0x3f0
Jan 07 05:21:50 kng kernel: ? __schedule+0x356/0x1740
Jan 07 05:21:50 kng kernel: kernel_thread+0x55/0x70
Jan 07 05:21:50 kng kernel: ? set_kthread_struct+0x50/0x50
Jan 07 05:21:50 kng kernel: kthreadd+0x284/0x2e0
Jan 07 05:21:50 kng kernel: ? kthread_is_per_cpu+0x30/0x30
Jan 07 05:21:50 kng kernel: ret_from_fork+0x22/0x30
Jan 07 05:21:50 kng kernel: </TASK>
Jan 07 05:21:50 kng kernel: Mem-Info:
Jan 07 05:21:50 kng kernel: active_anon:4347828 inactive_anon:2856677 isolated_anon:0
active_file:229 inactive_file:0 isolated_file:0
unevictable:3002 dirty:49 writeback:9
slab_reclaimable:8343 slab_unreclaimable:2579393
mapped:12398 shmem:9675 pagetables:16594 bounce:0
kernel_misc_reclaimable:0
free:121973 free_pcp:1064 free_cma:0
Jan 07 05:21:50 kng kernel: Node 0 active_anon:17391312kB inactive_anon:11426708kB active_file:916kB inactive_file:0kB unevictable:12008kB isolated(anon):0kB isolated(file):0kB mapped:49592kB dirty:196kB writeback:36kB shmem:38700kB shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 1843200kB writeback_tmp:0kB kernel_stack:7472kB pagetables:66376kB all_unreclaimable? no
Jan 07 05:21:50 kng kernel: Node 0 DMA free:11260kB min:20kB low:32kB high:44kB reserved_highatomic:0KB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:15992kB managed:15360kB mlocked:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
Jan 07 05:21:50 kng kernel: lowmem_reserve[]: 0 3343 47972 47972 47972
Jan 07 05:21:50 kng kernel: Node 0 DMA32 free:182812kB min:4708kB low:8128kB high:11548kB reserved_highatomic:2048KB active_anon:1962136kB inactive_anon:658192kB active_file:272kB inactive_file:76kB unevictable:0kB writepending:108kB present:3549124kB managed:3483384kB mlocked:0kB bounce:0kB free_pcp:576kB local_pcp:0kB free_cma:0kB
Jan 07 05:21:50 kng kernel: lowmem_reserve[]: 0 0 44628 44628 44628
Jan 07 05:21:50 kng kernel: Node 0 Normal free:293820kB min:294228kB low:339928kB high:385628kB reserved_highatomic:0KB active_anon:15429176kB inactive_anon:10768516kB active_file:1740kB inactive_file:240kB unevictable:12008kB writepending:124kB present:46648832kB managed:45707348kB mlocked:12008kB bounce:0kB free_pcp:3680kB local_pcp:236kB free_cma:0kB
Jan 07 05:21:50 kng kernel: lowmem_reserve[]: 0 0 0 0 0
Jan 07 05:21:50 kng kernel: Node 0 DMA: 1*4kB (U) 1*8kB (U) 1*16kB (U) 1*32kB (U) 1*64kB (U) 1*128kB (U) 1*256kB (U) 1*512kB (U) 0*1024kB 1*2048kB (M) 2*4096kB (M) = 11260kB
Jan 07 05:21:50 kng kernel: Node 0 DMA32: 1288*4kB (UME) 7528*8kB (UME) 6151*16kB (UME) 85*32kB (UMEH) 178*64kB (UMEH) 30*128kB (UMEH) 1*256kB (H) 1*512kB (H) 1*1024kB (H) 0*2048kB 0*4096kB = 183536kB
Jan 07 05:21:50 kng kernel: Node 0 Normal: 73405*4kB (U) 0*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 293620kB
Jan 07 05:21:50 kng kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
Jan 07 05:21:50 kng kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Jan 07 05:21:50 kng kernel: 12659 total pagecache pages
Jan 07 05:21:50 kng kernel: 0 pages in swap cache
Jan 07 05:21:50 kng kernel: Swap cache stats: add 0, delete 0, find 0/0
Jan 07 05:21:50 kng kernel: Free swap = 0kB
Jan 07 05:21:50 kng kernel: Total swap = 0kB
Jan 07 05:21:50 kng kernel: 12553487 pages RAM
Jan 07 05:21:50 kng kernel: 0 pages HighMem/MovableOnly
Jan 07 05:21:50 kng kernel: 251964 pages reserved
Jan 07 05:21:50 kng kernel: 0 pages hwpoisoned
Jan 07 05:21:50 kng kernel: Tasks state (memory values in pages):
Jan 07 05:21:50 kng kernel: [ pid ] uid tgid total_vm rss pgtables_bytes swapents oom_score_adj name
Jan 07 05:21:50 kng kernel: [ 1870] 103 1870 1939 455 57344 0 0 rpcbind
Jan 07 05:21:50 kng kernel: [ 1875] 102 1875 2042 611 53248 0 -900 dbus-daemon
Jan 07 05:21:50 kng kernel: [ 1878] 0 1878 37745 299 53248 0 -1000 lxcfs
Jan 07 05:21:50 kng kernel: [ 1881] 0 1881 1758 383 49152 0 0 ksmtuned
Jan 07 05:21:50 kng kernel: [ 1998] 0 1998 69482 400 81920 0 0 pve-lxc-syscall
Jan 07 05:21:50 kng kernel: [ 2004] 0 2004 55177 783 77824 0 0 rsyslogd
Jan 07 05:21:50 kng kernel: [ 2052] 0 2052 2920 799 65536 0 0 smartd
Jan 07 05:21:50 kng kernel: [ 2055] 0 2055 11674 783 90112 0 0 systemd-logind
Jan 07 05:21:50 kng kernel: [ 2056] 0 2056 582 286 40960 0 -1000 watchdog-mux
Jan 07 05:21:50 kng kernel: [ 2058] 0 2058 59332 734 77824 0 0 zed
Jan 07 05:21:50 kng kernel: [ 2086] 0 2086 1059 291 45056 0 0 qmeventd
Jan 07 05:21:50 kng kernel: [ 2210] 0 2210 2172 398 57344 0 0 lxc-monitord
Jan 07 05:21:50 kng kernel: [ 2267] 0 2267 2885 139 65536 0 0 iscsid
Jan 07 05:21:50 kng kernel: [ 2268] 0 2268 3011 2955 65536 0 -17 iscsid
Jan 07 05:21:50 kng kernel: [ 2295] 0 2295 1467 238 49152 0 0 agetty
Jan 07 05:21:50 kng kernel: [ 2309] 0 2309 3303 1500 69632 0 -1000 sshd
Jan 07 05:21:50 kng kernel: [ 2326] 101 2326 4656 575 57344 0 0 chronyd
Jan 07 05:21:50 kng kernel: [ 2329] 101 2329 2574 350 57344 0 0 chronyd
Jan 07 05:21:50 kng kernel: [ 2405] 0 2405 218554 894 184320 0 0 rrdcached
Jan 07 05:21:50 kng kernel: [ 2436] 0 2436 9952 601 69632 0 0 master
Jan 07 05:21:50 kng kernel: [ 2438] 106 2438 10063 618 77824 0 0 qmgr
Jan 07 05:21:50 kng kernel: [ 2447] 0 2447 200370 12706 376832 0 0 pmxcfs
Jan 07 05:21:50 kng kernel: [ 2649] 0 2649 1658 532 53248 0 0 cron
Jan 07 05:21:50 kng kernel: [ 2668] 0 2668 70908 22869 299008 0 0 pve-firewall
Jan 07 05:21:50 kng kernel: [ 2672] 0 2672 69225 23746 331776 0 0 pvestatd
Jan 07 05:21:50 kng kernel: [ 2673] 0 2673 615 228 49152 0 0 bpfilter_umh
Jan 07 05:21:50 kng kernel: [ 2696] 0 2696 89136 31372 401408 0 0 pvedaemon
Jan 07 05:21:50 kng kernel: [ 2706] 0 2706 86182 26128 376832 0 0 pve-ha-crm
Jan 07 05:21:50 kng kernel: [ 2710] 33 2710 89474 32741 442368 0 0 pveproxy
Jan 07 05:21:50 kng kernel: [ 2717] 33 2717 19107 13341 188416 0 0 spiceproxy
Jan 07 05:21:50 kng kernel: [ 2719] 0 2719 86036 26088 368640 0 0 pve-ha-lrm
Jan 07 05:21:50 kng kernel: [ 2945] 0 2945 91644 32988 413696 0 0 pvescheduler
Jan 07 05:21:50 kng kernel: [3851515] 0 3851515 5583 1049 69632 0 -1000 systemd-udevd
Jan 07 05:21:50 kng kernel: [1900960] 0 1900960 91281 32486 421888 0 0 pvedaemon worke
Jan 07 05:21:50 kng kernel: [2425911] 0 2425911 8006199 7325287 59748352 0 0 kvm
Jan 07 05:21:50 kng kernel: [2426180] 0 2426180 91245 32349 421888 0 0 pvedaemon worke
Jan 07 05:21:50 kng kernel: [2547141] 0 2547141 91245 32269 421888 0 0 pvedaemon worke
Jan 07 05:21:50 kng kernel: [1558563] 33 1558563 19163 12631 172032 0 0 spiceproxy work
Jan 07 05:21:50 kng kernel: [1558575] 33 1558575 89573 32338 397312 0 0 pveproxy worker
Jan 07 05:21:50 kng kernel: [1558576] 33 1558576 89573 32497 397312 0 0 pveproxy worker
Jan 07 05:21:50 kng kernel: [1558577] 33 1558577 89573 32365 397312 0 0 pveproxy worker
Jan 07 05:21:50 kng kernel: [1558721] 0 1558721 20041 285 57344 0 0 pvefw-logger
Jan 07 05:21:50 kng kernel: [3684985] 106 3684985 10016 584 77824 0 0 pickup
Jan 07 05:21:50 kng kernel: [3799286] 0 3799286 5894 1048 86016 0 -250 systemd-journal
Jan 07 05:21:50 kng kernel: [3842519] 0 3842519 93452 32937 421888 0 0 pvescheduler
Jan 07 05:21:50 kng kernel: [3842520] 0 3842520 91644 32657 413696 0 0 pvescheduler
Jan 07 05:21:50 kng kernel: [3846543] 0 3846543 1367 213 49152 0 0 sleep
Jan 07 05:21:50 kng kernel: oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/qemu.slice/100.scope,task=kvm,pid=2425911,uid=0
Jan 07 05:21:50 kng kernel: Out of memory: Killed process 2425911 (kvm) total-vm:32024796kB, anon-rss:29298948kB, file-rss:2200kB, shmem-rss:0kB, UID:0 pgtables:58348kB oom_score_adj:0
Jan 07 05:21:50 kng kernel: oom_reaper: reaped process 2425911 (kvm), now anon-rss:0kB, file-rss:36kB, shmem-rss:0kB
Jan 07 05:21:50 kng kernel: fwbr100i0: port 2(tap100i0) entered disabled state
Jan 07 05:21:50 kng kernel: fwbr100i0: port 2(tap100i0) entered disabled state
Jan 07 05:21:50 kng systemd[1]: 100.scope: A process of this unit has been killed by the OOM killer.
Jan 07 05:21:50 kng pve-ha-crm[2706]: loop take too long (40 seconds)
Jan 07 05:21:50 kng systemd[1]: 100.scope: Succeeded.
Jan 07 05:21:50 kng pvescheduler[3842520]: jobs: cfs-lock 'file-jobs_cfg' error: got lock request timeout
Jan 07 05:21:50 kng systemd[1]: 100.scope: Consumed 23h 2min 12.368s CPU time.


server spec below
CPU usage 8.43% of 16 CPU(s)
IO delay 17.37%
Load average 5.17,5.28,5.50
RAM usage 82.49% (38.71 GiB of 46.93 GiB)
KSM sharing 1.60 GiB
/ HD space 7.51% (60.55 GiB of 806.53 GiB)
SWAP usage N/A
CPU(s) 16 x AMD Ryzen 7 3700X 8-Core Processor (1 Socket)
Kernel Version Linux 5.15.102-1-pve #1 SMP PVE 5.15.102-1 (2023-03-14T13:48Z)
PVE Manager Version pve-manager/7.4-3/9002ab8a
Repository Status Proxmox VE updates Non production-ready repository enabled!
 
dickson lee said:
Jan 07 05:21:47 kng kernel: kthreadd invoked oom-killer:
Click to expand...
Just install more RAM. Alternatively assign less to VMs...

(To clarify: oom = Out-of-Memory...)


Edit: adding "swap" is not a solution but it might help a little bit. Nevertheless you can relieve pressure by some percent by utilizing "zram", which will compress some RAM:
Code: 
:~# apt show zram-tools
...
Description: utilities for working with zram
 zram is a Linux kernel module that allows you to set up compressed
 filesystems in RAM.
 
Last edited:
  • Like
Reactions: Kingneutron and dickson lee
UdoB said:
di
Click to expand...

UdoB said:
Just install more RAM. Alternatively assign less to VMs...

(To clarify: oom = Out-of-Memory...)


Edit: adding "swap" is not a solution but it might help a little bit. Nevertheless you can relieve pressure by some percent by utilizing "zram", which will compress some RAM:
Code: 
:~# apt show zram-tools
...
Description: utilities for working with zram
 zram is a Linux kernel module that allows you to set up compressed
 filesystems in RAM.
Click to expand...
after install zram need configure?
 
dickson lee said:
after install zram need configure?
Click to expand...
If the defaults do not fit your needs: yes.

After installation you might read man zramswap, which is very short. Then you will edit /etc/default/zramswap. At the end you get here:
Code: 
~# zramswap  status
NAME       ALGORITHM DISKSIZE DATA COMPR TOTAL STREAMS MOUNTPOINT
/dev/zram0 lz4             8G   4K   64B   20K      16
 
  • Like
Reactions: Kingneutron
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom