Vms suddenly stopped

D

Daniel Zuwala

New Member
Dec 30, 2016
5
0
1
45
Hi,

I'm running a proxmox server for a few days where I have about 10 running vms. Everything went fine, until yesterday, when the backup was running (snapshot mode). It seems to me that the server run out of memory and killed 3 running vm. It would be ok, if the server lacks memory, but it has 32Gb of RAM and only 15Gb RAM are used for the running vms. Maybe I don't understand the error log and I would appreciate what you think it could be.

Thanks and sorry for the very long log file...

Code: 
Feb  3 00:09:01 px01 kernel: [1150855.552119] sh cpuset=/ mems_allowed=0
Feb  3 00:09:01 px01 kernel: [1150855.552123] CPU: 2 PID: 22452 Comm: sh Tainted: G           O    4.4.35-1-pve #1
Feb  3 00:09:01 px01 kernel: [1150855.552124] Hardware name:                  /DH67BL, BIOS BLH6710H.86A.0160.2012.1204.1156 12/04/2012
Feb  3 00:09:01 px01 kernel: [1150855.552125]  0000000000000286 00000000fd771911 ffff880001dfbb50 ffffffff813f9743
Feb  3 00:09:01 px01 kernel: [1150855.552127]  ffff880001dfbd40 0000000000000000 ffff880001dfbbb8 ffffffff8120adcb
Feb  3 00:09:01 px01 kernel: [1150855.552128]  00000000fd771911 00000000ffffffff 0000000000000000 0000000000000000
Feb  3 00:09:01 px01 kernel: [1150855.552129] Call Trace:
Feb  3 00:09:01 px01 kernel: [1150855.552135]  [<ffffffff813f9743>] dump_stack+0x63/0x90
Feb  3 00:09:01 px01 kernel: [1150855.552138]  [<ffffffff8120adcb>] dump_header+0x67/0x1d5
Feb  3 00:09:01 px01 kernel: [1150855.552141]  [<ffffffff811925c5>] oom_kill_process+0x205/0x3c0
Feb  3 00:09:01 px01 kernel: [1150855.552142]  [<ffffffff81192a17>] out_of_memory+0x237/0x4a0
Feb  3 00:09:01 px01 kernel: [1150855.552145]  [<ffffffff81198d0e>] __alloc_pages_nodemask+0xcee/0xe20
Feb  3 00:09:01 px01 kernel: [1150855.552147]  [<ffffffff81198e8b>] alloc_kmem_pages_node+0x4b/0xd0
Feb  3 00:09:01 px01 kernel: [1150855.552150]  [<ffffffff8107f053>] copy_process+0x1c3/0x1c00
Feb  3 00:09:01 px01 kernel: [1150855.552153]  [<ffffffff813941b0>] ? apparmor_file_alloc_security+0x60/0x240
Feb  3 00:09:01 px01 kernel: [1150855.552156]  [<ffffffff813494b3>] ? security_file_alloc+0x33/0x50
Feb  3 00:09:01 px01 kernel: [1150855.552158]  [<ffffffff81080c20>] _do_fork+0x80/0x360
Feb  3 00:09:01 px01 kernel: [1150855.552161]  [<ffffffff810917ff>] ? sigprocmask+0x6f/0xa0
Feb  3 00:09:01 px01 kernel: [1150855.552163]  [<ffffffff81080fa9>] SyS_clone+0x19/0x20
Feb  3 00:09:01 px01 kernel: [1150855.552165]  [<ffffffff8185c276>] entry_SYSCALL_64_fastpath+0x16/0x75
Feb  3 00:09:01 px01 kernel: [1150855.552166] Mem-Info:
Feb  3 00:09:01 px01 kernel: [1150855.552169] active_anon:3699182 inactive_anon:430015 isolated_anon:0
Feb  3 00:09:01 px01 kernel: [1150855.552169]  active_file:1845960 inactive_file:1846262 isolated_file:0
Feb  3 00:09:01 px01 kernel: [1150855.552169]  unevictable:880 dirty:8 writeback:250436 unstable:471582
Feb  3 00:09:01 px01 kernel: [1150855.552169]  slab_reclaimable:244063 slab_unreclaimable:34446
Feb  3 00:09:01 px01 kernel: [1150855.552169]  mapped:25040 shmem:33352 pagetables:14307 bounce:0
Feb  3 00:09:01 px01 kernel: [1150855.552169]  free:51089 free_pcp:42 free_cma:0
Feb  3 00:09:01 px01 kernel: [1150855.552172] Node 0 DMA free:15900kB min:8kB low:8kB high:12kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15984kB managed:15900kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
Feb  3 00:09:01 px01 kernel: [1150855.552175] lowmem_reserve[]: 0 3372 32032 32032 32032
Feb  3 00:09:01 px01 kernel: [1150855.552177] Node 0 DMA32 free:125004kB min:2408kB low:3008kB high:3612kB active_anon:1389020kB inactive_anon:466156kB active_file:552348kB inactive_file:553452kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3561248kB managed:3480360kB mlocked:0kB dirty:0kB writeback:75092kB mapped:10436kB shmem:14540kB slab_reclaimable:361792kB slab_unreclaimable:11568kB kernel_stack:848kB pagetables:5916kB unstable:146104kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:4 all_unreclaimable? no
Feb  3 00:09:01 px01 kernel: [1150855.552180] lowmem_reserve[]: 0 0 28660 28660 28660
Feb  3 00:09:01 px01 kernel: [1150855.552182] Node 0 Normal free:63452kB min:20488kB low:25608kB high:30732kB active_anon:13407708kB inactive_anon:1253904kB active_file:6831492kB inactive_file:6831596kB unevictable:3520kB isolated(anon):0kB isolated(file):0kB present:29874176kB managed:29347912kB mlocked:3520kB dirty:32kB writeback:926652kB mapped:89724kB shmem:118868kB slab_reclaimable:614460kB slab_unreclaimable:126216kB kernel_stack:4288kB pagetables:51312kB unstable:1740224kB bounce:0kB free_pcp:168kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
Feb  3 00:09:01 px01 kernel: [1150855.552185] lowmem_reserve[]: 0 0 0 0 0
Feb  3 00:09:01 px01 kernel: [1150855.552186] Node 0 DMA: 1*4kB (U) 1*8kB (U) 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15900kB
Feb  3 00:09:01 px01 kernel: [1150855.552194] Node 0 DMA32: 14462*4kB (UME) 8426*8kB (UME) 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 125256kB
Feb  3 00:09:01 px01 kernel: [1150855.552198] Node 0 Normal: 10219*4kB (UME) 2666*8kB (UME) 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 1*2048kB (H) 0*4096kB = 64252kB
Feb  3 00:09:01 px01 kernel: [1150855.552203] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Feb  3 00:09:01 px01 kernel: [1150855.552204] 3738272 total pagecache pages
Feb  3 00:09:01 px01 kernel: [1150855.552205] 12210 pages in swap cache
Feb  3 00:09:01 px01 kernel: [1150855.552205] Swap cache stats: add 65731, delete 53521, find 7226439/7235621
Feb  3 00:09:01 px01 kernel: [1150855.552206] Free swap  = 1912672kB
Feb  3 00:09:01 px01 kernel: [1150855.552207] Total swap = 2095096kB
Feb  3 00:09:01 px01 kernel: [1150855.552207] 8362852 pages RAM
Feb  3 00:09:01 px01 kernel: [1150855.552208] 0 pages HighMem/MovableOnly
Feb  3 00:09:01 px01 kernel: [1150855.552208] 151809 pages reserved
Feb  3 00:09:01 px01 kernel: [1150855.552209] 0 pages cma reserved
Feb  3 00:09:01 px01 kernel: [1150855.552209] 0 pages hwpoisoned
 
please upgrade to the current version, there was a kernel bug triggering OOM kills in non-OOM situations
 
Hi Fabian,
what is latest version for kernel bug triger OOM kills?I'm using proxmox-ve 4.4-77 (running kernel : 4.4.35-1-pve)..please help..
 
You must log in or register to reply here.
Top Bottom