Vms suddenly stopped

Daniel Zuwala · Feb 3, 2017

Hi,

I'm running a proxmox server for a few days where I have about 10 running vms. Everything went fine, until yesterday, when the backup was running (snapshot mode). It seems to me that the server run out of memory and killed 3 running vm. It would be ok, if the server lacks memory, but it has 32Gb of RAM and only 15Gb RAM are used for the running vms. Maybe I don't understand the error log and I would appreciate what you think it could be.

Thanks and sorry for the very long log file...

Code:

Feb  3 00:09:01 px01 kernel: [1150855.552119] sh cpuset=/ mems_allowed=0
Feb  3 00:09:01 px01 kernel: [1150855.552123] CPU: 2 PID: 22452 Comm: sh Tainted: G           O    4.4.35-1-pve #1
Feb  3 00:09:01 px01 kernel: [1150855.552124] Hardware name:                  /DH67BL, BIOS BLH6710H.86A.0160.2012.1204.1156 12/04/2012
Feb  3 00:09:01 px01 kernel: [1150855.552125]  0000000000000286 00000000fd771911 ffff880001dfbb50 ffffffff813f9743
Feb  3 00:09:01 px01 kernel: [1150855.552127]  ffff880001dfbd40 0000000000000000 ffff880001dfbbb8 ffffffff8120adcb
Feb  3 00:09:01 px01 kernel: [1150855.552128]  00000000fd771911 00000000ffffffff 0000000000000000 0000000000000000
Feb  3 00:09:01 px01 kernel: [1150855.552129] Call Trace:
Feb  3 00:09:01 px01 kernel: [1150855.552135]  [<ffffffff813f9743>] dump_stack+0x63/0x90
Feb  3 00:09:01 px01 kernel: [1150855.552138]  [<ffffffff8120adcb>] dump_header+0x67/0x1d5
Feb  3 00:09:01 px01 kernel: [1150855.552141]  [<ffffffff811925c5>] oom_kill_process+0x205/0x3c0
Feb  3 00:09:01 px01 kernel: [1150855.552142]  [<ffffffff81192a17>] out_of_memory+0x237/0x4a0
Feb  3 00:09:01 px01 kernel: [1150855.552145]  [<ffffffff81198d0e>] __alloc_pages_nodemask+0xcee/0xe20
Feb  3 00:09:01 px01 kernel: [1150855.552147]  [<ffffffff81198e8b>] alloc_kmem_pages_node+0x4b/0xd0
Feb  3 00:09:01 px01 kernel: [1150855.552150]  [<ffffffff8107f053>] copy_process+0x1c3/0x1c00
Feb  3 00:09:01 px01 kernel: [1150855.552153]  [<ffffffff813941b0>] ? apparmor_file_alloc_security+0x60/0x240
Feb  3 00:09:01 px01 kernel: [1150855.552156]  [<ffffffff813494b3>] ? security_file_alloc+0x33/0x50
Feb  3 00:09:01 px01 kernel: [1150855.552158]  [<ffffffff81080c20>] _do_fork+0x80/0x360
Feb  3 00:09:01 px01 kernel: [1150855.552161]  [<ffffffff810917ff>] ? sigprocmask+0x6f/0xa0
Feb  3 00:09:01 px01 kernel: [1150855.552163]  [<ffffffff81080fa9>] SyS_clone+0x19/0x20
Feb  3 00:09:01 px01 kernel: [1150855.552165]  [<ffffffff8185c276>] entry_SYSCALL_64_fastpath+0x16/0x75
Feb  3 00:09:01 px01 kernel: [1150855.552166] Mem-Info:
Feb  3 00:09:01 px01 kernel: [1150855.552169] active_anon:3699182 inactive_anon:430015 isolated_anon:0
Feb  3 00:09:01 px01 kernel: [1150855.552169]  active_file:1845960 inactive_file:1846262 isolated_file:0
Feb  3 00:09:01 px01 kernel: [1150855.552169]  unevictable:880 dirty:8 writeback:250436 unstable:471582
Feb  3 00:09:01 px01 kernel: [1150855.552169]  slab_reclaimable:244063 slab_unreclaimable:34446
Feb  3 00:09:01 px01 kernel: [1150855.552169]  mapped:25040 shmem:33352 pagetables:14307 bounce:0
Feb  3 00:09:01 px01 kernel: [1150855.552169]  free:51089 free_pcp:42 free_cma:0
Feb  3 00:09:01 px01 kernel: [1150855.552172] Node 0 DMA free:15900kB min:8kB low:8kB high:12kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15984kB managed:15900kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
Feb  3 00:09:01 px01 kernel: [1150855.552175] lowmem_reserve[]: 0 3372 32032 32032 32032
Feb  3 00:09:01 px01 kernel: [1150855.552177] Node 0 DMA32 free:125004kB min:2408kB low:3008kB high:3612kB active_anon:1389020kB inactive_anon:466156kB active_file:552348kB inactive_file:553452kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3561248kB managed:3480360kB mlocked:0kB dirty:0kB writeback:75092kB mapped:10436kB shmem:14540kB slab_reclaimable:361792kB slab_unreclaimable:11568kB kernel_stack:848kB pagetables:5916kB unstable:146104kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:4 all_unreclaimable? no
Feb  3 00:09:01 px01 kernel: [1150855.552180] lowmem_reserve[]: 0 0 28660 28660 28660
Feb  3 00:09:01 px01 kernel: [1150855.552182] Node 0 Normal free:63452kB min:20488kB low:25608kB high:30732kB active_anon:13407708kB inactive_anon:1253904kB active_file:6831492kB inactive_file:6831596kB unevictable:3520kB isolated(anon):0kB isolated(file):0kB present:29874176kB managed:29347912kB mlocked:3520kB dirty:32kB writeback:926652kB mapped:89724kB shmem:118868kB slab_reclaimable:614460kB slab_unreclaimable:126216kB kernel_stack:4288kB pagetables:51312kB unstable:1740224kB bounce:0kB free_pcp:168kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
Feb  3 00:09:01 px01 kernel: [1150855.552185] lowmem_reserve[]: 0 0 0 0 0
Feb  3 00:09:01 px01 kernel: [1150855.552186] Node 0 DMA: 1*4kB (U) 1*8kB (U) 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15900kB
Feb  3 00:09:01 px01 kernel: [1150855.552194] Node 0 DMA32: 14462*4kB (UME) 8426*8kB (UME) 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 125256kB
Feb  3 00:09:01 px01 kernel: [1150855.552198] Node 0 Normal: 10219*4kB (UME) 2666*8kB (UME) 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 1*2048kB (H) 0*4096kB = 64252kB
Feb  3 00:09:01 px01 kernel: [1150855.552203] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Feb  3 00:09:01 px01 kernel: [1150855.552204] 3738272 total pagecache pages
Feb  3 00:09:01 px01 kernel: [1150855.552205] 12210 pages in swap cache
Feb  3 00:09:01 px01 kernel: [1150855.552205] Swap cache stats: add 65731, delete 53521, find 7226439/7235621
Feb  3 00:09:01 px01 kernel: [1150855.552206] Free swap  = 1912672kB
Feb  3 00:09:01 px01 kernel: [1150855.552207] Total swap = 2095096kB
Feb  3 00:09:01 px01 kernel: [1150855.552207] 8362852 pages RAM
Feb  3 00:09:01 px01 kernel: [1150855.552208] 0 pages HighMem/MovableOnly
Feb  3 00:09:01 px01 kernel: [1150855.552208] 151809 pages reserved
Feb  3 00:09:01 px01 kernel: [1150855.552209] 0 pages cma reserved
Feb  3 00:09:01 px01 kernel: [1150855.552209] 0 pages hwpoisoned

fabian · Feb 3, 2017

please upgrade to the current version, there was a kernel bug triggering OOM kills in non-OOM situations

bawono · Sep 5, 2017

Hi Fabian,
what is latest version for kernel bug triger OOM kills?I'm using proxmox-ve 4.4-77 (running kernel : 4.4.35-1-pve)..please help..

tom · Sep 5, 2017

bawono said:
Hi Fabian,
what is latest version for kernel bug triger OOM kills?I'm using proxmox-ve 4.4-77 (running kernel : 4.4.35-1-pve)..please help..

Your kernel is affected and outdated, please upgrade.

See https://pve.proxmox.com/wiki/Downlo...Proxmox_Virtual_Environment_4.x_to_latest_4.4

bawono · Sep 5, 2017

tom said:
Your kernel is affected and outdated, please upgrade.

See https://pve.proxmox.com/wiki/Downlo...Proxmox_Virtual_Environment_4.x_to_latest_4.4

So need to subscription?upgrade to enterprise,right?

tom · Sep 5, 2017

bawono said:
So need to subscription?upgrade to enterprise,right?

not mandatory, see https://pve.proxmox.com/wiki/Package_Repositories

bawono · Sep 6, 2017

tom said:
not mandatory, see https://pve.proxmox.com/wiki/Package_Repositories

I will try,is there any issues after upgrade?

Search

Search

Vms suddenly stopped

Daniel Zuwala

New Member

fabian

Proxmox Staff Member

bawono

New Member

tom

Proxmox Staff Member

bawono

New Member

tom

Proxmox Staff Member

bawono

New Member