Hi,
I'm running a proxmox server for a few days where I have about 10 running vms. Everything went fine, until yesterday, when the backup was running (snapshot mode). It seems to me that the server run out of memory and killed 3 running vm. It would be ok, if the server lacks memory, but it has 32Gb of RAM and only 15Gb RAM are used for the running vms. Maybe I don't understand the error log and I would appreciate what you think it could be.
Thanks and sorry for the very long log file...
I'm running a proxmox server for a few days where I have about 10 running vms. Everything went fine, until yesterday, when the backup was running (snapshot mode). It seems to me that the server run out of memory and killed 3 running vm. It would be ok, if the server lacks memory, but it has 32Gb of RAM and only 15Gb RAM are used for the running vms. Maybe I don't understand the error log and I would appreciate what you think it could be.
Thanks and sorry for the very long log file...
Code:
Feb 3 00:09:01 px01 kernel: [1150855.552119] sh cpuset=/ mems_allowed=0
Feb 3 00:09:01 px01 kernel: [1150855.552123] CPU: 2 PID: 22452 Comm: sh Tainted: G O 4.4.35-1-pve #1
Feb 3 00:09:01 px01 kernel: [1150855.552124] Hardware name: /DH67BL, BIOS BLH6710H.86A.0160.2012.1204.1156 12/04/2012
Feb 3 00:09:01 px01 kernel: [1150855.552125] 0000000000000286 00000000fd771911 ffff880001dfbb50 ffffffff813f9743
Feb 3 00:09:01 px01 kernel: [1150855.552127] ffff880001dfbd40 0000000000000000 ffff880001dfbbb8 ffffffff8120adcb
Feb 3 00:09:01 px01 kernel: [1150855.552128] 00000000fd771911 00000000ffffffff 0000000000000000 0000000000000000
Feb 3 00:09:01 px01 kernel: [1150855.552129] Call Trace:
Feb 3 00:09:01 px01 kernel: [1150855.552135] [<ffffffff813f9743>] dump_stack+0x63/0x90
Feb 3 00:09:01 px01 kernel: [1150855.552138] [<ffffffff8120adcb>] dump_header+0x67/0x1d5
Feb 3 00:09:01 px01 kernel: [1150855.552141] [<ffffffff811925c5>] oom_kill_process+0x205/0x3c0
Feb 3 00:09:01 px01 kernel: [1150855.552142] [<ffffffff81192a17>] out_of_memory+0x237/0x4a0
Feb 3 00:09:01 px01 kernel: [1150855.552145] [<ffffffff81198d0e>] __alloc_pages_nodemask+0xcee/0xe20
Feb 3 00:09:01 px01 kernel: [1150855.552147] [<ffffffff81198e8b>] alloc_kmem_pages_node+0x4b/0xd0
Feb 3 00:09:01 px01 kernel: [1150855.552150] [<ffffffff8107f053>] copy_process+0x1c3/0x1c00
Feb 3 00:09:01 px01 kernel: [1150855.552153] [<ffffffff813941b0>] ? apparmor_file_alloc_security+0x60/0x240
Feb 3 00:09:01 px01 kernel: [1150855.552156] [<ffffffff813494b3>] ? security_file_alloc+0x33/0x50
Feb 3 00:09:01 px01 kernel: [1150855.552158] [<ffffffff81080c20>] _do_fork+0x80/0x360
Feb 3 00:09:01 px01 kernel: [1150855.552161] [<ffffffff810917ff>] ? sigprocmask+0x6f/0xa0
Feb 3 00:09:01 px01 kernel: [1150855.552163] [<ffffffff81080fa9>] SyS_clone+0x19/0x20
Feb 3 00:09:01 px01 kernel: [1150855.552165] [<ffffffff8185c276>] entry_SYSCALL_64_fastpath+0x16/0x75
Feb 3 00:09:01 px01 kernel: [1150855.552166] Mem-Info:
Feb 3 00:09:01 px01 kernel: [1150855.552169] active_anon:3699182 inactive_anon:430015 isolated_anon:0
Feb 3 00:09:01 px01 kernel: [1150855.552169] active_file:1845960 inactive_file:1846262 isolated_file:0
Feb 3 00:09:01 px01 kernel: [1150855.552169] unevictable:880 dirty:8 writeback:250436 unstable:471582
Feb 3 00:09:01 px01 kernel: [1150855.552169] slab_reclaimable:244063 slab_unreclaimable:34446
Feb 3 00:09:01 px01 kernel: [1150855.552169] mapped:25040 shmem:33352 pagetables:14307 bounce:0
Feb 3 00:09:01 px01 kernel: [1150855.552169] free:51089 free_pcp:42 free_cma:0
Feb 3 00:09:01 px01 kernel: [1150855.552172] Node 0 DMA free:15900kB min:8kB low:8kB high:12kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15984kB managed:15900kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
Feb 3 00:09:01 px01 kernel: [1150855.552175] lowmem_reserve[]: 0 3372 32032 32032 32032
Feb 3 00:09:01 px01 kernel: [1150855.552177] Node 0 DMA32 free:125004kB min:2408kB low:3008kB high:3612kB active_anon:1389020kB inactive_anon:466156kB active_file:552348kB inactive_file:553452kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3561248kB managed:3480360kB mlocked:0kB dirty:0kB writeback:75092kB mapped:10436kB shmem:14540kB slab_reclaimable:361792kB slab_unreclaimable:11568kB kernel_stack:848kB pagetables:5916kB unstable:146104kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:4 all_unreclaimable? no
Feb 3 00:09:01 px01 kernel: [1150855.552180] lowmem_reserve[]: 0 0 28660 28660 28660
Feb 3 00:09:01 px01 kernel: [1150855.552182] Node 0 Normal free:63452kB min:20488kB low:25608kB high:30732kB active_anon:13407708kB inactive_anon:1253904kB active_file:6831492kB inactive_file:6831596kB unevictable:3520kB isolated(anon):0kB isolated(file):0kB present:29874176kB managed:29347912kB mlocked:3520kB dirty:32kB writeback:926652kB mapped:89724kB shmem:118868kB slab_reclaimable:614460kB slab_unreclaimable:126216kB kernel_stack:4288kB pagetables:51312kB unstable:1740224kB bounce:0kB free_pcp:168kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
Feb 3 00:09:01 px01 kernel: [1150855.552185] lowmem_reserve[]: 0 0 0 0 0
Feb 3 00:09:01 px01 kernel: [1150855.552186] Node 0 DMA: 1*4kB (U) 1*8kB (U) 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15900kB
Feb 3 00:09:01 px01 kernel: [1150855.552194] Node 0 DMA32: 14462*4kB (UME) 8426*8kB (UME) 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 125256kB
Feb 3 00:09:01 px01 kernel: [1150855.552198] Node 0 Normal: 10219*4kB (UME) 2666*8kB (UME) 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 1*2048kB (H) 0*4096kB = 64252kB
Feb 3 00:09:01 px01 kernel: [1150855.552203] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Feb 3 00:09:01 px01 kernel: [1150855.552204] 3738272 total pagecache pages
Feb 3 00:09:01 px01 kernel: [1150855.552205] 12210 pages in swap cache
Feb 3 00:09:01 px01 kernel: [1150855.552205] Swap cache stats: add 65731, delete 53521, find 7226439/7235621
Feb 3 00:09:01 px01 kernel: [1150855.552206] Free swap = 1912672kB
Feb 3 00:09:01 px01 kernel: [1150855.552207] Total swap = 2095096kB
Feb 3 00:09:01 px01 kernel: [1150855.552207] 8362852 pages RAM
Feb 3 00:09:01 px01 kernel: [1150855.552208] 0 pages HighMem/MovableOnly
Feb 3 00:09:01 px01 kernel: [1150855.552208] 151809 pages reserved
Feb 3 00:09:01 px01 kernel: [1150855.552209] 0 pages cma reserved
Feb 3 00:09:01 px01 kernel: [1150855.552209] 0 pages hwpoisoned