VM automatically shutdown

I

irsathahamed

New Member
Feb 13, 2024
13
0
1
Dear All,

My VM automatically shutdown/crashed every 02 days. I dont know where have to check what is the issue is.

Can someone plz assist how can i find out the issue and resolve?
 
First I would check your system log for "OOM" messages in case PVE kills your VM because of running out of RAM.
 
Dunuin said:
First I would check your system log for "OOM" messages in case PVE kills your VM because of running out of RAM.
Click to expand...
Thank you for the quick reply.

I am very beginner in Proxmox, can you please guide me where will i check system log and how i will check.

Thank you.
 
Here is the syslog:-


Code: 
Mar 25 04:18:07 Fsfmerp postfix/qmgr[1048]: CF5F644102A: from=<root@Fsfmerp.local>, size=19017, nrcpt=1 (queue active)

Mar 25 04:18:38 Fsfmerp postfix/smtp[2192543]: connect to eom-ae.mail.protection.outlook.com[52.101.148.13]:25: Connection timed out

Mar 25 04:19:08 Fsfmerp postfix/smtp[2192543]: connect to eom-ae.mail.protection.outlook.com[52.101.148.3]:25: Connection timed out

Mar 25 04:19:38 Fsfmerp postfix/smtp[2192543]: connect to eom-ae.mail.protection.outlook.com[52.101.148.12]:25: Connection timed out

Mar 25 04:20:08 Fsfmerp postfix/smtp[2192543]: connect to eom-ae.mail.protection.outlook.com[52.101.148.1]:25: Connection timed out

Mar 25 04:20:08 Fsfmerp postfix/smtp[2192543]: CF5F644102A: to=<i.ahamed@eom.ae>, relay=none, delay=111912, delays=111792/0.2/120/0, dsn=4.4.1, status=deferred (connect to eom-ae.mail.protection.outlook.com[52.101.148.1]:25: Connection timed out)

Mar 25 04:22:01 Fsfmerp kernel: pveproxy worker invoked oom-killer: gfp_mask=0x140cca(GFP_HIGHUSER_MOVABLE|__GFP_COMP), order=0, oom_score_adj=0

Mar 25 04:22:01 Fsfmerp kernel: CPU: 2 PID: 2152036 Comm: pveproxy worker Tainted: P           O       6.5.11-8-pve #1

Mar 25 04:22:01 Fsfmerp kernel: Hardware name: HPE ProLiant DL380 Gen10/ProLiant DL380 Gen10, BIOS U30 08/18/2017

Mar 25 04:22:01 Fsfmerp kernel: Call Trace:

Mar 25 04:22:01 Fsfmerp kernel:  <TASK>

Mar 25 04:22:01 Fsfmerp kernel:  dump_stack_lvl+0x48/0x70

Mar 25 04:22:01 Fsfmerp kernel:  dump_stack+0x10/0x20

Mar 25 04:22:01 Fsfmerp kernel:  dump_header+0x4f/0x260

Mar 25 04:22:01 Fsfmerp kernel:  oom_kill_process+0x10d/0x1c0

Mar 25 04:22:01 Fsfmerp kernel:  out_of_memory+0x270/0x560

Mar 25 04:22:01 Fsfmerp kernel:  __alloc_pages+0x114f/0x12e0

Mar 25 04:22:01 Fsfmerp kernel:  alloc_pages+0x90/0x160

Mar 25 04:22:01 Fsfmerp kernel:  folio_alloc+0x1d/0x60

Mar 25 04:22:01 Fsfmerp kernel:  filemap_alloc_folio+0xfd/0x110

Mar 25 04:22:01 Fsfmerp kernel:  __filemap_get_folio+0xd8/0x230

Mar 25 04:22:01 Fsfmerp kernel:  filemap_fault+0x584/0x9f0

Mar 25 04:22:01 Fsfmerp kernel:  __do_fault+0x39/0x150

Mar 25 04:22:01 Fsfmerp kernel:  do_fault+0x266/0x3e0

Mar 25 04:22:01 Fsfmerp kernel:  __handle_mm_fault+0x6cc/0xc30

Mar 25 04:22:01 Fsfmerp kernel:  handle_mm_fault+0x164/0x360

Mar 25 04:22:01 Fsfmerp kernel:  do_user_addr_fault+0x212/0x6a0

Mar 25 04:22:01 Fsfmerp kernel:  exc_page_fault+0x83/0x1b0

Mar 25 04:22:01 Fsfmerp kernel:  asm_exc_page_fault+0x27/0x30

Mar 25 04:22:01 Fsfmerp kernel: RIP: 0033:0x55ba3ccdceb2

Mar 25 04:22:01 Fsfmerp kernel: Code: Unable to access opcode bytes at 0x55ba3ccdce88.

Mar 25 04:22:01 Fsfmerp kernel: RSP: 002b:00007ffdc04c6950 EFLAGS: 00010202

Mar 25 04:22:01 Fsfmerp kernel: RAX: 0000000000000004 RBX: 000000000000005e RCX: 0000000000000004

Mar 25 04:22:01 Fsfmerp kernel: RDX: 000000000000005f RSI: 0000000000000004 RDI: 000055ba3cec59c0

Mar 25 04:22:01 Fsfmerp kernel: RBP: 0000000000000000 R08: 000055ba4178bf78 R09: 0000000000000000

Mar 25 04:22:01 Fsfmerp kernel: R10: 0000000000000000 R11: 000055ba3cec58e0 R12: 0000000000000000

Mar 25 04:22:01 Fsfmerp kernel: R13: 0000000000000004 R14: 000055ba468a51c8 R15: 000055ba3e8f62a0

Mar 25 04:22:01 Fsfmerp kernel:  </TASK>

Mar 25 04:22:01 Fsfmerp kernel: Mem-Info:

Mar 25 04:22:01 Fsfmerp kernel: active_anon:9747318 inactive_anon:6042931 isolated_anon:0 active_file:0 inactive_file:67 isolated_file:0 unevictable:6354 dirty:10 writeback:0 slab_reclaimable:16329 slab_unreclaimable:332588 mapped:5986 shmem:4909 pagetables:42587 sec_pagetables:33527 bounce:0 kernel_misc_reclaimable:0 free:82401 free_pcp:128 free_cma:0

Mar 25 04:22:01 Fsfmerp kernel: Node 0 active_anon:38989272kB inactive_anon:24171724kB active_file:0kB inactive_file:268kB unevictable:25416kB isolated(anon):0kB isolated(file):0kB mapped:23944kB dirty:40kB writeback:0kB shmem:19636kB shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 13092864kB writeback_tmp:0kB kernel_stack:5600kB pagetables:170348kB sec_pagetables:134108kB all_unreclaimable? no

Mar 25 04:22:01 Fsfmerp kernel: Node 0 DMA free:11264kB boost:0kB min:12kB low:24kB high:36kB reserved_highatomic:0KB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:15988kB managed:15360kB mlocked:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB

Mar 25 04:22:01 Fsfmerp kernel: lowmem_reserve[]: 0 2595 63878 63878 63878

Mar 25 04:22:01 Fsfmerp kernel: Node 0 DMA32 free:247752kB boost:0kB min:2744kB low:5400kB high:8056kB reserved_highatomic:0KB active_anon:120348kB inactive_anon:2333684kB active_file:0kB inactive_file:472kB unevictable:0kB writepending:0kB present:2782404kB managed:2716668kB mlocked:0kB bounce:0kB free_pcp:508kB local_pcp:0kB free_cma:0kB

Mar 25 04:22:01 Fsfmerp kernel: lowmem_reserve[]: 0 0 61282 61282 61282

Mar 25 04:22:01 Fsfmerp kernel: Node 0 Normal free:70588kB boost:0kB min:64820kB low:127572kB high:190324kB reserved_highatomic:6144KB active_anon:38879472kB inactive_anon:21827492kB active_file:0kB inactive_file:1312kB unevictable:25416kB writepending:40kB present:63963132kB managed:62761932kB mlocked:25416kB bounce:0kB free_pcp:4kB local_pcp:0kB free_cma:0kB

Mar 25 04:22:01 Fsfmerp kernel: lowmem_reserve[]: 0 0 0 0 0

Mar 25 04:22:01 Fsfmerp kernel: Node 0 DMA: 0*4kB 0*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 1*1024kB (U) 1*2048kB (M) 2*4096kB (M) = 11264kB

Mar 25 04:22:01 Fsfmerp kernel: Node 0 DMA32: 3034*4kB (UME) 751*8kB (UME) 571*16kB (UME) 994*32kB (UME) 411*64kB (UME) 207*128kB (UME) 129*256kB (UME) 61*512kB (UME) 22*1024kB (UME) 24*2048kB (M) 0*4096kB = 247824kB

Mar 25 04:22:01 Fsfmerp kernel: Node 0 Normal: 473*4kB (UME) 348*8kB (UME) 482*16kB (UME) 271*32kB (UME) 143*64kB (UME) 89*128kB (UME) 98*256kB (UME) 13*512kB (UM) 0*1024kB 0*2048kB 0*4096kB = 73348kB

Mar 25 04:22:01 Fsfmerp kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB

Mar 25 04:22:01 Fsfmerp kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB

Mar 25 04:22:01 Fsfmerp kernel: 46807 total pagecache pages

Mar 25 04:22:01 Fsfmerp kernel: 39103 pages in swap cache

Mar 25 04:22:01 Fsfmerp kernel: Free swap  = 248kB

Mar 25 04:22:01 Fsfmerp kernel: Total swap = 8388604kB

Mar 25 04:22:01 Fsfmerp kernel: 16690381 pages RAM

Mar 25 04:22:01 Fsfmerp kernel: 0 pages HighMem/MovableOnly

Mar 25 04:22:01 Fsfmerp kernel: 316891 pages reserved

Mar 25 04:22:01 Fsfmerp kernel: 0 pages hwpoisoned

Mar 25 04:22:01 Fsfmerp kernel: Tasks state (memory values in pages):

Mar 25 04:22:01 Fsfmerp kernel: [  pid  ]   uid  tgid total_vm      rss pgtables_bytes swapents oom_score_adj name

Mar 25 04:22:01 Fsfmerp kernel: [    445]     0   445    15474      896   155648      256          -250 systemd-journal

Mar 25 04:22:01 Fsfmerp kernel: [    467]     0   467    20145     6280    90112        0         -1000 dmeventd

Mar 25 04:22:01 Fsfmerp kernel: [    471]     0   471     6711      830    77824      160         -1000 systemd-udevd

Mar 25 04:22:01 Fsfmerp kernel: [    756]   103   756     1969      800    57344       32             0 rpcbind

Mar 25 04:22:01 Fsfmerp kernel: [    762]   101   762     2326     1024    65536       64          -900 dbus-daemon

Mar 25 04:22:01 Fsfmerp kernel: [    765]     0   765    38186      448    61440       32         -1000 lxcfs

Mar 25 04:22:01 Fsfmerp kernel: [    766]     0   766    69539      768    77824       64             0 pve-lxc-syscall

Mar 25 04:22:01 Fsfmerp kernel: [    768]     0   768     2797      864    65536      160             0 smartd

Mar 25 04:22:01 Fsfmerp kernel: [    769]     0   769     6338      864    73728      224             0 systemd-logind

Mar 25 04:22:01 Fsfmerp kernel: [    770]     0   770      583      384    40960        0         -1000 watchdog-mux

Mar 25 04:22:01 Fsfmerp kernel: [    774]     0   774    41731      800    81920      256             0 zed

Mar 25 04:22:01 Fsfmerp kernel: [    780]     0   780     1328      480    49152        0             0 qmeventd

Mar 25 04:22:01 Fsfmerp kernel: [    785]     0   785     1766      307    53248       32             0 ksmtuned

Mar 25 04:22:01 Fsfmerp kernel: [    861]     0   861     1256      384    45056       32             0 lxc-monitord

Mar 25 04:22:01 Fsfmerp kernel: [    873]     0   873     1468      416    53248       32             0 agetty

Mar 25 04:22:01 Fsfmerp kernel: [    886]     0   886     3853      960    73728      224         -1000 sshd

Mar 25 04:22:01 Fsfmerp kernel: [    923]   100   923     4715      641    61440       96             0 chronyd

Mar 25 04:22:01 Fsfmerp kernel: [    940]   100   940     2633      516    61440       96             0 chronyd

Mar 25 04:22:01 Fsfmerp kernel: [   1046]     0  1046    10664      581    73728       96             0 master

Mar 25 04:22:01 Fsfmerp kernel: [   1048]   104  1048    10809      576    73728      128             0 qmgr

Mar 25 04:22:01 Fsfmerp kernel: [   1050]     0  1050   233042      478   188416      320             0 rrdcached

Mar 25 04:22:01 Fsfmerp kernel: [   1058]     0  1058   169116     4449   401408     2383             0 pmxcfs

Mar 25 04:22:01 Fsfmerp kernel: [   1064]     0  1064     1652      480    49152       32             0 cron

Mar 25 04:22:01 Fsfmerp kernel: [   1072]     0  1072    39231     6637   299008    17664             0 pve-firewall

Mar 25 04:22:01 Fsfmerp kernel: [   1076]     0  1076    37985     5627   307200    17152             0 pvestatd

Mar 25 04:22:01 Fsfmerp kernel: [   1077]     0  1077      615      288    40960        0             0 bpfilter_umh

Mar 25 04:22:01 Fsfmerp kernel: [   1102]     0  1102    58213      775   389120    33152             0 pvedaemon

Mar 25 04:22:01 Fsfmerp kernel: [   1115]     0  1115    54742     1233   356352    26400             0 pve-ha-crm

Mar 25 04:22:01 Fsfmerp kernel: [   1126]    33  1126    58565     4453   454656    30336             0 pveproxy

Mar 25 04:22:01 Fsfmerp kernel: [   1132]    33  1132    20204     4704   196608     8736             0 spiceproxy

Mar 25 04:22:01 Fsfmerp kernel: [   1134]     0  1134    54613     4460   348160    23104             0 pve-ha-lrm

Mar 25 04:22:01 Fsfmerp kernel: [   1157]     0  1157    53541     1594   344064    26432             0 pvescheduler
 
milew said:
How much memory does have host? and show your vm-s configuration
Click to expand...
Thank you for the reply.

Here is the Node memoery details:-

1711383436948.png

I am running 2 VM and the configuration below:-

VM 100:-

1711383481024.png


VM 101:-

1711383516130.png


VM 101 is the one who crashes frequently.
 
84GB RAM assigned to VMs while your host has only 64GB is a bad idea. You‘ll have to lower RAM assignment to both VMs, for example 24GB on each VM. If you use ZFS additional 10% of your RAM will be assigned for caching.
 
Last edited:
cwt said:
If you use ZFS additional 10% of your RAM will be assigned for caching
Click to expand...
X to 50% by default.

And yes. You are massivly overprovisioning your RAM so no wonder if PVE kills your VM when running out of memory as it will usually kill the process first that consumes the most RAM.
 
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom