Proxmox shuting down all VMs randomly

[BoomblasterDani]

Hi,
I've been hosting my Proxmox server for about one year now and it randomly shuts down my VMs about every week or two. Theres no real damage done, it's just anoying to have to start them all again every week or two. I'm not two experienced with reading logs, so I wanted to ask if someone could help me find out what causes these random shutdowns.

My last shutdown occurred about 11:50

Here's my Log:
Mar 05 11:38:00 play systemd[1]: Starting Proxmox VE replication runner... Mar 05 11:38:01 play systemd[1]: pvesr.service: Succeeded. Mar 05 11:38:01 play systemd[1]: Finished Proxmox VE replication runner. Mar 05 11:39:00 play systemd[1]: Starting Proxmox VE replication runner... Mar 05 11:39:01 play systemd[1]: pvesr.service: Succeeded. Mar 05 11:39:01 play systemd[1]: Finished Proxmox VE replication runner. Mar 05 11:39:01 play systemd[1]: pvesr.service: Consumed 1.056s CPU time. Mar 05 11:39:29 play pvedaemon[1265]: <root@pam> end task UPID:play:0000C218:00209921:6404C23A:vncproxy:100:root@pam: OK Mar 05 11:39:30 play pvedaemon[1265]: <root@pam> starting task UPID:play:0000CA1E:0021FA4E:6404C5C2:vncproxy:100:root@pam: Mar 05 11:39:30 play pvedaemon[51742]: starting vnc proxy UPID:play:0000CA1E:0021FA4E:6404C5C2:vncproxy:100:root@pam: Mar 05 11:40:00 play systemd[1]: Starting Proxmox VE replication runner... Mar 05 11:40:01 play systemd[1]: pvesr.service: Succeeded. Mar 05 11:40:01 play systemd[1]: Finished Proxmox VE replication runner. Mar 05 11:40:01 play systemd[1]: pvesr.service: Consumed 1.153s CPU time. Mar 05 11:41:00 play systemd[1]: Starting Proxmox VE replication runner... Mar 05 11:41:01 play systemd[1]: pvesr.service: Succeeded. Mar 05 11:41:01 play systemd[1]: Finished Proxmox VE replication runner. Mar 05 11:42:00 play systemd[1]: Starting Proxmox VE replication runner... Mar 05 11:42:01 play systemd[1]: pvesr.service: Succeeded. Mar 05 11:42:01 play systemd[1]: Finished Proxmox VE replication runner. Mar 05 11:42:01 play systemd[1]: pvesr.service: Consumed 1.068s CPU time. Mar 05 11:43:00 play systemd[1]: Starting Proxmox VE replication runner... Mar 05 11:43:01 play systemd[1]: pvesr.service: Succeeded. Mar 05 11:43:01 play systemd[1]: Finished Proxmox VE replication runner. Mar 05 11:43:01 play systemd[1]: pvesr.service: Consumed 1.200s CPU time. Mar 05 11:44:00 play systemd[1]: Starting Proxmox VE replication runner... Mar 05 11:44:01 play systemd[1]: pvesr.service: Succeeded. Mar 05 11:44:01 play systemd[1]: Finished Proxmox VE replication runner. Mar 05 11:45:00 play systemd[1]: Starting Proxmox VE replication runner... Mar 05 11:45:01 play systemd[1]: pvesr.service: Succeeded. Mar 05 11:45:01 play systemd[1]: Finished Proxmox VE replication runner. Mar 05 11:45:01 play systemd[1]: pvesr.service: Consumed 1.139s CPU time. Mar 05 11:46:00 play systemd[1]: Starting Proxmox VE replication runner... Mar 05 11:46:01 play systemd[1]: pvesr.service: Succeeded. Mar 05 11:46:01 play systemd[1]: Finished Proxmox VE replication runner. Mar 05 11:46:01 play systemd[1]: pvesr.service: Consumed 1.007s CPU time. Mar 05 11:46:32 play pveproxy[1273]: worker 1276 finished Mar 05 11:46:32 play pveproxy[1273]: starting 1 worker(s) Mar 05 11:46:32 play pveproxy[1273]: worker 52716 started Mar 05 11:46:33 play pveproxy[52715]: got inotify poll request in wrong process - disabling inotify Mar 05 11:47:00 play systemd[1]: Starting Proxmox VE replication runner... Mar 05 11:47:01 play systemd[1]: pvesr.service: Succeeded. Mar 05 11:47:01 play systemd[1]: Finished Proxmox VE replication runner. Mar 05 11:47:25 play pvedaemon[1265]: <root@pam> successful auth for user 'root@pam' Mar 05 11:48:00 play systemd[1]: Starting Proxmox VE replication runner... Mar 05 11:48:01 play systemd[1]: pvesr.service: Succeeded. Mar 05 11:48:01 play systemd[1]: Finished Proxmox VE replication runner. Mar 05 11:48:01 play systemd[1]: pvesr.service: Consumed 1.038s CPU time. Mar 05 11:49:00 play systemd[1]: Starting Proxmox VE replication runner... Mar 05 11:49:01 play systemd[1]: pvesr.service: Succeeded. Mar 05 11:49:01 play systemd[1]: Finished Proxmox VE replication runner. Mar 05 11:50:00 play systemd[1]: Starting Proxmox VE replication runner... Mar 05 11:50:01 play systemd[1]: pvesr.service: Succeeded. Mar 05 11:50:01 play systemd[1]: Finished Proxmox VE replication runner. Mar 05 11:50:01 play systemd[1]: pvesr.service: Consumed 1.048s CPU time. Mar 05 11:51:00 play systemd[1]: Starting Proxmox VE replication runner... Mar 05 11:51:01 play systemd[1]: pvesr.service: Succeeded. Mar 05 11:51:01 play systemd[1]: Finished Proxmox VE replication runner. -- Reboot -- Mar 05 11:55:59 play kernel: Linux version 5.11.22-1-pve (build@proxmox) (gcc (Debian 10.2.1-6) 10.2.1 20210110, GNU ld (GNU Binutils for Debian) 2.35.2) #1 SMP PVE 5.11.22-2 (Fri, 02 Jul 2021 16:22:45 +0200) () Mar 05 11:55:59 play kernel: Command line: BOOT_IMAGE=/boot/vmlinuz-5.11.22-1-pve root=/dev/mapper/pve-root ro quiet Mar 05 11:55:59 play kernel: KERNEL supported cpus: Mar 05 11:55:59 play kernel: Intel GenuineIntel Mar 05 11:55:59 play kernel: AMD AuthenticAMD Mar 05 11:55:59 play kernel: Hygon HygonGenuine Mar 05 11:55:59 play kernel: Centaur CentaurHauls Mar 05 11:55:59 play kernel: zhaoxin Shanghai Mar 05 11:55:59 play kernel: x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' Mar 05 11:55:59 play kernel: x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' Mar 05 11:55:59 play kernel: x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' Mar 05 11:55:59 play kernel: x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 Mar 05 11:55:59 play kernel: x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. Mar 05 11:55:59 play kernel: BIOS-provided physical RAM map: Mar 05 11:55:59 play kernel: BIOS-e820: [mem 0x0000000000000000-0x000000000009b3ff] usable Mar 05 11:55:59 play kernel: BIOS-e820: [mem 0x000000000009b400-0x000000000009bfff] reserved Mar 05 11:55:59 play kernel: BIOS-e820: [mem 0x000000000009e000-0x000000000009ffff] reserved Mar 05 11:55:59 play kernel: BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved Mar 05 11:55:59 play kernel: BIOS-e820: [mem 0x0000000000100000-0x0000000075dabfff] usable Mar 05 11:55:59 play kernel: BIOS-e820: [mem 0x0000000075dac000-0x0000000075dddfff] ACPI data Mar 05 11:55:59 play kernel: BIOS-e820: [mem 0x0000000075dde000-0x000000008fffffff] reserved Mar 05 11:55:59 play kernel: BIOS-e820: [mem 0x00000000fec00000-0x00000000fee0ffff] reserved Mar 05 11:55:59 play kernel: BIOS-e820: [mem 0x00000000ff800000-0x00000000ffffffff] reserved Mar 05 11:55:59 play kernel: BIOS-e820: [mem 0x0000000100000000-0x0000001c7fffefff] usable Mar 05 11:55:59 play kernel: NX (Execute Disable) protection: active Mar 05 11:55:59 play kernel: SMBIOS 2.8 present. Mar 05 11:55:59 play kernel: DMI: HP ProLiant ML350e Gen8, BIOS J02 11/12/2013 Mar 05 11:55:59 play kernel: tsc: Fast TSC calibration using PIT Mar 05 11:55:59 play kernel: tsc: Detected 2393.883 MHz processor Mar 05 11:55:59 play kernel: e820: update [mem 0x00000000-0x00000fff] usable ==> reserved Mar 05 11:55:59 play kernel: e820: remove [mem 0x000a0000-0x000fffff] usable Mar 05 11:55:59 play kernel: last_pfn = 0x1c7ffff max_arch_pfn = 0x400000000 Mar 05 11:55:59 play kernel: MTRR default type: write-back Mar 05 11:55:59 play kernel: MTRR fixed ranges enabled: Mar 05 11:55:59 play kernel: 00000-9FFFF write-back Mar 05 11:55:59 play kernel: A0000-BFFFF uncachable Mar 05 11:55:59 play kernel: C0000-FFFFF write-protect Mar 05 11:55:59 play kernel: MTRR variable ranges enabled: Mar 05 11:55:59 play kernel: 0 base 000080000000 mask 3FFF80000000 uncachable Mar 05 11:55:59 play kernel: 1 disabled Mar 05 11:55:59 play kernel: 2 disabled Mar 05 11:55:59 play kernel: 3 disabled Mar 05 11:55:59 play kernel: 4 disabled Mar 05 11:55:59 play kernel: 5 disabled Mar 05 11:55:59 play kernel: 6 disabled Mar 05 11:55:59 play kernel: 7 disabled Mar 05 11:55:59 play kernel: 8 disabled Mar 05 11:55:59 play kernel: 9 disabled Mar 05 11:55:59 play kernel: x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT Mar 05 11:55:59 play kernel: last_pfn = 0x75dac max_arch_pfn = 0x400000000 Mar 05 11:55:59 play kernel: found SMP MP-table at [mem 0x000f4f80-0x000f4f8f] Mar 05 11:55:59 play kernel: check: Scanning 1 areas for low memory corruption Mar 05 11:55:59 play kernel: Using GB pages for direct mapping Mar 05 11:55:59 play kernel: RAMDISK: [mem 0x310f5000-0x34871fff] Mar 05 11:55:59 play kernel: ACPI: Early table checksum verification disabled Mar 05 11:55:59 play kernel: ACPI: RSDP 0x00000000000F4F00 000024 (v02 HP ) Mar 05 11:55:59 play kernel: ACPI: XSDT 0x0000000075DAECC0 0000E4 (v01 HP ProLiant 00000002 Ò? 0000162E) Mar 05 11:55:59 play kernel: ACPI: FACP 0x0000000075DAEE00 0000F4 (v03 HP ProLiant 00000002 Ò? 0000162E) Mar 05 11:55:59 play kernel: ACPI BIOS Warning (bug): Invalid length for FADT/Pm1aControlBlock: 32, using default 16 (20201113/tbfadt-669) Mar 05 11:55:59 play kernel: ACPI BIOS Warning (bug): Invalid length for FADT/Pm2ControlBlock: 32, using default 8 (20201113/tbfadt-669) Mar 05 11:55:59 play kernel: ACPI: DSDT 0x0000000075DAEF00 0026DC (v01 HP DSDT 00000001 INTL 20030228) Mar 05 11:55:59 play kernel: ACPI: FACS 0x0000000075DAC140 000040 Mar 05 11:55:59 play kernel: ACPI: FACS 0x0000000075DAC140 000040 Mar 05 11:55:59 play kernel: ACPI: SPCR 0x0000000075DAC180 000050 (v01 HP SPCRRBSU 00000001 Ò? 0000162E) Mar 05 11:55:59 play kernel: ACPI: MCFG 0x0000000075DAC200 00003C (v01 HP ProLiant 00000001 00000000) Mar 05 11:55:59 play kernel: ACPI: HPET 0x0000000075DAC240 000038 (v01 HP ProLiant 00000002 Ò? 0000162E) Mar 05 11:55:59 play kernel: ACPI: FFFF 0x0000000075DAC280 000064 (v02 HP ProLiant 00000002 Ò? 0000162E) Mar 05 11:55:59 play kernel: ACPI: SPMI 0x0000000075DAC300 000040 (v05 HP ProLiant 00000001 Ò? 0000162E) Mar 05 11:55:59 play kernel: ACPI: ERST 0x0000000075DAC340 000230 (v01 HP ProLiant 00000001 Ò? 0000162E) Mar 05 11:55:59 play kernel: ACPI: APIC 0x0000000075DAC580 00026A (v01 HP ProLiant 00000002 00000000) Mar 05 11:55:59 play kernel: ACPI: SRAT 0x0000000075DAC800 000750 (v01 HP Proliant 00000001 Ò? 0000162E) Mar 05 11:55:59 play kernel: ACPI: FFFF 0x0000000075DACF80 000176 (v01 HP ProLiant 00000001 Ò? 0000162E) Mar 05 11:55:59 play kernel: ACPI: BERT 0x0000000075DAD100 000030 (v01 HP ProLiant 00000001 Ò? 0000162E) Mar 05 11:55:59 play kernel: ACPI: HEST 0x0000000075DAD140 0000BC (v01 HP ProLiant 00000001 Ò? 0000162E) Mar 05 11:55:59 play kernel: ACPI: DMAR 0x0000000075DAD200 000570 (v01 HP ProLiant 00000001 Ò? 0000162E) Mar 05 11:55:59 play kernel: ACPI: FFFF 0x0000000075DAEC00 000030 (v01 HP ProLiant 00000001 00000000) Mar 05 11:55:59 play kernel: ACPI: PCCT 0x0000000075DAEC40 00006E (v01 HP Proliant 00000001 PH 0000504D) Mar 05 11:55:59 play kernel: ACPI: SSDT 0x0000000075DB1600 0007EA (v01 HP DEV_PCI1 00000001 INTL 20120503) Mar 05 11:55:59 play kernel: ACPI: SSDT 0x0000000075DB1E00 000137 (v03 HP CRSPCI0 00000002 HP 00000001) Mar 05 11:55:59 play kernel: ACPI: SSDT 0x0000000075DB1F40 000098 (v03 HP CRSPCI1 00000002 HP 00000001) Mar 05 11:55:59 play kernel: ACPI: SSDT 0x0000000075DB2000 0007F5 (v03 HP dsmpci0 00000002 INTL 20030228) Mar 05 11:55:59 play kernel: ACPI: SSDT 0x0000000075DB2800 000386 (v03 HP dsmpci1 00000002 INTL 20030228) Mar 05 11:55:59 play kernel: ACPI: SSDT 0x0000000075DB2BC0 000BB9 (v01 HP pcc 00000001 INTL 20120503) Mar 05 11:55:59 play kernel: ACPI: SSDT 0x0000000075DB3780 000377 (v01 HP pmab 00000001 INTL 20120503) Mar 05 11:55:59 play kernel: ACPI: SSDT 0x0000000075DB3B00 005524 (v01 HP pcc2 00000001 INTL 20120503) Mar 05 11:55:59 play kernel: ACPI: SSDT 0x0000000075DB9040 003AEC (v01 INTEL PPM RCM 00000001 INTL 20061109) Mar 05 11:55:59 play kernel: ACPI: Reserving FACP table memory at [mem 0x75daee00-0x75daeef3] Mar 05 11:55:59 play kernel: ACPI: Reserving DSDT table memory at [mem 0x75daef00-0x75db15db] Mar 05 11:55:59 play kernel: ACPI: Reserving FACS table memory at [mem 0x75dac140-0x75dac17f] Mar 05 11:55:59 play kernel: ACPI: Reserving FACS table memory at [mem 0x75dac140-0x75dac17f] Mar 05 11:55:59 play kernel: ACPI: Reserving SPCR table memory at [mem 0x75dac180-0x75dac1cf] Mar 05 11:55:59 play kernel: ACPI: Reserving MCFG table memory at [mem 0x75dac200-0x75dac23b] Mar 05 11:55:59 play kernel: ACPI: Reserving HPET table memory at [mem 0x75dac240-0x75dac277] Mar 05 11:55:59 play kernel: ACPI: Reserving FFFF table memory at [mem 0x75dac280-0x75dac2e3] Mar 05 11:55:59 play kernel: ACPI: Reserving SPMI table memory at [mem 0x75dac300-0x75dac33f] Mar 05 11:55:59 play kernel: ACPI: Reserving ERST table memory at [mem 0x75dac340-0x75dac56f] Mar 05 11:55:59 play kernel: ACPI: Reserving APIC table memory at [mem 0x75dac580-0x75dac7e9] Mar 05 11:55:59 play kernel: ACPI: Reserving SRAT table memory at [mem 0x75dac800-0x75dacf4f] Mar 05 11:55:59 play kernel: ACPI: Reserving FFFF table memory at [mem 0x75dacf80-0x75dad0f5] Mar 05 11:55:59 play kernel: ACPI: Reserving BERT table memory at [mem 0x75dad100-0x75dad12f] Mar 05 11:55:59 play kernel: ACPI: Reserving HEST table memory at [mem 0x75dad140-0x75dad1fb] Mar 05 11:55:59 play kernel: ACPI: Reserving DMAR table memory at [mem 0x75dad200-0x75dad76f] Mar 05 11:55:59 play kernel: ACPI: Reserving FFFF table memory at [mem 0x75daec00-0x75daec2f] Mar 05 11:55:59 play kernel: ACPI: Reserving PCCT table memory at [mem 0x75daec40-0x75daecad] Mar 05 11:55:59 play kernel: ACPI: Reserving SSDT table memory at [mem 0x75db1600-0x75db1de9] Mar 05 11:55:59 play kernel: ACPI: Reserving SSDT table memory at [mem 0x75db1e00-0x75db1f36] Mar 05 11:55:59 play kernel: ACPI: Reserving SSDT table memory at [mem 0x75db1f40-0x75db1fd7] Mar 05 11:55:59 play kernel: ACPI: Reserving SSDT table memory at [mem 0x75db2000-0x75db27f4] Mar 05 11:55:59 play kernel: ACPI: Reserving SSDT table memory at [mem 0x75db2800-0x75db2b85] Mar 05 11:55:59 play kernel: ACPI: Reserving SSDT table memory at [mem 0x75db2bc0-0x75db3778] Mar 05 11:55:59 play kernel: ACPI: Reserving SSDT table memory at [mem 0x75db3780-0x75db3af6] Mar 05 11:55:59 play kernel: ACPI: Reserving SSDT table memory at [mem 0x75db3b00-0x75db9023] Mar 05 11:55:59 play kernel: ACPI: Reserving SSDT table memory at [mem 0x75db9040-0x75dbcb2b] Mar 05 11:55:59 play kernel: ACPI: Local APIC address 0xfee00000 Mar 05 11:55:59 play kernel: SRAT: PXM 0 -> APIC 0x00 -> Node 0 Mar 05 11:55:59 play kernel: SRAT: PXM 0 -> APIC 0x01 -> Node 0 Mar 05 11:55:59 play kernel: SRAT: PXM 0 -> APIC 0x02 -> Node 0
If there is anything else I can/should send you guys that could help, please just tell me.

 

[leesteken]

Your system appears to be resetting hard and unexpected. Check your hardware and your power grid, as I don't expect any Proxmox configuration to be the cause of this (otherwise Proxmox would have shut down the VMs gracefully). And you might want to update your Proxmox, after you find and correct the problem.