Self-Shutdowns - how to diagnose?

B

boomam

Member
Jan 5, 2023
60
10
13
Hi,
For some reason today, one of my notes shuts itself down after a few minutes - nothing in logs, nothing displayed in `journalctl -b` either.
System passes memory tests and is otherwise operational.

How do i diagnose this?

Last `journalctl -b` -
Code: 
Nov 25 17:12:27 node01 kernel: Linux version 6.8.12-3-pve (build@proxmox) (gcc (Debian 12.2.0-14) 12.2.0, GNU ld (GNU Binutils for Debian) 2.40) #1 S>
Nov 25 17:12:27 node01 kernel: Command line: BOOT_IMAGE=/boot/vmlinuz-6.8.12-3-pve root=/dev/mapper/pve-root ro quiet
Nov 25 17:12:27 node01 kernel: KERNEL supported cpus:
Nov 25 17:12:27 node01 kernel:   Intel GenuineIntel
Nov 25 17:12:27 node01 kernel:   AMD AuthenticAMD
Nov 25 17:12:27 node01 kernel:   Hygon HygonGenuine
Nov 25 17:12:27 node01 kernel:   Centaur CentaurHauls
Nov 25 17:12:27 node01 kernel:   zhaoxin   Shanghai
Nov 25 17:12:27 node01 kernel: BIOS-provided physical RAM map:
Nov 25 17:12:27 node01 kernel: BIOS-e820: [mem 0x0000000000000000-0x0000000000057fff] usable
Nov 25 17:12:27 node01 kernel: BIOS-e820: [mem 0x0000000000058000-0x0000000000058fff] reserved
Nov 25 17:12:27 node01 kernel: BIOS-e820: [mem 0x0000000000059000-0x000000000009dfff] usable
Nov 25 17:12:27 node01 kernel: BIOS-e820: [mem 0x000000000009e000-0x00000000000fffff] reserved
Nov 25 17:12:27 node01 kernel: BIOS-e820: [mem 0x0000000000100000-0x00000000b9c56fff] usable
Nov 25 17:12:27 node01 kernel: BIOS-e820: [mem 0x00000000b9c57000-0x00000000b9c57fff] reserved
Nov 25 17:12:27 node01 kernel: BIOS-e820: [mem 0x00000000b9c58000-0x00000000c90fafff] usable
Nov 25 17:12:27 node01 kernel: BIOS-e820: [mem 0x00000000c90fb000-0x00000000c95fafff] type 20
Nov 25 17:12:27 node01 kernel: BIOS-e820: [mem 0x00000000c95fb000-0x00000000c9c7efff] reserved
Nov 25 17:12:27 node01 kernel: BIOS-e820: [mem 0x00000000c9c7f000-0x00000000c9e7efff] ACPI NVS
Nov 25 17:12:27 node01 kernel: BIOS-e820: [mem 0x00000000c9e7f000-0x00000000c9efefff] ACPI data
Nov 25 17:12:27 node01 kernel: BIOS-e820: [mem 0x00000000c9eff000-0x00000000c9efffff] usable
Nov 25 17:12:27 node01 kernel: BIOS-e820: [mem 0x00000000c9f00000-0x00000000ce7fffff] reserved
Nov 25 17:12:27 node01 kernel: BIOS-e820: [mem 0x00000000f80fa000-0x00000000f80fafff] reserved
Nov 25 17:12:27 node01 kernel: BIOS-e820: [mem 0x00000000f80fd000-0x00000000f80fdfff] reserved
Nov 25 17:12:27 node01 kernel: BIOS-e820: [mem 0x00000000fe000000-0x00000000fe010fff] reserved
Nov 25 17:12:27 node01 kernel: BIOS-e820: [mem 0x0000000100000000-0x000000082f7fffff] usable
Nov 25 17:12:27 node01 kernel: NX (Execute Disable) protection: active
Nov 25 17:12:27 node01 kernel: APIC: Static calls initialized
Nov 25 17:12:27 node01 kernel: efi: EFI v2.4 by HP
Nov 25 17:12:27 node01 kernel: efi: ACPI=0xc9efe000 ACPI 2.0=0xc9efe014 SMBIOS=0xc98db000 ESRT=0xc98d8798
Nov 25 17:12:27 node01 kernel: efi: Not removing mem34: MMIO range=[0xf80fa000-0xf80fafff] (4KB) from e820 map
Nov 25 17:12:27 node01 kernel: efi: Not removing mem35: MMIO range=[0xf80fd000-0xf80fdfff] (4KB) from e820 map
Nov 25 17:12:27 node01 kernel: efi: Not removing mem36: MMIO range=[0xfe000000-0xfe010fff] (68KB) from e820 map
Nov 25 17:12:27 node01 kernel: secureboot: Secure boot disabled
Nov 25 17:12:27 node01 kernel: SMBIOS 2.7 present.
Nov 25 17:12:27 node01 kernel: DMI: HP HP ProDesk 600 G2 DM/8169, BIOS N22 Ver. 02.58 07/29/2022
Nov 25 17:12:27 node01 kernel: tsc: Detected 2500.000 MHz processor
Nov 25 17:12:27 node01 kernel: tsc: Detected 2499.952 MHz TSC
Nov 25 17:12:27 node01 kernel: e820: update [mem 0x00000000-0x00000fff] usable ==> reserved
Nov 25 17:12:27 node01 kernel: e820: remove [mem 0x000a0000-0x000fffff] usable
Nov 25 17:12:27 node01 kernel: last_pfn = 0x82f800 max_arch_pfn = 0x400000000
Nov 25 17:12:27 node01 kernel: MTRR map: 4 entries (3 fixed + 1 variable; max 23), built from 10 variable MTRRs
Nov 25 17:12:27 node01 kernel: x86/PAT: Configuration [0-7]: WB  WC  UC- UC  WB  WP  UC- WT
Nov 25 17:12:27 node01 kernel: last_pfn = 0xc9f00 max_arch_pfn = 0x400000000
Nov 25 17:12:27 node01 kernel: esrt: Reserving ESRT space from 0x00000000c98d8798 to 0x00000000c98d87d0.
Nov 25 17:12:27 node01 kernel: Using GB pages for direct mapping
Nov 25 17:12:27 node01 kernel: secureboot: Secure boot disabled
Nov 25 17:12:27 node01 kernel: RAMDISK: [mem 0x30b09000-0x3457bfff]
Nov 25 17:12:27 node01 kernel: ACPI: Early table checksum verification disabled
Nov 25 17:12:27 node01 kernel: ACPI: RSDP 0x00000000C9EFE014 000024 (v02 HPQOEM)
Nov 25 17:12:27 node01 kernel: ACPI: XSDT 0x00000000C9EFD0E8 0000B4 (v01 HPQOEM SLIC-BPC 00000000      01000013)
Nov 25 17:12:27 node01 kernel: ACPI: FACP 0x00000000C9EF1000 0000F4 (v05 HPQOEM SLIC-BPC 00000000 HP   00000001)
Nov 25 17:12:27 node01 kernel: ACPI: DSDT 0x00000000C9ECF000 01E369 (v02 HPQOEM 8169     00000000 INTL 20121018)
Nov 25 17:12:27 node01 kernel: ACPI: FACS 0x00000000C9E5B000 000040
Nov 25 17:12:27 node01 kernel: ACPI: SSDT 0x00000000C9EFC000 000108 (v02 HP     ShmTable 00000001 INTL 20121018)
Nov 25 17:12:27 node01 kernel: ACPI: TCPA 0x00000000C9EFB000 000032 (v02 HPQOEM EDK2     00000002      01000013)
Nov 25 17:12:27 node01 kernel: ACPI: SSDT 0x00000000C9EFA000 0003B8 (v02 HPQOEM TcgTable 00001000 INTL 20121018)
Nov 25 17:12:27 node01 kernel: ACPI: UEFI 0x00000000C9E6A000 000042 (v01 HPQOEM EDK2     00000002      01000013)
Nov 25 17:12:27 node01 kernel: ACPI: SSDT 0x00000000C9EF4000 0051FA (v02 SaSsdt SaSsdt   00003000 INTL 20121018)
lines 5-59
Broadcast message from root@node01 (Mon 2024-11-25 17:13:01 EST):

The system will power off at Mon 2024-11-25 17:14:01 EST!

Nov 25 17:12:27 node01 kernel: ACPI: SLIC 0x00000000C9EF3000 000176 (v01 HPQOEM SLIC-BPC 00000001 HP   00000001)
Nov 25 17:12:27 node01 kernel: ACPI: WSMT 0x00000000C9EF2000 000028 (v01 HPQOEM 8169     00000001 HP   00000001)
Nov 25 17:12:27 node01 kernel: ACPI: HPET 0x00000000C9EF0000 000038 (v01 HPQOEM 8169     00000001 HP   00000001)
Nov 25 17:12:27 node01 kernel: ACPI: APIC 0x00000000C9EEF000 0000BC (v01 HPQOEM 8169     00000001 HP   00000001)
Nov 25 17:12:27 node01 kernel: ACPI: MCFG 0x00000000C9EEE000 00003C (v01 HPQOEM 8169     00000001 HP   00000001)
Nov 25 17:12:27 node01 kernel: ACPI: SSDT 0x00000000C9ECE000 00019A (v02 HPQOEM Sata0Ide 00001000 INTL 20121018)
Nov 25 17:12:27 node01 kernel: ACPI: SSDT 0x00000000C9ECD000 000729 (v01 HPQOEM PtidDevc 00001000 INTL 20121018)
Nov 25 17:12:27 node01 kernel: ACPI: SSDT 0x00000000C9ECC000 000E73 (v02 CpuRef CpuSsdt  00003000 INTL 20121018)
Nov 25 17:12:27 node01 kernel: ACPI: DMAR 0x00000000C9ECB000 0000A8 (v01 INTEL  SKL      00000001 INTL 00000001)
Nov 25 17:12:27 node01 kernel: ACPI: ASF! 0x00000000C9ECA000 0000A5 (v32 HPQOEM  UYA     00000001 TFSM 000F4240)
Nov 25 17:12:27 node01 kernel: Command line: BOOT_IMAGE=/boot/vmlinuz-6.8.12-3-pve root=/dev/mapper/pve-root ro quiet
Nov 25 17:12:27 node01 kernel: KERNEL supported cpus:
Nov 25 17:12:27 node01 kernel:   Intel GenuineIntel
Nov 25 17:12:27 node01 kernel:   AMD AuthenticAMD
Nov 25 17:12:27 node01 kernel:   Hygon HygonGenuine
Nov 25 17:12:27 node01 kernel:   Centaur CentaurHauls
Nov 25 17:12:27 node01 kernel:   zhaoxin   Shanghai
Nov 25 17:12:27 node01 kernel: BIOS-provided physical RAM map:
Nov 25 17:12:27 node01 kernel: BIOS-e820: [mem 0x0000000000000000-0x0000000000057fff] usable
Nov 25 17:12:27 node01 kernel: BIOS-e820: [mem 0x0000000000058000-0x0000000000058fff] reserved
Nov 25 17:12:27 node01 kernel: BIOS-e820: [mem 0x0000000000059000-0x000000000009dfff] usable
Nov 25 17:12:27 node01 kernel: BIOS-e820: [mem 0x000000000009e000-0x00000000000fffff] reserved
Nov 25 17:12:27 node01 kernel: BIOS-e820: [mem 0x0000000000100000-0x00000000b9c56fff] usable
Nov 25 17:12:27 node01 kernel: BIOS-e820: [mem 0x00000000b9c57000-0x00000000b9c57fff] reserved
Nov 25 17:12:27 node01 kernel: BIOS-e820: [mem 0x00000000b9c58000-0x00000000c90fafff] usable
Nov 25 17:12:27 node01 kernel: BIOS-e820: [mem 0x00000000c90fb000-0x00000000c95fafff] type 20
Nov 25 17:12:27 node01 kernel: BIOS-e820: [mem 0x00000000c95fb000-0x00000000c9c7efff] reserved
Nov 25 17:12:27 node01 kernel: BIOS-e820: [mem 0x00000000c9c7f000-0x00000000c9e7efff] ACPI NVS
Nov 25 17:12:27 node01 kernel: BIOS-e820: [mem 0x00000000c9e7f000-0x00000000c9efefff] ACPI data
Nov 25 17:12:27 node01 kernel: BIOS-e820: [mem 0x00000000c9eff000-0x00000000c9efffff] usable
Nov 25 17:12:27 node01 kernel: BIOS-e820: [mem 0x00000000c9f00000-0x00000000ce7fffff] reserved
Nov 25 17:12:27 node01 kernel: BIOS-e820: [mem 0x00000000f80fa000-0x00000000f80fafff] reserved
Nov 25 17:12:27 node01 kernel: BIOS-e820: [mem 0x00000000f80fd000-0x00000000f80fdfff] reserved
Nov 25 17:12:27 node01 kernel: BIOS-e820: [mem 0x00000000fe000000-0x00000000fe010fff] reserved
Nov 25 17:12:27 node01 kernel: BIOS-e820: [mem 0x0000000100000000-0x000000082f7fffff] usable
Nov 25 17:12:27 node01 kernel: NX (Execute Disable) protection: active
Nov 25 17:12:27 node01 kernel: APIC: Static calls initialized
Nov 25 17:12:27 node01 kernel: efi: EFI v2.4 by HP
Nov 25 17:12:27 node01 kernel: efi: ACPI=0xc9efe000 ACPI 2.0=0xc9efe014 SMBIOS=0xc98db000 ESRT=0xc98d8798
Nov 25 17:12:27 node01 kernel: efi: Not removing mem34: MMIO range=[0xf80fa000-0xf80fafff] (4KB) from e820 map
Nov 25 17:12:27 node01 kernel: efi: Not removing mem35: MMIO range=[0xf80fd000-0xf80fdfff] (4KB) from e820 map
Nov 25 17:12:27 node01 kernel: efi: Not removing mem36: MMIO range=[0xfe000000-0xfe010fff] (68KB) from e820 map
Nov 25 17:12:27 node01 kernel: secureboot: Secure boot disabled
Nov 25 17:12:27 node01 kernel: SMBIOS 2.7 present.
Nov 25 17:12:27 node01 kernel: DMI: HP HP ProDesk 600 G2 DM/8169, BIOS N22 Ver. 02.58 07/29/2022
Nov 25 17:12:27 node01 kernel: tsc: Detected 2500.000 MHz processor
Nov 25 17:12:27 node01 kernel: tsc: Detected 2499.952 MHz TSC
Nov 25 17:12:27 node01 kernel: e820: update [mem 0x00000000-0x00000fff] usable ==> reserved
Nov 25 17:12:27 node01 kernel: e820: remove [mem 0x000a0000-0x000fffff] usable
Nov 25 17:12:27 node01 kernel: last_pfn = 0x82f800 max_arch_pfn = 0x400000000
Nov 25 17:12:27 node01 kernel: MTRR map: 4 entries (3 fixed + 1 variable; max 23), built from 10 variable MTRRs
Nov 25 17:12:27 node01 kernel: x86/PAT: Configuration [0-7]: WB  WC  UC- UC  WB  WP  UC- WT
Nov 25 17:12:27 node01 kernel: last_pfn = 0xc9f00 max_arch_pfn = 0x400000000
Nov 25 17:12:27 node01 kernel: esrt: Reserving ESRT space from 0x00000000c98d8798 to 0x00000000c98d87d0.
Nov 25 17:12:27 node01 kernel: Using GB pages for direct mapping
Nov 25 17:12:27 node01 kernel: secureboot: Secure boot disabled
Nov 25 17:12:27 node01 kernel: RAMDISK: [mem 0x30b09000-0x3457bfff]
Nov 25 17:12:27 node01 kernel: ACPI: Early table checksum verification disabled
Nov 25 17:12:27 node01 kernel: ACPI: RSDP 0x00000000C9EFE014 000024 (v02 HPQOEM)
Nov 25 17:12:27 node01 kernel: ACPI: XSDT 0x00000000C9EFD0E8 0000B4 (v01 HPQOEM SLIC-BPC 00000000      01000013)
Nov 25 17:12:27 node01 kernel: ACPI: FACP 0x00000000C9EF1000 0000F4 (v05 HPQOEM SLIC-BPC 00000000 HP   00000001)
Nov 25 17:12:27 node01 kernel: ACPI: DSDT 0x00000000C9ECF000 01E369 (v02 HPQOEM 8169     00000000 INTL 20121018)
Nov 25 17:12:27 node01 kernel: ACPI: FACS 0x00000000C9E5B000 000040
Nov 25 17:12:27 node01 kernel: ACPI: SSDT 0x00000000C9EFC000 000108 (v02 HP     ShmTable 00000001 INTL 20121018)
Nov 25 17:12:27 node01 kernel: ACPI: TCPA 0x00000000C9EFB000 000032 (v02 HPQOEM EDK2     00000002      01000013)
lines 2-56
Broadcast message from root@node01 (Mon 2024-11-25 17:14:01 EST):

The system will power off now!

Anyone any ideas?

Thanks in advance!
 
From a SSH session, if I'm able to catch the shutdown broadcast and run 'shutdown -c', the shutdown does not continue for a few mins, but then attempts again a short while later.
So its clearly something in software doing it - just no idea what.
 
ok, i think i've solved it -
Its caused by the UPS reporting as 'stale' by nut, which then just stays quite.
So I've implemented a script that once a minute that if it detects 'stale' in the output for upsc, will restart nut-server & nut-monitor.
Since doing so, the node is staying online.

If it stays that way for the next hour, i'll start moving services back across and call it a win. :-)
 
Last edited:
You must log in or register to reply here.
Top Bottom