Troubleshooting random reboots/subsequent failure to start

A

acidxxxburn

New Member
Feb 13, 2025
2
0
1
Hello! I've been running my Proxmox VE 8.3.3 instance for a few months now, and for the past few weeks I keep noticing random reboots. The only consistency I can really pinpoint is that it happens in the early hours of the morning [00:00 - 07:00], but does not seem to be consistent on the day of the week/month [so I believe it isn't the result of any Cron jobs]
Hardware: HP ProDesk G3 600 w/ 64GB of RAM, using an Intel i5-6500
Kernel: vmlinuz-6.8.12-8-pve

The symptom that alerted me this morning was all of my lights were on, which happens when my Home Assistant VM goes offline. Sure enough, I walked over to my Proxmox machine and noticed it was powered off.

Checking journalctl for the time between 00:00 and 07:00, I observed the following:
Code: 
root@PVE-PRD2:~# journalctl --since "2025-02-13 00:00:00" --until "2025-02-14 00:07:00"
Feb 13 00:00:08 PVE-PRD2 pvedaemon[787950]: worker exit
Feb 13 00:00:09 PVE-PRD2 pvedaemon[1134]: worker 787950 finished
Feb 13 00:00:09 PVE-PRD2 pvedaemon[1134]: starting 1 worker(s)
Feb 13 00:00:09 PVE-PRD2 pvedaemon[1134]: worker 804067 started
Feb 13 00:00:41 PVE-PRD2 smartd[777]: Device: /dev/sda [SAT], SMART Usage Attribute: 194 Temperature_Celsius changed from 69 to 71
Feb 13 00:00:41 PVE-PRD2 systemd[1]: Starting dpkg-db-backup.service - Daily dpkg database backup service...
Feb 13 00:00:41 PVE-PRD2 systemd[1]: Starting logrotate.service - Rotate log files...
Feb 13 00:00:41 PVE-PRD2 systemd[1]: dpkg-db-backup.service: Deactivated successfully.
Feb 13 00:00:41 PVE-PRD2 systemd[1]: Finished dpkg-db-backup.service - Daily dpkg database backup service.
Feb 13 00:00:41 PVE-PRD2 systemd[1]: Reloading pveproxy.service - PVE API Proxy Server...
Feb 13 00:00:42 PVE-PRD2 pveproxy[804192]: send HUP to 1143
Feb 13 00:00:42 PVE-PRD2 pveproxy[1143]: received signal HUP
Feb 13 00:00:42 PVE-PRD2 pveproxy[1143]: server closing
Feb 13 00:00:42 PVE-PRD2 pveproxy[1143]: server shutdown (restart)
Feb 13 00:00:42 PVE-PRD2 systemd[1]: Reloaded pveproxy.service - PVE API Proxy Server.
Feb 13 00:00:42 PVE-PRD2 systemd[1]: Reloading spiceproxy.service - PVE SPICE Proxy Server...
Feb 13 00:00:42 PVE-PRD2 spiceproxy[804194]: send HUP to 1149
Feb 13 00:00:42 PVE-PRD2 spiceproxy[1149]: received signal HUP
Feb 13 00:00:42 PVE-PRD2 spiceproxy[1149]: server closing
Feb 13 00:00:42 PVE-PRD2 spiceproxy[1149]: server shutdown (restart)
Feb 13 00:00:42 PVE-PRD2 systemd[1]: Reloaded spiceproxy.service - PVE SPICE Proxy Server.
Feb 13 00:00:42 PVE-PRD2 systemd[1]: Stopping pvefw-logger.service - Proxmox VE firewall logger...
Feb 13 00:00:42 PVE-PRD2 pvefw-logger[507953]: received terminate request (signal)
Feb 13 00:00:42 PVE-PRD2 pvefw-logger[507953]: stopping pvefw logger
Feb 13 00:00:43 PVE-PRD2 systemd[1]: pvefw-logger.service: Deactivated successfully.
Feb 13 00:00:43 PVE-PRD2 systemd[1]: Stopped pvefw-logger.service - Proxmox VE firewall logger.
Feb 13 00:00:43 PVE-PRD2 systemd[1]: pvefw-logger.service: Consumed 10.830s CPU time.
Feb 13 00:00:43 PVE-PRD2 systemd[1]: Starting pvefw-logger.service - Proxmox VE firewall logger...
Feb 13 00:00:43 PVE-PRD2 pvefw-logger[804205]: starting pvefw logger
Feb 13 00:00:43 PVE-PRD2 systemd[1]: Started pvefw-logger.service - Proxmox VE firewall logger.
Feb 13 00:00:43 PVE-PRD2 systemd[1]: logrotate.service: Deactivated successfully.
Feb 13 00:00:43 PVE-PRD2 systemd[1]: logrotate.service: Deactivated successfully.
Feb 13 00:00:43 PVE-PRD2 systemd[1]: Finished logrotate.service - Rotate log files.
Feb 13 00:00:43 PVE-PRD2 spiceproxy[1149]: restarting server
Feb 13 00:00:43 PVE-PRD2 spiceproxy[1149]: starting 1 worker(s)
Feb 13 00:00:43 PVE-PRD2 spiceproxy[1149]: worker 804209 started
Feb 13 00:00:43 PVE-PRD2 pveproxy[1143]: restarting server
Feb 13 00:00:43 PVE-PRD2 pveproxy[1143]: starting 3 worker(s)
Feb 13 00:00:43 PVE-PRD2 pveproxy[1143]: worker 804212 started
Feb 13 00:00:43 PVE-PRD2 pveproxy[1143]: worker 804213 started
Feb 13 00:00:43 PVE-PRD2 pveproxy[1143]: worker 804214 started
Feb 13 00:00:48 PVE-PRD2 spiceproxy[507957]: worker exit
Feb 13 00:00:48 PVE-PRD2 spiceproxy[1149]: worker 507957 finished
Feb 13 00:00:48 PVE-PRD2 pveproxy[801483]: worker exit
Feb 13 00:00:48 PVE-PRD2 pveproxy[802030]: worker exit
Feb 13 00:00:48 PVE-PRD2 pveproxy[801051]: worker exit
Feb 13 00:00:48 PVE-PRD2 pveproxy[1143]: worker 802030 finished
Feb 13 00:00:48 PVE-PRD2 pveproxy[1143]: worker 801483 finished
Feb 13 00:00:48 PVE-PRD2 pveproxy[1143]: worker 801051 finished
Feb 13 00:09:22 PVE-PRD2 pvedaemon[790105]: <root@pam> successful auth for user 'root@pam'
Feb 13 00:09:39 PVE-PRD2 pvedaemon[790352]: worker exit
Feb 13 00:09:39 PVE-PRD2 pvedaemon[1134]: worker 790352 finished
Feb 13 00:09:39 PVE-PRD2 pvedaemon[1134]: starting 1 worker(s)
Feb 13 00:09:39 PVE-PRD2 pvedaemon[1134]: worker 806032 started
Feb 13 00:16:57 PVE-PRD2 pvedaemon[790105]: worker exit
Feb 13 00:16:57 PVE-PRD2 pvedaemon[1134]: worker 790105 finished
Feb 13 00:16:57 PVE-PRD2 pvedaemon[1134]: starting 1 worker(s)
Feb 13 00:16:57 PVE-PRD2 pvedaemon[1134]: worker 807488 started
Feb 13 00:17:01 PVE-PRD2 CRON[807519]: pam_unix(cron:session): session opened for user root(uid=0) by (uid=0)
Feb 13 00:17:01 PVE-PRD2 CRON[807520]: (root) CMD (cd / && run-parts --report /etc/cron.hourly)
Feb 13 00:17:01 PVE-PRD2 CRON[807519]: pam_unix(cron:session): session closed for user root
Feb 13 00:24:01 PVE-PRD2 CRON[808986]: pam_unix(cron:session): session opened for user root(uid=0) by (uid=0)
Feb 13 00:24:01 PVE-PRD2 CRON[808987]: (root) CMD (if [ $(date +%w) -eq 0 ] && [ -x /usr/lib/zfs-linux/scrub ]; then /usr/lib/zfs->
Feb 13 00:24:01 PVE-PRD2 CRON[808986]: pam_unix(cron:session): session closed for user root
Feb 13 00:25:22 PVE-PRD2 pvedaemon[806032]: <root@pam> successful auth for user 'root@pam'
Feb 13 00:30:41 PVE-PRD2 smartd[777]: Device: /dev/sda [SAT], SMART Usage Attribute: 194 Temperature_Celsius changed from 71 to 69
Feb 13 00:40:22 PVE-PRD2 pvedaemon[806032]: <root@pam> successful auth for user 'root@pam'
Feb 13 00:41:19 PVE-PRD2 pveproxy[804213]: worker exit
Feb 13 00:41:19 PVE-PRD2 pveproxy[1143]: worker 804213 finished
Feb 13 00:41:19 PVE-PRD2 pveproxy[1143]: starting 1 worker(s)
Feb 13 00:41:19 PVE-PRD2 pveproxy[1143]: worker 812495 started
Feb 13 00:46:37 PVE-PRD2 pveproxy[804214]: worker exit
Feb 13 00:46:37 PVE-PRD2 pveproxy[1143]: worker 804214 finished
Feb 13 00:46:37 PVE-PRD2 pveproxy[1143]: starting 1 worker(s)
Feb 13 00:46:37 PVE-PRD2 pveproxy[1143]: worker 813544 started
Feb 13 00:47:23 PVE-PRD2 pveproxy[804212]: worker exit
Feb 13 00:47:23 PVE-PRD2 pveproxy[1143]: worker 804212 finished
Feb 13 00:47:23 PVE-PRD2 pveproxy[1143]: starting 1 worker(s)
Feb 13 00:47:23 PVE-PRD2 pveproxy[1143]: worker 813711 started
Feb 13 00:55:22 PVE-PRD2 pvedaemon[806032]: <root@pam> successful auth for user 'root@pam'
Feb 13 01:11:22 PVE-PRD2 pvedaemon[807488]: <root@pam> successful auth for user 'root@pam'
Feb 13 01:17:01 PVE-PRD2 CRON[819705]: pam_unix(cron:session): session opened for user root(uid=0) by (uid=0)
Feb 13 01:17:01 PVE-PRD2 CRON[819706]: (root) CMD (cd / && run-parts --report /etc/cron.hourly)
Feb 13 01:17:01 PVE-PRD2 CRON[819705]: pam_unix(cron:session): session closed for user root
Feb 13 01:19:44 PVE-PRD2 pvedaemon[806032]: worker exit
Feb 13 01:19:44 PVE-PRD2 pvedaemon[1134]: worker 806032 finished
Feb 13 01:19:44 PVE-PRD2 pvedaemon[1134]: starting 1 worker(s)
Feb 13 01:19:44 PVE-PRD2 pvedaemon[1134]: worker 820241 started
Feb 13 01:21:48 PVE-PRD2 pvedaemon[804067]: worker exit
Feb 13 01:21:48 PVE-PRD2 pvedaemon[1134]: worker 804067 finished
Feb 13 01:21:48 PVE-PRD2 pvedaemon[1134]: starting 1 worker(s)
Feb 13 01:21:48 PVE-PRD2 pvedaemon[1134]: worker 820649 started
-- Boot 8bff7b949bb94a61aada969ebd0a0059 --
Feb 13 05:01:43 PVE-PRD2 kernel: Linux version 6.8.12-8-pve (build@proxmox) (gcc (Debian 12.2.0-14) 12.2.0, GNU ld (GNU Binutils f>
Feb 13 05:01:43 PVE-PRD2 kernel: Command line: BOOT_IMAGE=/boot/vmlinuz-6.8.12-8-pve root=/dev/mapper/pve-root ro quiet intel_iomm>
Feb 13 05:01:43 PVE-PRD2 kernel: KERNEL supported cpus:
Feb 13 05:01:43 PVE-PRD2 kernel:   Intel GenuineIntel
Feb 13 05:01:43 PVE-PRD2 kernel:   AMD AuthenticAMD
Feb 13 05:01:43 PVE-PRD2 kernel:   Hygon HygonGenuine
Feb 13 05:01:43 PVE-PRD2 kernel:   Centaur CentaurHauls
Feb 13 05:01:43 PVE-PRD2 kernel:   zhaoxin   Shanghai 
Feb 13 05:01:43 PVE-PRD2 kernel: BIOS-provided physical RAM map:
Feb 13 00:41:19 PVE-PRD2 pveproxy[804213]: worker exit
Feb 13 00:41:19 PVE-PRD2 pveproxy[1143]: worker 804213 finished
Feb 13 00:41:19 PVE-PRD2 pveproxy[1143]: starting 1 worker(s)
Feb 13 00:41:19 PVE-PRD2 pveproxy[1143]: worker 812495 started
Feb 13 00:46:37 PVE-PRD2 pveproxy[804214]: worker exit
Feb 13 00:46:37 PVE-PRD2 pveproxy[1143]: worker 804214 finished
Feb 13 00:46:37 PVE-PRD2 pveproxy[1143]: starting 1 worker(s)
Feb 13 00:46:37 PVE-PRD2 pveproxy[1143]: worker 813544 started
Feb 13 00:47:23 PVE-PRD2 pveproxy[804212]: worker exit
Feb 13 00:47:23 PVE-PRD2 pveproxy[1143]: worker 804212 finished
Feb 13 00:47:23 PVE-PRD2 pveproxy[1143]: starting 1 worker(s)
Feb 13 00:47:23 PVE-PRD2 pveproxy[1143]: worker 813711 started
Feb 13 00:55:22 PVE-PRD2 pvedaemon[806032]: <root@pam> successful auth for user 'root@pam'
Feb 13 01:11:22 PVE-PRD2 pvedaemon[807488]: <root@pam> successful auth for user 'root@pam'
Feb 13 01:17:01 PVE-PRD2 CRON[819705]: pam_unix(cron:session): session opened for user root(uid=0) by (uid=0)
Feb 13 01:17:01 PVE-PRD2 CRON[819706]: (root) CMD (cd / && run-parts --report /etc/cron.hourly)
Feb 13 01:17:01 PVE-PRD2 CRON[819705]: pam_unix(cron:session): session closed for user root
Feb 13 01:19:44 PVE-PRD2 pvedaemon[806032]: worker exit
Feb 13 01:19:44 PVE-PRD2 pvedaemon[1134]: worker 806032 finished
Feb 13 01:19:44 PVE-PRD2 pvedaemon[1134]: starting 1 worker(s)
Feb 13 01:19:44 PVE-PRD2 pvedaemon[1134]: worker 820241 started
Feb 13 01:21:48 PVE-PRD2 pvedaemon[804067]: worker exit
Feb 13 01:21:48 PVE-PRD2 pvedaemon[1134]: worker 804067 finished
Feb 13 01:21:48 PVE-PRD2 pvedaemon[1134]: starting 1 worker(s)
Feb 13 01:21:48 PVE-PRD2 pvedaemon[1134]: worker 820649 started
-- Boot 8bff7b949bb94a61aada969ebd0a0059 --
Feb 13 05:01:43 PVE-PRD2 kernel: Linux version 6.8.12-8-pve (build@proxmox) (gcc (Debian 12.2.0-14) 12.2.0, GNU ld (GNU Binutils f>
Feb 13 05:01:43 PVE-PRD2 kernel: Command line: BOOT_IMAGE=/boot/vmlinuz-6.8.12-8-pve root=/dev/mapper/pve-root ro quiet intel_iomm>
Feb 13 05:01:43 PVE-PRD2 kernel: KERNEL supported cpus:
Feb 13 05:01:43 PVE-PRD2 kernel:   Intel GenuineIntel
Feb 13 05:01:43 PVE-PRD2 kernel:   AMD AuthenticAMD
Feb 13 05:01:43 PVE-PRD2 kernel:   Hygon HygonGenuine
Feb 13 05:01:43 PVE-PRD2 kernel:   Centaur CentaurHauls
Feb 13 05:01:43 PVE-PRD2 kernel:   zhaoxin   Shanghai 
Feb 13 05:01:43 PVE-PRD2 kernel: BIOS-provided physical RAM map:
Feb 13 05:01:43 PVE-PRD2 kernel: BIOS-e820: [mem 0x0000000000000000-0x0000000000057fff] usable
Feb 13 05:01:43 PVE-PRD2 kernel: BIOS-e820: [mem 0x0000000000058000-0x0000000000058fff] reserved
Feb 13 05:01:43 PVE-PRD2 kernel: BIOS-e820: [mem 0x0000000000059000-0x000000000009dfff] usable
Feb 13 05:01:43 PVE-PRD2 kernel: BIOS-e820: [mem 0x000000000009e000-0x000000000009efff] reserved
Feb 13 05:01:43 PVE-PRD2 kernel: BIOS-e820: [mem 0x000000000009f000-0x000000000009ffff] usable
Feb 13 05:01:43 PVE-PRD2 kernel: BIOS-e820: [mem 0x00000000000a0000-0x00000000000fffff] reserved
Feb 13 05:01:43 PVE-PRD2 kernel: BIOS-e820: [mem 0x0000000000100000-0x00000000a80dcfff] usable
Feb 13 05:01:43 PVE-PRD2 kernel: BIOS-e820: [mem 0x00000000a80dd000-0x00000000a80ddfff] ACPI NVS
Feb 13 05:01:43 PVE-PRD2 kernel: BIOS-e820: [mem 0x00000000a80de000-0x00000000a80defff] reserved
Feb 13 05:01:43 PVE-PRD2 kernel: BIOS-e820: [mem 0x00000000a80df000-0x00000000b8f9afff] usable
Feb 13 05:01:43 PVE-PRD2 kernel: BIOS-e820: [mem 0x00000000b8f9b000-0x00000000b949afff] type 20
Feb 13 05:01:43 PVE-PRD2 kernel: BIOS-e820: [mem 0x00000000b949b000-0x00000000b9d7efff] reserved
Feb 13 05:01:43 PVE-PRD2 kernel: BIOS-e820: [mem 0x00000000b9d7f000-0x00000000b9f7efff] ACPI NVS
Feb 13 05:01:43 PVE-PRD2 kernel: BIOS-e820: [mem 0x00000000b9f7f000-0x00000000b9ffefff] ACPI data
Feb 13 05:01:43 PVE-PRD2 kernel: BIOS-e820: [mem 0x00000000b9fff000-0x00000000b9ffffff] usable
Feb 13 05:01:43 PVE-PRD2 kernel: BIOS-e820: [mem 0x00000000ba000000-0x00000000be7fffff] reserved
Feb 13 05:01:43 PVE-PRD2 kernel: BIOS-e820: [mem 0x00000000f8000000-0x00000000fbffffff] reserved
Feb 13 05:01:43 PVE-PRD2 kernel: BIOS-e820: [mem 0x00000000fe010000-0x00000000fe010fff] reserved
Feb 13 05:01:43 PVE-PRD2 kernel: BIOS-e820: [mem 0x0000000100000000-0x000000103f7fffff] usable
Feb 13 05:01:43 PVE-PRD2 kernel: NX (Execute Disable) protection: active
Feb 13 05:01:43 PVE-PRD2 kernel: APIC: Static calls initialized
Feb 13 05:01:43 PVE-PRD2 kernel: efi: EFI v2.5 by HP
Feb 13 05:01:43 PVE-PRD2 kernel: efi: ACPI=0xb9ffe000 ACPI 2.0=0xb9ffe014 SMBIOS=0xb9765000 SMBIOS 3.0=0xb9763000 ESRT=0xb6869d58 >
Feb 13 05:01:43 PVE-PRD2 kernel: efi: Remove mem43: MMIO range=[0xf8000000-0xfbffffff] (64MB) from e820 map
Feb 13 05:01:43 PVE-PRD2 kernel: e820: remove [mem 0xf8000000-0xfbffffff] reserved
Feb 13 05:01:43 PVE-PRD2 kernel: efi: Not removing mem44: MMIO range=[0xfe010000-0xfe010fff] (4KB) from e820 map
Feb 13 05:01:43 PVE-PRD2 kernel: secureboot: Secure boot disabled
Feb 13 05:01:43 PVE-PRD2 kernel: SMBIOS 3.0.0 present.
Feb 13 05:01:43 PVE-PRD2 kernel: DMI: HP HP ProDesk 600 G3 SFF/82B4, BIOS P07 Ver. 02.04 03/30/2017
......

Initially I thought it was receiving a reset signal, but you'll notice that the last "received signal HUP" was at 00:00:42, while the last journal log (until I powered the machine on) was at "Feb 13 01:21:48 PVE-PRD2 pvedaemon[1134]: worker 820649 started".
Also, there are times where the log seems to jump around... specifically after my manual power-on.

I'm not sure what could be causing it, but I'm going to document anything I've changed to the system since installation, in hope that maybe you fine folks will know what's causing these random shutdowns/reboots.
- Node was previously part of a HA cluster. This is how I transferred all of my VMs and LXCs to this current node before decommissioning PRV-PRD1. HA was turned on, PVE2 [this machine] was removed from the cluster and PRD2 was set to the master. Min quorum was set to 1
- Node had powertop run on it in an attempt to optimize power usage.
- Node has iommu and AMD GPU passthrough enabled for one of my Ubuntu VMs. GPU is a single-slot RX6400

If there are any other logs I can grab to assist, please let me know! More of a minor annoyance since I do not run any mission critical vms on this instance, but I would like to stop being woken up by all of my lights at random hours :).
 
Have you checked the pstore (/var/lib/systemd/pstore) for messages yet?
 
I had similar issues in January. In the end I assumed it had to with an update (which I applied before crashes started), and another update of packages solved the issues.

Timeline:

Crashed on:
- 2025-01-17: ~02:30
- 2025-01-19: ~03:30
- 2025-01-23: ~03:00

This was usually around the time of backups running, but I never found a connection.

2025-01-23: Updates applied, kernel updated. Rebooted after.
No crashes since then, and haven't investigated further.
 
Last edited:
hd-- said:
Have you checked the pstore (/var/lib/systemd/pstore) for messages yet?
Click to expand...
That directory appears to be empty, even as root, as well as /sys/fs/pstore

Code: 
root@PVE-PRD2:/var/lib/systemd/coredump# sudo systemctl status systemd-pstore
○ systemd-pstore.service - Platform Persistent Storage Archival
     Loaded: loaded (/lib/systemd/system/systemd-pstore.service; enabled; preset: enabled)
     Active: inactive (dead)
  Condition: start condition failed at Mon 2025-02-17 09:44:40 CST; 1s ago
             └─ ConditionDirectoryNotEmpty=/sys/fs/pstore was not met
       Docs: man:systemd-pstore(8)
[.........]
root@PVE-PRD2:/sys/fs/pstore# cd /var/lib/systemd/pstore
root@PVE-PRD2:/var/lib/systemd/pstore# ls
root@PVE-PRD2:/var/lib/systemd/pstore#


mii said:
I had similar issues in January. In the end I assumed it had to with an update (which I applied before crashes started), and another update of packages solved the issues.

Timeline:

Crashed on:
- 2025-01-17: ~02:30
- 2025-01-19: ~03:30
- 2025-01-23: ~03:00

This was usually around the time of backups running, but I never found a connection.

2025-01-23: Updates applied, kernel updated. Rebooted after.
No crashes since then, and haven't investigated further.
Click to expand...
I can try and run any updates that are available!
 
Last edited:
You must log in or register to reply here.
Top Bottom