proxmox reboots then crashes hours later

[kysersoze81]

Hi All,
Quick post because im on lunch break.
apparently my system is deciding to perform a restart after running the dpkg-db-backup.service and then it doesn't start properly or goes down on its own with errors in ext4 where the containers are stored. It's happened several times since i setup this machine in July

logs below if anyone can offer some assitance or advice
Thanks

Oct 15 23:49:32 pve smartd[2484]: Device: /dev/sdg [SAT], SMART Usage Attribute: 194 Temperature_Celsius changed from 56 to 57
Oct 15 23:55:01 pve CRON[3935245]: pam_unix(cron:session): session opened for user root(uid=0) by (uid=0)
Oct 15 23:55:02 pve CRON[3935246]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Oct 15 23:55:02 pve CRON[3935245]: pam_unix(cron:session): session closed for user root
Oct 15 23:59:01 pve CRON[3945357]: pam_unix(cron:session): session opened for user root(uid=0) by (uid=0)
Oct 15 23:59:01 pve CRON[3945358]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 60 2)
Oct 15 23:59:02 pve CRON[3945357]: pam_unix(cron:session): session closed for user root
Oct 16 00:00:10 pve systemd[1]: Starting dpkg-db-backup.service - Daily dpkg database backup service...
Oct 16 00:00:10 pve systemd[1]: Starting logrotate.service - Rotate log files...
Oct 16 00:00:10 pve systemd[1]: Reloading pveproxy.service - PVE API Proxy Server...
Oct 16 00:00:10 pve systemd[1]: dpkg-db-backup.service: Deactivated successfully.
Oct 16 00:00:10 pve systemd[1]: Finished dpkg-db-backup.service - Daily dpkg database backup service.
Oct 16 00:00:11 pve pveproxy[3948588]: send HUP to 2933
Oct 16 00:00:11 pve pveproxy[2933]: received signal HUP
Oct 16 00:00:11 pve pveproxy[2933]: server closing
Oct 16 00:00:11 pve pveproxy[2933]: server shutdown (restart)
Oct 16 00:00:11 pve systemd[1]: Reloaded pveproxy.service - PVE API Proxy Server.
Oct 16 00:00:11 pve systemd[1]: Reloading spiceproxy.service - PVE SPICE Proxy Server...
Oct 16 00:00:12 pve spiceproxy[3948674]: send HUP to 2939
Oct 16 00:00:12 pve systemd[1]: Reloaded spiceproxy.service - PVE SPICE Proxy Server.
Oct 16 00:00:12 pve pvefw-logger[1434465]: received terminate request (signal)
Oct 16 00:00:12 pve pvefw-logger[1434465]: stopping pvefw logger
Oct 16 00:00:12 pve systemd[1]: Stopping pvefw-logger.service - Proxmox VE firewall logger...
Oct 16 00:00:12 pve spiceproxy[2939]: received signal HUP
Oct 16 00:00:12 pve spiceproxy[2939]: server closing
Oct 16 00:00:12 pve spiceproxy[2939]: server shutdown (restart)
Oct 16 00:00:12 pve pveproxy[2933]: restarting server
Oct 16 00:00:12 pve pveproxy[2933]: starting 3 worker(s)
Oct 16 00:00:12 pve pveproxy[2933]: worker 3948766 started
Oct 16 00:00:12 pve pveproxy[2933]: worker 3948767 started
Oct 16 00:00:12 pve pveproxy[2933]: worker 3948768 started
Oct 16 00:00:12 pve systemd[1]: pvefw-logger.service: Deactivated successfully.
Oct 16 00:00:12 pve systemd[1]: Stopped pvefw-logger.service - Proxmox VE firewall logger.
Oct 16 00:00:12 pve systemd[1]: pvefw-logger.service: Consumed 5.127s CPU time.
Oct 16 00:00:12 pve spiceproxy[2939]: restarting server
Oct 16 00:00:12 pve spiceproxy[2939]: starting 1 worker(s)
Oct 16 00:00:12 pve spiceproxy[2939]: worker 3948770 started
Oct 16 00:00:12 pve systemd[1]: Starting pvefw-logger.service - Proxmox VE firewall logger...
Oct 16 00:00:12 pve pvefw-logger[3948771]: starting pvefw logger
Oct 16 00:00:12 pve systemd[1]: Started pvefw-logger.service - Proxmox VE firewall logger.
Oct 16 00:00:12 pve systemd[1]: logrotate.service: Deactivated successfully.
Oct 16 00:00:12 pve systemd[1]: Finished logrotate.service - Rotate log files.
Oct 16 00:00:17 pve pveproxy[2887958]: worker exit
Oct 16 00:00:17 pve pveproxy[3513262]: worker exit
Oct 16 00:00:17 pve pveproxy[3513260]: worker exit
Oct 16 00:00:17 pve spiceproxy[3513259]: worker exit
Oct 16 00:00:19 pve pveproxy[2933]: worker 3513260 finished
Oct 16 00:00:19 pve pveproxy[2933]: worker 3513262 finished
Oct 16 00:00:19 pve pveproxy[2933]: worker 2887958 finished
Oct 16 00:00:20 pve spiceproxy[2939]: worker 3513259 finished
Oct 16 00:05:02 pve CRON[3961143]: pam_unix(cron:session): session opened for user root(uid=0) by (uid=0)
Oct 16 00:05:02 pve CRON[3961144]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Oct 16 00:05:02 pve CRON[3961143]: pam_unix(cron:session): session closed for user root
Oct 16 00:15:01 pve CRON[3985039]: pam_unix(cron:session): session opened for user root(uid=0) by (uid=0)
Oct 16 00:15:01 pve CRON[3985040]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Oct 16 00:15:01 pve CRON[3985039]: pam_unix(cron:session): session closed for user root
Oct 16 00:17:01 pve CRON[3989948]: pam_unix(cron:session): session opened for user root(uid=0) by (uid=0)
Oct 16 00:17:01 pve CRON[3989949]: (root) CMD (cd / && run-parts --report /etc/cron.hourly)
Oct 16 00:17:01 pve CRON[3989948]: pam_unix(cron:session): session closed for user root
Oct 16 00:19:32 pve smartd[2484]: Device: /dev/sdc [SAT], SMART Usage Attribute: 194 Temperature_Celsius changed from 52 to 53
Oct 16 00:19:32 pve smartd[2484]: Device: /dev/sdd [SAT], SMART Usage Attribute: 194 Temperature_Celsius changed from 53 to 54
Oct 16 00:23:59 pve pvestatd[2895]: auth key pair too old, rotating..
Oct 16 00:25:01 pve CRON[4015977]: pam_unix(cron:session): session opened for user root(uid=0) by (uid=0)
Oct 16 00:25:01 pve CRON[4015978]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
(this goes on for a bit)
Oct 16 04:49:32 pve smartd[2484]: Device: /dev/sdf [SAT], SMART Usage Attribute: 194 Temperature_Celsius changed from 59 to 60
Oct 16 04:55:01 pve CRON[200663]: pam_unix(cron:session): session opened for user root(uid=0) by (uid=0)
Oct 16 04:55:01 pve CRON[200664]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Oct 16 04:55:01 pve CRON[200663]: pam_unix(cron:session): session closed for user root
-- Reboot --
Oct 16 12:56:18 pve kernel: Linux version 6.2.16-15-pve (build@proxmox) (gcc (Debian 12.2.0-14) 12.2.0, GNU ld (GNU Binutils for Debian) 2.40) #1 SMP PREEMPT_DYNAMIC PMX 6.2.16-15 (2023-09-28T13:53Z) ()
Oct 16 12:56:18 pve kernel: Command line: BOOT_IMAGE=/boot/vmlinuz-6.2.16-15-pve root=/dev/mapper/pve-root ro quiet
Oct 16 12:56:18 pve kernel: KERNEL supported cpus:
Oct 16 12:56:18 pve kernel:   Intel GenuineIntel
Oct 16 12:56:18 pve kernel:   AMD AuthenticAMD
Oct 16 12:56:18 pve kernel:   Hygon HygonGenuine
Oct 16 12:56:18 pve kernel:   Centaur CentaurHauls
Oct 16 12:56:18 pve kernel:   zhaoxin   Shanghai 
Oct 16 12:56:18 pve kernel: BIOS-provided physical RAM map:

 

[Max Carrara]

Hi there!

From what I can tell, your system doesn't immediately reboot after dpkg-db-backup.service runs, but rather a couple hours later. Are you sure you don't have anything in your crontab or any of the files in /etc/cron.* that would signal a reboot?

[...] goes down on its own with errors in ext4 where the containers are stored.

I suggest running fsck on all of your disks then, in this case. How old are you drives? What do your containers do?

 

[kysersoze81]

It's a new system a few months old. Containers are mostly extra service's around Plex media server. But they are running from a nvme drive. Data is stored on a btrfs RAID 10 array. I have a weekly balance and monthly scrub from btrfsmaintenance tools. That last ran multiple days before the issue came about. Generally of the couple of times it's happened the system has errors on screen related to ext4 inside of a container. The containers were restored from backups to the new system when I replaced the old physical server. Maybe it's the nvme drive I'll try to fsck it when I can.

 

[Max Carrara]

Alright, let me know what you can find!