System suddenly crashed, how to find out what happened

alfwro

Member
Jan 10, 2022
11
2
8
54
I've been running proxmox on Intel Nuc (i7-10710U CPU) device for the past several years and it has always been rock solid. Today however it crashed - stopped responding to pings and I had to pull the power plug to restart it.
Code:
root@pve:~# pveversion -v
proxmox-ve: 8.2.0 (running kernel: 6.8.8-3-pve)
pve-manager: 8.2.4 (running version: 8.2.4/faa83925c9641325)
proxmox-kernel-helper: 8.1.0
pve-kernel-5.13: 7.1-9
proxmox-kernel-6.8: 6.8.8-3
proxmox-kernel-6.8.8-3-pve-signed: 6.8.8-3
proxmox-kernel-6.8.8-2-pve-signed: 6.8.8-2
proxmox-kernel-6.8.8-1-pve-signed: 6.8.8-1
proxmox-kernel-6.8.4-3-pve-signed: 6.8.4-3
proxmox-kernel-6.5.13-5-pve-signed: 6.5.13-5
proxmox-kernel-6.5: 6.5.13-5
pve-kernel-5.13.19-6-pve: 5.13.19-15
pve-kernel-5.13.19-2-pve: 5.13.19-4
ceph-fuse: 16.2.11+ds-2
corosync: 3.1.7-pve3
criu: 3.17.1-2
glusterfs-client: 10.3-5
ifupdown2: 3.2.0-1+pmx9
ksm-control-daemon: 1.5-1
libjs-extjs: 7.0.0-4
libknet1: 1.28-pve1
libproxmox-acme-perl: 1.5.1
libproxmox-backup-qemu0: 1.4.1
libproxmox-rs-perl: 0.3.3
libpve-access-control: 8.1.4
libpve-apiclient-perl: 3.3.2
libpve-cluster-api-perl: 8.0.7
libpve-cluster-perl: 8.0.7
libpve-common-perl: 8.2.1
libpve-guest-common-perl: 5.1.3
libpve-http-server-perl: 5.1.0
libpve-network-perl: 0.9.8
libpve-rs-perl: 0.8.9
libpve-storage-perl: 8.2.3
libspice-server1: 0.15.1-1
lvm2: 2.03.16-2
lxc-pve: 6.0.0-1
lxcfs: 6.0.0-pve2
novnc-pve: 1.4.0-3
proxmox-backup-client: 3.2.7-1
proxmox-backup-file-restore: 3.2.7-1
proxmox-firewall: 0.4.2
proxmox-kernel-helper: 8.1.0
proxmox-mail-forward: 0.2.3
proxmox-mini-journalreader: 1.4.0
proxmox-offline-mirror-helper: 0.6.6
proxmox-widget-toolkit: 4.2.3
pve-cluster: 8.0.7
pve-container: 5.1.12
pve-docs: 8.2.2
pve-edk2-firmware: 4.2023.08-4
pve-esxi-import-tools: 0.7.1
pve-firewall: 5.0.7
pve-firmware: 3.13-1
pve-ha-manager: 4.0.5
pve-i18n: 3.2.2
pve-qemu-kvm: 9.0.0-6
pve-xtermjs: 5.3.0-3
qemu-server: 8.2.2
smartmontools: 7.3-pve1
spiceterm: 3.3.0
swtpm: 0.8.0+pve1
vncterm: 1.8.0
zfsutils-linux: 2.2.4-pve1
root@pve:~#
here are bits from the journalctl before and after:
Code:
Jul 25 15:33:25 pve systemd[1]: Stopping user@0.service - User Manager for UID 0...
Jul 25 15:33:25 pve systemd[991896]: Activating special unit exit.target...
Jul 25 15:33:25 pve systemd[991896]: Stopped target default.target - Main User Target.
Jul 25 15:33:25 pve systemd[991896]: Stopped target basic.target - Basic System.
Jul 25 15:33:25 pve systemd[991896]: Stopped target paths.target - Paths.
Jul 25 15:33:25 pve systemd[991896]: Stopped target sockets.target - Sockets.
Jul 25 15:33:25 pve systemd[991896]: Stopped target timers.target - Timers.
Jul 25 15:33:25 pve systemd[991896]: Closed dbus.socket - D-Bus User Message Bus Socket.
Jul 25 15:33:25 pve systemd[991896]: Closed dirmngr.socket - GnuPG network certificate management daemon.
Jul 25 15:33:25 pve systemd[991896]: Closed gpg-agent-browser.socket - GnuPG cryptographic agent and passphrase cache (access for web browsers).
Jul 25 15:33:25 pve systemd[991896]: Closed gpg-agent-extra.socket - GnuPG cryptographic agent and passphrase cache (restricted).
Jul 25 15:33:25 pve systemd[991896]: Closed gpg-agent-ssh.socket - GnuPG cryptographic agent (ssh-agent emulation).
Jul 25 15:33:25 pve systemd[991896]: Closed gpg-agent.socket - GnuPG cryptographic agent and passphrase cache.
Jul 25 15:33:25 pve systemd[991896]: Removed slice app.slice - User Application Slice.
Jul 25 15:33:25 pve systemd[991896]: Reached target shutdown.target - Shutdown.
Jul 25 15:33:25 pve systemd[991896]: Finished systemd-exit.service - Exit the Session.
Jul 25 15:33:25 pve systemd[991896]: Reached target exit.target - Exit the Session.
Jul 25 15:33:25 pve systemd[1]: user@0.service: Deactivated successfully.
Jul 25 15:33:25 pve systemd[1]: Stopped user@0.service - User Manager for UID 0.
Jul 25 15:33:25 pve systemd[1]: Stopping user-runtime-dir@0.service - User Runtime Directory /run/user/0...
Jul 25 15:33:25 pve systemd[1]: run-user-0.mount: Deactivated successfully.
Jul 25 15:33:25 pve systemd[1]: user-runtime-dir@0.service: Deactivated successfully.
Jul 25 15:33:25 pve systemd[1]: Stopped user-runtime-dir@0.service - User Runtime Directory /run/user/0.
Jul 25 15:33:25 pve systemd[1]: Removed slice user-0.slice - User Slice of UID 0.
-- Reboot --
Jul 25 16:39:58 pve kernel: Linux version 6.8.8-3-pve (build@proxmox) (gcc (Debian 12.2.0-14) 12.2.0, GNU ld (GNU Binutils for Debian) 2.40) #1 SMP PREEMPT_DYNAMIC PMX 6.8.8-3 (2024-07-16T16:16Z) ()
Jul 25 16:39:58 pve kernel: Command line: BOOT_IMAGE=/boot/vmlinuz-6.8.8-3-pve root=/dev/mapper/pve-root ro quiet intel_iommu=on
Jul 25 16:39:58 pve kernel: KERNEL supported cpus:
Jul 25 16:39:58 pve kernel:   Intel GenuineIntel
Jul 25 16:39:58 pve kernel:   AMD AuthenticAMD
Jul 25 16:39:58 pve kernel:   Hygon HygonGenuine
Jul 25 16:39:58 pve kernel:   Centaur CentaurHauls
I cannot see anything in the obvious log files (journalctl - no errors or faults etc)
How do I find out what caused it? Where else do I look?
Thanks
 
Last edited:

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!