Yes, another Reboot post

[bix88]

Hello everyone
On my lab i'm running a Fujitsu Futro s920 since couple of years, running PFsense/OPNsense bare metal without any notable issues.
Weeks ago i decided to install Proxmox 7.4-4 VE and deploy a Fortigate VM on it.
After less than 24 hours the entire proxmox node start rebooting without any apparent reason and continuing rebooting every random hours.

This is what i discovered troubleshooting so far:

- Proxmox reboot only if a VM is runnig
- Changing hw resourced to the VM doesn't effect the reboot
- Plugged/unplugged network cables doesn't effect

Some spec about the HW:
Fujitsu Futro s920, 8GB RAM, 2,20 GHz GX-222GC CPU, brand new 256GB mSATA SSD, 4x1GB NIC card from Intel + active cooling to exclude a potential temperature issue).

And follow you can found some log, (attached even more).
One thing i found on the log is a series of "Stopping" before the reboot.

Jun 27 14:47:12 node1 systemd[1]: pvedaemon.service: Succeeded.
Jun 27 14:47:12 node1 systemd[1]: Stopped PVE API Daemon.
Jun 27 14:47:12 node1 systemd[1]: pvedaemon.service: Consumed 6.797s CPU time.
Jun 27 14:47:12 node1 systemd[1]: Stopping The Proxmox VE cluster filesystem...
Jun 27 14:47:12 node1 pmxcfs[871]: [main] notice: teardown filesystem
Jun 27 14:47:12 node1 systemd[1]: etc-pve.mount: Succeeded.
Jun 27 14:47:12 node1 systemd[1]: Unmounted /etc/pve.
Jun 27 14:47:12 node1 pmxcfs[871]: [main] notice: exit proxmox configuration filesystem (0)
Jun 27 14:47:12 node1 systemd[1]: pve-cluster.service: Succeeded.
Jun 27 14:47:12 node1 systemd[1]: Stopped The Proxmox VE cluster filesystem.
Jun 27 14:47:12 node1 systemd[1]: Stopping LSB: start or stop rrdcached...
Jun 27 14:47:13 node1 rrdcached[3326]: rrdcached has been stopped.
Jun 27 14:47:13 node1 systemd[1]: rrdcached.service: Succeeded.
Jun 27 14:47:13 node1 systemd[1]: Stopped LSB: start or stop rrdcached.
Jun 27 14:47:13 node1 systemd[1]: Stopped target Remote File Systems.
Jun 27 14:47:13 node1 systemd[1]: Stopped target Remote File Systems (Pre).
Jun 27 14:47:13 node1 systemd[1]: Stopped target NFS client services.
Jun 27 14:47:13 node1 systemd[1]: Stopped target System Time Synchronized.
Jun 27 14:47:13 node1 systemd[1]: Stopped target System Time Set.
Jun 27 14:47:13 node1 systemd[1]: Stopping chrony, an NTP client/server...
Jun 27 14:47:13 node1 chronyd[786]: chronyd exiting
Jun 27 14:47:13 node1 systemd[1]: Stopping Login to default iSCSI targets...
Jun 27 14:47:13 node1 systemd[1]: chrony.service: Succeeded.
Jun 27 14:47:13 node1 systemd[1]: Stopped chrony, an NTP client/server.
Jun 27 14:47:13 node1 systemd[1]: Stopped target Basic System.
Jun 27 14:47:13 node1 systemd[1]: Stopped target Paths.
Jun 27 14:47:13 node1 systemd[1]: Stopped target Slices.
Jun 27 14:47:13 node1 systemd[1]: Removed slice User and Session Slice.
Jun 27 14:47:13 node1 systemd[1]: Stopped target Sockets.
Jun 27 14:47:13 node1 systemd[1]: dbus.socket: Succeeded.
Jun 27 14:47:13 node1 systemd[1]: Closed D-Bus System Message Bus Socket.
Jun 27 14:47:13 node1 systemd[1]: Stopped target System Initialization.
Jun 27 14:47:13 node1 systemd[1]: Stopped target Local Encrypted Volumes.
Jun 27 14:47:13 node1 systemd[1]: systemd-ask-password-console.path: Succeeded.
Jun 27 14:47:13 node1 systemd[1]: Stopped Dispatch Password Requests to Console Directory Watch.
Jun 27 14:47:13 node1 systemd[1]: systemd-ask-password-wall.path: Succeeded.
Jun 27 14:47:13 node1 systemd[1]: Stopped Forward Password Requests to Wall Directory Watch.
Jun 27 14:47:13 node1 systemd[1]: Stopped target Swap.
Jun 27 14:47:13 node1 systemd[1]: syslog.socket: Succeeded.
Jun 27 14:47:13 node1 systemd[1]: Closed Syslog Socket.
Jun 27 14:47:13 node1 systemd[1]: Deactivating swap /dev/disk/by-id/dm-name-pve-swap...
Jun 27 14:47:13 node1 systemd[1]: Stopping Load/Save Screen Backlight Brightness of backlight:acpi_video0...
Jun 27 14:47:13 node1 systemd[1]: Stopping Update UTMP about System Boot/Shutdown...
Jun 27 14:47:13 node1 systemd[1]: dev-disk-by\x2duuid-8ccb6d4b\x2d0a04\x2d4136\x2dba85\x2de79091405e90.swap: Succeeded.
Jun 27 14:47:13 node1 systemd[1]: Deactivated swap /dev/disk/by-uuid/8ccb6d4b-0a04-4136-ba85-e79091405e90.
Jun 27 14:47:13 node1 systemd[1]: dev-disk-by\x2did-dm\x2duuid\x2dLVM\x2dCF4A4gCeOo2ysNtaqrlXHFVQuPyG6173v3P5BoKHOp9auD5N1bTVlGldkDfLAu6t.swap: Succeeded.
Jun 27 14:47:13 node1 systemd[1]: Deactivated swap /dev/disk/by-id/dm-uuid-LVM-CF4A4gCeOo2ysNtaqrlXHFVQuPyG6173v3P5BoKHOp9auD5N1bTVlGldkDfLAu6t.
Jun 27 14:47:13 node1 systemd[1]: dev-disk-by\x2did-dm\x2dname\x2dpve\x2dswap.swap: Succeeded.
Jun 27 14:47:13 node1 systemd[1]: Deactivated swap /dev/disk/by-id/dm-name-pve-swap.
Jun 27 14:47:13 node1 systemd[1]: dev-mapper-pve\x2dswap.swap: Succeeded.
Jun 27 14:47:13 node1 systemd[1]: Deactivated swap /dev/mapper/pve-swap.
Jun 27 14:47:13 node1 systemd[1]: dev-dm\x2d0.swap: Succeeded.
Jun 27 14:47:13 node1 systemd[1]: Deactivated swap /dev/dm-0.
Jun 27 14:47:13 node1 systemd[1]: dev-pve-swap.swap: Succeeded.
Jun 27 14:47:13 node1 systemd[1]: Deactivated swap /dev/pve/swap.
Jun 27 14:47:13 node1 systemd[1]: Reached target Unmount All Filesystems.
Jun 27 14:47:13 node1 systemd[1]: systemd-backlight@backlight:acpi_video0.service: Succeeded.
Jun 27 14:47:13 node1 systemd[1]: Stopped Load/Save Screen Backlight Brightness of backlight:acpi_video0.
Jun 27 14:47:13 node1 systemd[1]: Removed slice system-systemd\x2dbacklight.slice.
Jun 27 14:47:13 node1 systemd[1]: systemd-update-utmp.service: Succeeded.
Jun 27 14:47:13 node1 systemd[1]: Stopped Update UTMP about System Boot/Shutdown.
Jun 27 14:47:13 node1 systemd[1]: systemd-tmpfiles-setup.service: Succeeded.
Jun 27 14:47:13 node1 systemd[1]: Stopped Create Volatile Files and Directories.
Jun 27 14:47:13 node1 logout-all.sh[3375]: iscsiadm: No matching sessions found
Jun 27 14:47:13 node1 systemd[1]: open-iscsi.service: Succeeded.
Jun 27 14:47:13 node1 systemd[1]: Stopped Login to default iSCSI targets.
Jun 27 14:47:13 node1 systemd[1]: Stopping iSCSI initiator daemon (iscsid)...
Jun 27 14:47:13 node1 iscsid[758]: iscsid shutting down.
Jun 27 14:47:13 node1 systemd[1]: iscsid.service: Succeeded.
Jun 27 14:47:13 node1 systemd[1]: Stopped iSCSI initiator daemon (iscsid).
Jun 27 14:47:13 node1 systemd[1]: Stopped target Network is Online.
Jun 27 14:47:13 node1 systemd[1]: Stopped target Network.
Jun 27 14:47:13 node1 systemd[1]: Stopping Network initialization...
Jun 27 14:47:13 node1 networking[3377]: networking: Skipping deconfiguring network interfaces
Jun 27 14:47:13 node1 systemd[1]: networking.service: Succeeded.
Jun 27 14:47:13 node1 systemd[1]: Stopped Network initialization.
Jun 27 14:47:13 node1 systemd[1]: Stopped target Local File Systems.
Jun 27 14:47:13 node1 systemd[1]: Stopped target Local File Systems (Pre).
Jun 27 14:47:13 node1 systemd[1]: Stopping Monitoring of LVM2 mirrors, snapshots etc. using dmeventd or progress polling...
Jun 27 14:47:13 node1 systemd[1]: systemd-sysctl.service: Succeeded.
Jun 27 14:47:13 node1 systemd[1]: Stopped Apply Kernel Variables.
Jun 27 14:47:13 node1 systemd[1]: systemd-modules-load.service: Succeeded.
Jun 27 14:47:13 node1 systemd[1]: Stopped Load Kernel Modules.
Jun 27 14:47:13 node1 systemd[1]: systemd-tmpfiles-setup-dev.service: Succeeded.
Jun 27 14:47:13 node1 systemd[1]: Stopped Create Static Device Nodes in /dev.
Jun 27 14:47:13 node1 systemd[1]: systemd-sysusers.service: Succeeded.
Jun 27 14:47:13 node1 systemd[1]: Stopped Create System Users.
Jun 27 14:47:13 node1 systemd[1]: systemd-remount-fs.service: Succeeded.
Jun 27 14:47:13 node1 systemd[1]: Stopped Remount Root and Kernel File Systems.
Jun 27 14:47:13 node1 lvm[3380]:   8 logical volume(s) in volume group "pve" unmonitored
Jun 27 14:47:13 node1 dmeventd[351]: No longer monitoring thin pool pve-data-tpool.
Jun 27 14:47:13 node1 systemd[1]: lvm2-monitor.service: Succeeded.
Jun 27 14:47:13 node1 systemd[1]: Stopped Monitoring of LVM2 mirrors, snapshots etc. using dmeventd or progress polling.
Jun 27 14:47:13 node1 systemd[1]: Stopping Device-mapper event daemon...
Jun 27 14:47:13 node1 dmeventd[351]: dmeventd detected break while being idle for 0 second(s), exiting.
Jun 27 14:47:13 node1 dmeventd[351]: dmeventd shutting down.
Jun 27 14:47:13 node1 systemd[1]: dm-event.service: Succeeded.
Jun 27 14:47:13 node1 systemd[1]: Stopped Device-mapper event daemon.
Jun 27 14:47:13 node1 systemd[1]: Reached target Shutdown.
Jun 27 14:47:13 node1 systemd[1]: Reached target Final Step.
Jun 27 14:47:13 node1 systemd[1]: systemd-poweroff.service: Succeeded.
Jun 27 14:47:13 node1 systemd[1]: Finished Power-Off.
Jun 27 14:47:13 node1 systemd[1]: Reached target Power-Off.
Jun 27 14:47:13 node1 systemd[1]: Shutting down.
Jun 27 14:47:14 node1 systemd-shutdown: Syncing filesystems and block devices.
Jun 27 14:47:14 node1 systemd-shutdown: Sending SIGTERM to remaining processes...
Jun 27 14:47:14 node1 systemd-journald[341]: Journal stopped
-- Reboot --
Jun 27 15:50:15 node1 kernel: Linux version 5.15.107-2-pve (build@proxmox) (gcc (Debian 10.2.1-6) 10.2.1 20210110, GNU ld (GNU Binutils for Debian) 2.35.2) #1 SMP PVE 5.15.107-2 (2023-05-10T09:10Z) ()

Thanks everyone

 

[spirit]

this look like a clean shutdown..... (with shutdown or reboot command) .... like a manual action

 

[bix88]

Tthanks for reply.
Is there any way to debug what is triggering the shutdown sequence or any option i misconfigured?
I've checked on VM logs too but i have only found something about an "unexpected shutdown" logs when Proxmox reboot.

Thanks