Proxmox 5 not booting after crash - Ovh

[spiderdk9900]

Hi, server crashed, I rebooted and it does not boot anymore. Into rescue mode it got accessible. Where is the correct log file location and name for me to know the exact problem?

(Not sure if this log is useful)
SYSLOG file:

Dec 31 06:48:03 ns505710 liblogging-stdlog: [origin software="rsyslogd" swVersion="8.24.0" x-pid="1880" x-info="http://www.rsyslog.com"] rsyslogd was HUPed
Dec 31 06:48:03 ns505710 spiceproxy[2729]: restarting server
Dec 31 06:48:03 ns505710 spiceproxy[2729]: starting 1 worker(s)
Dec 31 06:48:03 ns505710 spiceproxy[2729]: worker 31309 started
Dec 31 06:48:03 ns505710 pveproxy[2698]: restarting server
Dec 31 06:48:03 ns505710 pveproxy[2698]: starting 3 worker(s)
Dec 31 06:48:03 ns505710 pveproxy[2698]: worker 31318 started
Dec 31 06:48:03 ns505710 pveproxy[2698]: worker 31319 started
Dec 31 06:48:03 ns505710 pveproxy[2698]: worker 31320 started
Dec 31 06:48:08 ns505710 spiceproxy[7043]: worker exit
Dec 31 06:48:08 ns505710 spiceproxy[2729]: worker 7043 finished
Dec 31 06:48:08 ns505710 pveproxy[7054]: worker exit
Dec 31 06:48:08 ns505710 pveproxy[7053]: worker exit
Dec 31 06:48:08 ns505710 pveproxy[7052]: worker exit
Dec 31 06:48:08 ns505710 pveproxy[2698]: worker 7054 finished
Dec 31 06:48:08 ns505710 pveproxy[2698]: worker 7053 finished
Dec 31 06:48:08 ns505710 pveproxy[2698]: worker 7052 finished

Dec 31 07:27:36 ns505710 rrdcached[2344]: flushing old values
Dec 31 07:27:36 ns505710 rrdcached[2344]: rotating journals
Dec 31 07:27:36 ns505710 rrdcached[2344]: started new journal /var/lib/rrdcached/journal/rrd.journal.1577788056.396478
Dec 31 07:27:36 ns505710 rrdcached[2344]: removing old journal /var/lib/rrdcached/journal/rrd.journal.1577780856.396468
Dec 31 07:27:40 ns505710 smartd[1862]: Device: /dev/sdb [SAT], SMART Usage Attribute: 190 Airflow_Temperature_Cel changed from 80 to 79

Dec 31 07:43:00 ns505710 systemd[1]: Started Proxmox VE replication runner.
Dec 31 07:43:01 ns505710 CRON[21591]: (root) CMD ( cd / && run-parts --report /etc/cron.hourly)

Dec 31 07:57:40 ns505710 smartd[1862]: Device: /dev/sdb [SAT], SMART Usage Attribute: 190 Airflow_Temperature_Cel changed from 79 to 80

Dec 31 08:27:36 ns505710 rrdcached[2344]: flushing old values
Dec 31 08:27:36 ns505710 rrdcached[2344]: rotating journals
Dec 31 08:27:36 ns505710 rrdcached[2344]: started new journal /var/lib/rrdcached/journal/rrd.journal.1577791656.396410
Dec 31 08:27:36 ns505710 rrdcached[2344]: removing old journal /var/lib/rrdcached/journal/rrd.journal.1577784456.396430

Dec 31 08:35:29 ns505710 systemd[1]: Stopped 102.scope.
Dec 31 08:35:29 ns505710 systemd[1]: Stopped 790.scope.
Dec 31 08:35:29 ns505710 systemd[1]: Stopped 791.scope.
Dec 31 08:35:29 ns505710 systemd[1]: Stopped target Timers.
Dec 31 08:35:29 ns505710 systemd[1]: Unmounting RPC Pipe File System...
Dec 31 08:35:29 ns505710 systemd[1]: Stopped 787.scope.
Dec 31 08:35:29 ns505710 systemd[1]: Stopped target Graphical Interface.
Dec 31 08:35:29 ns505710 systemd[1]: Stopped target Multi-User System.
Dec 31 08:35:29 ns505710 systemd[1]: Stopping PVE Qemu Event Daemon...
Dec 31 08:35:29 ns505710 systemd[1]: Stopping LSB: Ceph RBD Mapping...
Dec 31 08:35:29 ns505710 systemd[1]: Stopping Login Service...
Dec 31 08:35:29 ns505710 systemd[1]: Stopping PVE guests...
Dec 31 08:35:29 ns505710 systemd[1]: Stopping D-Bus System Message Bus...
Dec 31 08:35:29 ns505710 systemd[1]: Stopping irqbalance daemon...
Dec 31 08:35:29 ns505710 systemd[1]: Stopping Self Monitoring and Reporting Technology (SMART) Daemon...
Dec 31 08:35:29 ns505710 smartd[1862]: smartd received signal 15: Terminated
Dec 31 08:35:29 ns505710 smartd[1862]: Device: /dev/sda [SAT], state written to /var/lib/smartmontools/smartd.INTEL_SSDSC2BB480G7-PHDV7020030T480BGN.ata.state
Dec 31 08:35:29 ns505710 smartd[1862]: Device: /dev/sdb [SAT], state written to /var/lib/smartmontools/smartd.INTEL_SSDSC2BB480G7-PHDV647601D0480BGN.ata.state
Dec 31 08:35:29 ns505710 systemd[1]: Stopped Postfix Mail Transport Agent.
Dec 31 08:35:29 ns505710 smartd[1862]: Device: /dev/sdc [SAT], state written to /var/lib/smartmontools/smartd.HGST_HUS724020ALA640-PN1134P6HY8GVW.ata.state
Dec 31 08:35:29 ns505710 systemd[1]: Closed Load/Save RF Kill Switch Status /dev/rfkill Watch.
Dec 31 08:35:29 ns505710 smartd[1862]: smartd is exiting (exit status 0)
Dec 31 08:35:29 ns505710 systemd[1]: Stopped Daily PVE download activities.
Dec 31 08:35:29 ns505710 systemd[1]: Stopping LSB: disk temperature monitoring daemon...
Dec 31 08:35:29 ns505710 systemd[1]: Stopped target ZFS startup target.
Dec 31 08:35:29 ns505710 systemd[1]: Stopped target ZFS pool import target.
Dec 31 08:35:29 ns505710 systemd[1]: Stopped ZFS file system shares.

Dec 31 08:35:29 ns505710 systemd[1]: Stopping Regular background program processing daemon...
Dec 31 08:35:29 ns505710 systemd[1]: Stopped 792.scope.
Dec 31 08:35:29 ns505710 systemd[1]: Stopped Proxmox VE replication runner.
Dec 31 08:35:29 ns505710 systemd[1]: Stopped target Sound Card.
Dec 31 08:35:29 ns505710 systemd[1]: Stopped target RPC Port Mapper.
Dec 31 08:35:29 ns505710 systemd[1]: Stopped 300.scope.
Dec 31 08:35:29 ns505710 systemd[1]: Removed slice qemu.slice.
Dec 31 08:35:29 ns505710 systemd[1]: Stopped Daily Cleanup of Temporary Directories.
Dec 31 08:35:29 ns505710 systemd[1]: Stopping Postfix Mail Transport Agent (instance -)...
Dec 31 08:35:29 ns505710 systemd[1]: Stopped target Login Prompts.
Dec 31 08:35:29 ns505710 systemd[1]: Stopping Getty on tty1...
Dec 31 08:35:29 ns505710 systemd[1]: Stopped irqbalance daemon.
Dec 31 08:35:29 ns505710 systemd[1]: Stopped Self Monitoring and Reporting Technology (SMART) Daemon.
Dec 31 08:35:29 ns505710 systemd[1]: Stopped D-Bus System Message Bus.
Dec 31 08:35:29 ns505710 systemd[1]: Stopped PVE Qemu Event Daemon.
Dec 31 08:35:29 ns505710 systemd[1]: Stopped Kernel Samepage Merging (KSM) Tuning Daemon.
Dec 31 08:35:29 ns505710 systemd[1]: Stopped LXC Container Monitoring Daemon.
Dec 31 08:35:29 ns505710 systemd[1]: Stopped Regular background program processing daemon.
Dec 31 08:35:29 ns505710 systemd[1]: Failed to propagate agent release message: Transport endpoint is not connected

Dec 31 08:35:29 ns505710 systemd[1]: Removed slice system-postfix.slice.
Dec 31 08:35:30 ns505710 pve-guests[11160]: <root@pam> starting task UPID:ns505710:00002BA1:08A10119:5E0B3282:qmshutdown:300:root@pam:
Dec 31 08:35:30 ns505710 pve-guests[11169]: shutdown VM 300: UPID:ns505710:00002BA1:08A10119:5E0B3282:qmshutdown:300:root@pam:
Dec 31 08:35:30 ns505710 pvesh[11153]: Stopping VM 102 (timeout = 180 seconds)
Dec 31 08:38:31 ns505710 kernel: [1447898.270258] fwbr787i0: port 2(tap787i0) entered disabled state
Dec 31 08:38:31 ns505710 systemd-timesyncd[1598]: Network configuration changed, trying to establish connection.
Dec 31 08:38:31 ns505710 kernel: [1447898.310313] fwbr790i0: port 1(fwln790i0) entered disabled state

 

[spiderdk9900]

Tried to reboot with original iptables (firewall) configuration and its not acessible. Any clues? Where is the log location of startup Proxmox 5 ?

The machine does not even boot (it seems), where can i get information about it?

The machine is a Xeon E3-1245v2, 32gb, 2x480gb SSD + 2x2Tb HDD on ovh, it was running for a month with no issue, than it crashed and when i reboot it does not goes up anymore.

 

[spiderdk9900]

daemon.log information:



Dec 31 06:48:00 ns505710 systemd[1]: Started Proxmox VE replication runner.
Dec 31 06:48:01 ns505710 systemd[1]: Reloading PVE API Proxy Server.
Dec 31 06:48:02 ns505710 pveproxy[31253]: send HUP to 2698
Dec 31 06:48:02 ns505710 pveproxy[2698]: received signal HUP
Dec 31 06:48:02 ns505710 pveproxy[2698]: server closing
Dec 31 06:48:02 ns505710 pveproxy[2698]: server shutdown (restart)
Dec 31 06:48:02 ns505710 systemd[1]: Reloaded PVE API Proxy Server.
Dec 31 06:48:02 ns505710 systemd[1]: Reloading PVE SPICE Proxy Server.
Dec 31 06:48:03 ns505710 spiceproxy[31261]: send HUP to 2729
Dec 31 06:48:03 ns505710 spiceproxy[2729]: received signal HUP
Dec 31 06:48:03 ns505710 spiceproxy[2729]: server closing
Dec 31 06:48:03 ns505710 spiceproxy[2729]: server shutdown (restart)
Dec 31 06:48:03 ns505710 systemd[1]: Reloaded PVE SPICE Proxy Server.
Dec 31 06:48:03 ns505710 systemd[1]: Stopping Proxmox VE firewall logger...
Dec 31 06:48:03 ns505710 pvefw-logger[7015]: received terminate request (signal)
Dec 31 06:48:03 ns505710 pvefw-logger[7015]: stopping pvefw logger
Dec 31 06:48:03 ns505710 systemd[1]: Stopped Proxmox VE firewall logger.
Dec 31 06:48:03 ns505710 systemd[1]: Starting Proxmox VE firewall logger...
Dec 31 06:48:03 ns505710 pvefw-logger[31281]: starting pvefw logger
Dec 31 06:48:03 ns505710 systemd[1]: Started Proxmox VE firewall logger.
Dec 31 06:48:03 ns505710 spiceproxy[2729]: restarting server
Dec 31 06:48:03 ns505710 spiceproxy[2729]: starting 1 worker(s)
Dec 31 06:48:03 ns505710 spiceproxy[2729]: worker 31309 started
Dec 31 06:48:03 ns505710 pveproxy[2698]: restarting server
Dec 31 06:48:03 ns505710 pveproxy[2698]: starting 3 worker(s)
Dec 31 06:48:03 ns505710 pveproxy[2698]: worker 31318 started
Dec 31 06:48:03 ns505710 pveproxy[2698]: worker 31319 started
Dec 31 06:48:03 ns505710 pveproxy[2698]: worker 31320 started
Dec 31 06:48:08 ns505710 spiceproxy[7043]: worker exit
Dec 31 06:48:08 ns505710 spiceproxy[2729]: worker 7043 finished
Dec 31 06:48:08 ns505710 pveproxy[7054]: worker exit
Dec 31 06:48:08 ns505710 pveproxy[7053]: worker exit
Dec 31 06:48:08 ns505710 pveproxy[7052]: worker exit
Dec 31 06:48:08 ns505710 pveproxy[2698]: worker 7054 finished
Dec 31 06:48:08 ns505710 pveproxy[2698]: worker 7053 finished
Dec 31 06:48:08 ns505710 pveproxy[2698]: worker 7052 finished
Dec 31 06:49:00 ns505710 systemd[1]: Starting Proxmox VE replication runner...

 

[spiderdk9900]

Daemon log on the reboot/cash time:


..
08:35:00 systemd[1]: Started Proxmox VE replication runner.
08:35:29 systemd[1]: Stopped 102.scope.
08:35:29 systemd[1]: Stopped 790.scope.
08:35:29 systemd[1]: Stopped 791.scope.
08:35:29 systemd[1]: Stopped target Timers.
08:35:29 systemd[1]: Unmounting RPC Pipe File System...
08:35:29 systemd[1]: Stopped 787.scope.
08:35:29 systemd[1]: Stopped target Graphical Interface.
08:35:29 systemd[1]: Stopped target Multi-User System.
08:35:29 systemd[1]: Stopping PVE Qemu Event Daemon...
08:35:29 systemd[1]: Stopping LSB: Ceph RBD Mapping...
08:35:29 systemd[1]: Stopping Login Service...
08:35:29 systemd[1]: Stopping PVE guests...
08:35:29 systemd[1]: Stopping D-Bus System Message Bus...
08:35:29 systemd[1]: Stopping irqbalance daemon...
08:35:29 systemd[1]: Stopping Self Monitoring and Reporting Technology (SMART) Daemon...
08:35:29 smartd[1862]: smartd received signal 15: Terminated
08:35:29 smartd[1862]: Device: /dev/sda [SAT], state written to /var/lib/smartmontools/smartd.INTEL_SSDSC2BB480G7-PHDV7020030T480BGN.ata.state
08:35:29 smartd[1862]: Device: /dev/sdb [SAT], state written to /var/lib/smartmontools/smartd.INTEL_SSDSC2BB480G7-PHDV647601D0480BGN.ata.state
08:35:29 systemd[1]: Stopped Postfix Mail Transport Agent.
08:35:29 smartd[1862]: Device: /dev/sdc [SAT], state written to /var/lib/smartmontools/smartd.HGST_HUS724020ALA640-PN1134P6HY8GVW.ata.state
08:35:29 systemd[1]: Stopping LXC Container Monitoring Daemon...
08:35:29 smartd[1862]: Device: /dev/sdd [SAT], state written to /var/lib/smartmontools/smartd.HGST_HUS724020ALA640-PN1134P6HWZDWW.ata.state
08:35:29 systemd[1]: Stopping Kernel Samepage Merging (KSM) Tuning Daemon...
08:35:29 systemd[1]: Closed Load/Save RF Kill Switch Status /dev/rfkill Watch.
08:35:29 smartd[1862]: smartd is exiting (exit status 0)
08:35:29 systemd[1]: Stopped Daily PVE download activities.
08:35:29 systemd[1]: Stopping LSB: disk temperature monitoring daemon...
08:35:29 systemd[1]: Stopped target ZFS startup target.
08:35:29 systemd[1]: Stopped target ZFS pool import target.
08:35:29 systemd[1]: Stopped ZFS file system shares.
08:35:29 systemd[1]: Stopped Daily apt upgrade and clean activities.
08:35:29 systemd[1]: Stopped Daily apt download activities.
08:35:29 systemd[1]: Stopping Regular background program processing daemon...
08:35:29 systemd[1]: Stopped 792.scope.
08:35:29 systemd[1]: Stopped Proxmox VE replication runner.
08:35:29 systemd[1]: Stopped target Sound Card.
08:35:29 systemd[1]: Stopped target RPC Port Mapper.
08:35:29 systemd[1]: Stopped 300.scope.
08:35:29 systemd[1]: Removed slice qemu.slice.
08:35:29 systemd[1]: Stopped Daily Cleanup of Temporary Directories.
08:35:29 systemd[1]: Stopping Postfix Mail Transport Agent (instance -)...
08:35:29 systemd[1]: Stopped target Login Prompts.
08:35:29 systemd[1]: Stopping Getty on tty1...
08:35:29 systemd[1]: Stopped irqbalance daemon.
08:35:29 systemd[1]: Stopped Self Monitoring and Reporting Technology (SMART) Daemon.
08:35:29 systemd[1]: Stopped D-Bus System Message Bus.
08:35:29 systemd[1]: Stopped PVE Qemu Event Daemon.
08:35:29 systemd[1]: Stopped Kernel Samepage Merging (KSM) Tuning Daemon.
08:35:29 systemd[1]: Stopped LXC Container Monitoring Daemon.
08:35:29 systemd[1]: Stopped Regular background program processing daemon.
08:35:29 systemd[1]: Failed to propagate agent release message: Transport endpoint is not connected
08:35:29 systemd[1]: Failed to propagate agent release message: Transport endpoint is not connected

08:35:29 systemd[1]: Stopped Getty on tty1.
08:35:29 systemd[1]: Failed to propagate agent release message: Transport endpoint is not connected
08:35:29 systemd[1]: Stopping Permit User Sessions...
08:35:29 systemd[1]: Removed slice system-getty.slice.
08:35:29 systemd[1]: Unmounted RPC Pipe File System.
08:35:29 systemd[1]: Stopped Permit User Sessions.
08:35:29 systemd[1]: Stopped LSB: disk temperature monitoring daemon.
08:35:29 systemd[1]: Stopped LSB: Ceph RBD Mapping.
08:35:29 systemd[1]: Stopped Postfix Mail Transport Agent (instance -).
08:35:29 systemd[1]: Removed slice system-postfix.slice.
08:35:29 systemd[1]: Stopped Login Service.
08:35:30 pvesh[11153]: Stopping VM 792 (timeout = 180 seconds)
08:35:30 pve-guests[11162]: shutdown VM 792: UPID:ns505710:00002B9A:08A10115:5E0B3282:qmshutdown:792:root@pam:
08:35:30 pvesh[11153]: Stopping VM 791 (timeout = 180 seconds)
08:35:30 pve-guests[11163]: shutdown VM 791: UPID:ns505710:00002B9B:08A10116:5E0B3282:qmshutdown:791:root@pam:

08:35:31 pve-guests[11171]: shutdown VM 102: UPID:ns505710:00002BA3:08A1011A:5E0B3282:qmshutdown:102:root@pam:
08:38:09 pvedaemon[12338]: starting vnc proxy UPID:ns505710:00003032:08A13F15:5E0B3321:vncproxy:300:root@pam:
08:38:31 systemd-networkd[1620]: fwln790i0: Lost carrier
08:38:31 systemd-networkd[1620]: fwpr790p0: Lost carrier
08:38:31 systemd-networkd[1620]: fwbr790i0: Lost carrier

08:38:32 systemd-networkd[1620]: tap102i0: Lost carrier
08:38:32 pve-guests[11160]: end task UPID:ns505710:00002B9D:08A10117:5E0B3282:qmshutdown:790:root@pam:

08:38:32 systemd-networkd[1620]: vmbr2: Lost carrier
08:38:33 pve-guests[11160]: end task UPID:ns505710:00002BA3:08A1011A:5E0B3282:qmshutdown:102:root@pam:
08:38:33 pve-guests[11160]: end task UPID:ns505710:00002BA1:08A10119:5E0B3282:qmshutdown:300:root@pam:
08:38:34 pve-guests[11160]: all VMs and CTs stopped
08:38:34 systemd[1]: Stopped PVE guests.
08:38:34 systemd[1]: Stopping PVE Local HA Ressource Manager Daemon...
08:38:34 systemd[1]: Stopping PVE SPICE Proxy Server...
08:38:34 systemd[1]: Stopping PVE Status Daemon...
08:38:34 systemd[1]: Stopping Proxmox VE firewall...
08:38:34 pve-firewall[2638]: received signal TERM
08:38:34 pve-firewall[2638]: server closing
08:38:34 pve-firewall[2638]: clear firewall rules
08:38:34 pvestatd[2631]: received signal TERM
08:38:34 pvestatd[2631]: server closing
08:38:34 pvestatd[2631]: server stopped
08:38:34 pve-firewall[2638]: server stopped
08:38:34 spiceproxy[2729]: received signal TERM
08:38:34 spiceproxy[2729]: server closing
08:38:34 spiceproxy[31309]: worker exit
08:38:34 spiceproxy[2729]: worker 31309 finished
08:38:34 spiceproxy[2729]: server stopped
08:38:34 pve-ha-lrm[19874]: received signal TERM
08:38:34 pve-ha-lrm[19874]: got shutdown request with shutdown policy 'conditional'
08:38:34 pve-ha-lrm[19874]: reboot LRM, stop and freeze all services
08:38:34 pve-ha-lrm[19874]: server stopped
08:38:35 systemd[1]: Stopped Proxmox VE firewall.
08:38:35 systemd[1]: Stopping Proxmox VE firewall logger...
08:38:35 pvefw-logger[31281]: received terminate request (signal)
08:38:35 pvefw-logger[31281]: stopping pvefw logger
08:38:35 systemd[1]: Stopped PVE Status Daemon.
08:38:35 systemd[1]: Stopped PVE SPICE Proxy Server.
08:38:35 systemd[1]: Stopping PVE API Proxy Server...
08:38:35 systemd[1]: Stopped Proxmox VE firewall logger.
08:38:35 systemd[1]: Stopped PVE Local HA Ressource Manager Daemon.
08:38:35 systemd[1]: Stopping LXC Container Initialization and Autoboot Code...
08:38:35 systemd[1]: Stopping PVE Cluster Ressource Manager Daemon...
08:38:35 systemd[1]: lxc.service: Control process exited, code=exited status=1
08:38:35 systemd[1]: Stopped LXC Container Initialization and Autoboot Code.
08:38:35 systemd[1]: lxc.service: Unit entered failed state.
08:38:35 systemd[1]: lxc.service: Failed with result 'exit-code'.
08:38:35 systemd[1]: Stopping LXC network bridge setup...
08:38:35 systemd[1]: Stopping FUSE filesystem for LXC...
08:38:35 systemd[1]: Unmounted /var/lib/lxcfs.
08:38:35 systemd[1]: Stopped LXC network bridge setup.
08:38:35 fusermount[12697]: /bin/fusermount: failed to unmount /var/lib/lxcfs: Invalid argument
08:38:35 systemd[1]: Stopped FUSE filesystem for LXC.
08:38:36 pveproxy[2698]: received signal TERM
08:38:36 pveproxy[2698]: server closing
08:38:36 pveproxy[2698]: worker 31318 finished
08:38:36 pveproxy[2698]: worker 31319 finished
08:38:36 pveproxy[2698]: worker 31320 finished
08:38:36 pveproxy[2698]: server stopped
08:38:36 pve-ha-crm[19972]: received signal TERM
08:38:36 pve-ha-crm[19972]: server received shutdown request
08:38:36 pve-ha-crm[19972]: server stopped
08:38:36 pvedaemon[12711]: starting vnc proxy UPID:ns505710:000031A7:08A14993:5E0B333C:vncproxy:300:root@pam:
08:38:37 pveproxy[12710]: worker exit
08:38:37 pveproxy[12709]: worker exit
08:38:37 systemd[1]: Stopped PVE API Proxy Server.
08:38:37 systemd[1]: Stopping OpenBSD Secure Shell server...
08:38:37 systemd[1]: Stopped target PVE Storage Target.
08:38:37 systemd[1]: Stopped OpenBSD Secure Shell server.
08:38:37 systemd[1]: Stopped PVE Cluster Ressource Manager Daemon.