server spontaneously reboots

jmckee

Renowned Member
Oct 19, 2013
17
0
66
Sever rebooted during the night. I am wondering if anyone can explain this log?
Code:
un 24 00:00:02 proxmox1 pvescheduler[1417502]: <root@pam> starting task UPID:proxmox1:0015A121:0D62F8AD:64969472:vzdump:103:root@pam:
Jun 24 00:00:03 proxmox1 systemd[1]: Starting Rotate log files...
Jun 24 00:00:03 proxmox1 systemd[1]: Starting Daily man-db regeneration...
Jun 24 00:00:04 proxmox1 systemd[1]: Reloading PVE API Proxy Server.
Jun 24 00:00:05 proxmox1 systemd[1]: man-db.service: Succeeded.
Jun 24 00:00:05 proxmox1 systemd[1]: Finished Daily man-db regeneration.
Jun 24 00:00:06 proxmox1 pvescheduler[1417505]: INFO: starting new backup job: vzdump 103 --mailnotification failure --storage newbacks --mode snapshot --mailto (redacted)  --compress zstd --node proxmox1 --quiet 1
Jun 24 00:00:07 proxmox1 pvescheduler[1417505]: INFO: Starting Backup of VM 103 (qemu)
Jun 24 00:00:10 proxmox1 pveproxy[1417522]: send HUP to 1635
Jun 24 00:00:10 proxmox1 pveproxy[1635]: received signal HUP
Jun 24 00:00:10 proxmox1 pveproxy[1635]: server closing
Jun 24 00:00:10 proxmox1 pveproxy[1635]: server shutdown (restart)
Jun 24 00:00:10 proxmox1 systemd[1]: Reloaded PVE API Proxy Server.
Jun 24 00:00:10 proxmox1 systemd[1]: Reloading PVE SPICE Proxy Server.
Jun 24 00:00:10 proxmox1 spiceproxy[1417581]: send HUP to 1642
Jun 24 00:00:13 proxmox1 systemd[1]: Reloaded PVE SPICE Proxy Server.
Jun 24 00:00:13 proxmox1 pveproxy[1635]: restarting server
Jun 24 00:00:13 proxmox1 systemd[1]: Stopping Proxmox VE firewall logger...
Jun 24 00:00:19 proxmox1 pveproxy[1635]: starting 3 worker(s)
Jun 24 00:00:19 proxmox1 systemd[1]: pvefw-logger.service: Succeeded.
Jun 24 00:00:19 proxmox1 pveproxy[1635]: worker 1417629 started
Jun 24 00:00:19 proxmox1 systemd[1]: Stopped Proxmox VE firewall logger.
Jun 24 00:00:19 proxmox1 pveproxy[1635]: worker 1417630 started
Jun 24 00:00:20 proxmox1 systemd[1]: pvefw-logger.service: Consumed 6.274s CPU time.
Jun 24 00:00:20 proxmox1 pveproxy[1635]: worker 1417631 started
Jun 24 00:00:20 proxmox1 systemd[1]: Starting Proxmox VE firewall logger...
Jun 24 00:00:20 proxmox1 pvefw-logger[1039788]: received terminate request (signal)
Jun 24 00:00:20 proxmox1 systemd[1]: Started Proxmox VE firewall logger.
Jun 24 00:00:20 proxmox1 pvefw-logger[1039788]: stopping pvefw logger
Jun 24 00:00:20 proxmox1 systemd[1]: logrotate.service: Succeeded.
Jun 24 00:00:20 proxmox1 spiceproxy[1642]: received signal HUP
Jun 24 00:00:21 proxmox1 systemd[1]: Finished Rotate log files.
Jun 24 00:00:21 proxmox1 spiceproxy[1642]: server closing
Jun 24 00:00:21 proxmox1 pvefw-logger[1417635]: starting pvefw logger
Jun 24 00:00:21 proxmox1 spiceproxy[1642]: server shutdown (restart)
Jun 24 00:00:21 proxmox1 pveproxy[1039800]: worker exit
Jun 24 00:00:21 proxmox1 pveproxy[1357822]: worker exit
Jun 24 00:00:22 proxmox1 pveproxy[1039799]: worker exit
Jun 24 00:00:22 proxmox1 pveproxy[1635]: worker 1357822 finished
Jun 24 00:00:22 proxmox1 pveproxy[1635]: worker 1039799 finished
Jun 24 00:00:22 proxmox1 pveproxy[1635]: worker 1039800 finished
Jun 24 00:00:37 proxmox1 spiceproxy[1642]: restarting server
Jun 24 00:00:39 proxmox1 spiceproxy[1642]: starting 1 worker(s)
Jun 24 00:00:42 proxmox1 spiceproxy[1642]: worker 1417738 started
Jun 24 00:00:42 proxmox1 pvestatd[1522]: status update time (11.053 seconds)
Jun 24 00:00:45 proxmox1 spiceproxy[1039812]: worker exit
Jun 24 00:00:48 proxmox1 spiceproxy[1642]: worker 1039812 finished
Jun 24 00:00:48 proxmox1 pvestatd[1522]: status update time (8.531 seconds)
Jun 24 00:01:05 proxmox1 pvestatd[1522]: status update time (5.399 seconds)
Jun 24 00:01:15 proxmox1 pvestatd[1522]: status update time (6.125 seconds)
Jun 24 00:01:28 proxmox1 pvestatd[1522]: status update time (9.662 seconds)
Jun 24 00:01:59 proxmox1 pvestatd[1522]: status update time (10.411 seconds)
Jun 24 00:02:09 proxmox1 pve-firewall[1520]: firewall update time (5.515 seconds)
Jun 24 00:02:18 proxmox1 pvestatd[1522]: status update time (19.014 seconds)
Jun 24 00:02:40 proxmox1 pvestatd[1522]: status update time (11.633 seconds)
Jun 24 00:02:46 proxmox1 pvestatd[1522]: status update time (5.732 seconds)
Jun 24 00:03:48 proxmox1 pvestatd[1522]: status update time (37.749 seconds)
Jun 24 00:04:13 proxmox1 pve-ha-lrm[1644]: loop take too long (59 seconds)
Jun 24 00:04:18 proxmox1 pvestatd[1522]: status update time (29.826 seconds)
Jun 24 00:04:59 proxmox1 pve-ha-crm[1634]: loop take too long (55 seconds)
Jun 24 00:04:59 proxmox1 pve-firewall[1520]: firewall update time (5.171 seconds)
Jun 24 00:05:00 proxmox1 pve-ha-lrm[1644]: loop take too long (47 seconds)
Jun 24 00:05:04 proxmox1 pvescheduler[1418558]: replication: cfs-lock 'file-replication_cfg' error: got lock request timeout
Jun 24 00:05:07 proxmox1 pvestatd[1522]: status update time (47.442 seconds)
Jun 24 00:05:32 proxmox1 pvestatd[1522]: status update time (6.933 seconds)
Jun 24 00:05:53 proxmox1 pve-firewall[1520]: firewall update time (9.652 seconds)
Jun 24 00:06:07 proxmox1 pvestatd[1522]: status update time (31.811 seconds)
Jun 24 00:07:04 proxmox1 pvestatd[1522]: status update time (37.689 seconds)
Jun 24 00:07:38 proxmox1 watchdog-mux:
-- Reboot --
 
Jun 24 00:05:00 proxmox1 pve-ha-lrm[1644]: loop take too long (47 seconds)

seem that you have loose quorum, and HA have restart your server for failover.

how many nodes do you have in your cluster ? are you sure that the corosync network link is not saturated ? (backup for example)
 
ah ok. I suspected am issue with the cluster network but wasn't too sure. I think there may be a hardware issue in there. Thanks for pointing me in the direction.
(I have 3 nodes with dedicated cluster network )
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!