I had a strange thing happen to my cluster last night. While watching a movie off a Plex VM it all of a sudden it stopped. I look at the cluster and ALL 8 of my VM's are stopped. The hosts all look fine and didn't report anything bad as far as I could tell from the GUI.
This morning I was looking at the log files to see find the reason for it and I noticed that an hour after my Plex stopped, two of my host nodes had rebooted themselves in less than a minute of each other. The uptime on each nodes is about 12 hours from the time I'm writing this. My third node stayed up and it's uptime shows many days.
Interestingly all the VM's that I restarted last night are still running and they have the EXACT same uptime as the hosts do. I find that odd because I would think that if the the VM's would have a start time of many minutes later IF they all came down and up again. I don't have a HA configured to bring up any nodes. I do have a backup that's supposed to to run on a VM every Friday at 3am.
The logs files don't show anything before the time of the reboot. It's about 5 hours of no entry then all of a sudden the machine starts. Is there some automated process that cleans the hosts up and resets the uptime or something that I don't know about?
This morning I was looking at the log files to see find the reason for it and I noticed that an hour after my Plex stopped, two of my host nodes had rebooted themselves in less than a minute of each other. The uptime on each nodes is about 12 hours from the time I'm writing this. My third node stayed up and it's uptime shows many days.
Interestingly all the VM's that I restarted last night are still running and they have the EXACT same uptime as the hosts do. I find that odd because I would think that if the the VM's would have a start time of many minutes later IF they all came down and up again. I don't have a HA configured to bring up any nodes. I do have a backup that's supposed to to run on a VM every Friday at 3am.
The logs files don't show anything before the time of the reboot. It's about 5 hours of no entry then all of a sudden the machine starts. Is there some automated process that cleans the hosts up and resets the uptime or something that I don't know about?
Code:
Feb 17 01:10:01 artemis kernel: [82445.598445] XFS (dm-4): Ending clean mount
Feb 17 06:25:09 artemis rsyslogd: [origin software="rsyslogd" swVersion="8.4.2" x-pid="1465" x-info="http://www.rsyslog.com"] rsyslogd
was HUPed
Feb 17 21:16:23 artemis rsyslogd: [origin software="rsyslogd" swVersion="8.4.2" x-pid="1412" x-info="http://www.rsyslog.com"] start