[SOLVED] ceph time skew issue after reboot of cluster [it was DNS]

[scyto]

I shut all 3 nodes for maintenance
When they came back up I have the following ceph eerors

    health: HEALTH_WARN
            clock skew detected on mon.pve1-IPv6

On this node and this node only, choronyc shows no sources (simple restart of chrony service did not fix this)

root@pve1:~# chronyc sources
MS Name/IP address         Stratum Poll Reach LastRx Last sample              
===============================================================================
root@pve1:~#

i also note on this node, and this node only that the system clock is not synchronized and the others are

root@pve1:~#  timedatectl status
               Local time: Sun 2023-10-08 14:54:41 PDT
           Universal time: Sun 2023-10-08 21:54:41 UTC
                 RTC time: Sun 2023-10-08 21:54:41
                Time zone: America/Los_Angeles (PDT, -0700)
System clock synchronized: no
              NTP service: active
          RTC in local TZ: no

chronyc makestep did not resolve this issue

what should i try next?

 

[bbgeek17]

You can look at "systemctl status chronyd"
and check the output of "cat /etc/chrony/chrony.conf |egrep -v "^#|^$""

Compare the outputs to working nodes. If you manually configured additional stuff that may be "sourcedir" or "configdif" - check those too.

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[scyto]

@bbgeek17 sorry to waste your time, thanks for the help

(it of course would have helped if chrony had actually said in any of its chronyc output or chronyd logs it couldn't resolve the Debian pool name - it didn't... that seems like a huge miss, unless i am missing something else...)



this one node had slightly different DNS to the other two - it was looking for a container i had not yet started....