Clock Skew Detected Among Monitors (Health_Warn)

[Rum_YR]

Hello Proxmox Team,

I am running a 3-node Ceph cluster (ceph1, ceph2, and ceph3). Currently, ceph1 is configured as the primary local time source, while ceph2 and ceph3 use Chrony to sync their time from ceph1.

Recently, ceph3 experienced an unexpected incident and required a reboot to recover. A few days after the reboot, the active ceph1 dashboard started reporting a health warning: "Clock skew detected among monitors (WARNING)" on ceph2 and ceph3.

Here is the status payload from the cluster:

"ceph2": {
"skew": -0.16136153235400391,
"latency": 0.00091213023097808923,
"health": "HEALTH_WARN",
"details": "clock skew 0.161362s > max 0.05s"
}

Current Chrony tracking status on the affected nodes shows that the clock skew is exceeding the 0.05s maximum threshold required by Ceph.

Based on your experience, what is the best practice to resolve this permanently?

Thank you for your advanced support and guidance.

 

[BobhWasatch]

Most NTP servers will say their time is invalid or a high stratum unless they are synchronized to an external source like another NTP server or a GPS clock. One common symptom of this is that the clients of that server will refuse to synchronize. You can check the status of the clock with "timedatectl".

If you do not have NTP or a GPS clock and want to use just the local clock on your master node, please review "man chrony.conf" and read about the the "local" option.

 

[Rum_YR]

Most NTP servers will say their time is invalid or a high stratum unless they are synchronized to an external source like another NTP server or a GPS clock. One common symptom of this is that the clients of that server will refuse to synchronize. You can check the status of the clock with "timedatectl".

If you do not have NTP or a GPS clock and want to use just the local clock on your master node, please review "man chrony.conf" and read about the the "local" option.

Hello BobhWasatch,

thank you for your info,
however, as you mentioned above, mean all clients always rejected not follow as NTP server on master node in local.
So, if we start disable the below option do the client node will synchronizing with master node or not?

# Do NOT serve time to others or act as a fallback clock.
# These belong only on ceph1 (the internal NTP server).
#local stratum 10
#allow 0.0.0.0/0
#maxdistance 16.0

Thank you.

 

[BobhWasatch]

On the master that is serving time to clients you need the "allow" line at least. In addition, IF the master does NOT have NTP available and you want to use the local clock instead, then you also need "local", but the stratum needs to be less than 10.

See here for an example:

https://docs.redhat.com/en/document...rony#setting-up-chrony-in-an-isolated-network

This is standard NTP setup, it is not special to Proxmox. There are many guides on the Google machine.

 

[Rum_YR]

On the master that is serving time to clients you need the "allow" line at least. In addition, IF the master does NOT have NTP available and you want to use the local clock instead, then you also need "local", but the stratum needs to be less than 10.

See here for an example:

https://docs.redhat.com/en/document...rony#setting-up-chrony-in-an-isolated-network

This is standard NTP setup, it is not special to Proxmox. There are many guides on the Google machine.

No, we have the NTP server on the master ceph1 already and we also use NTP client on ceph2, and ceph3 from master, but currently, we got WARNING alerts from Clock Skew on Ceph2 and 3 and we didn't know how to fix it.

 

[woma]

You have two problems. In the first place you have to allow that ceph1 can act as NTP source for others. And secondly ceph1 must "fake" enough time accuracy so that it is not rejected by clients.