Ceph clock screw

[Canadian Pinguin]

Hi,
We are using ceph with encrypted OSD. Every once in a while we have the OSD nodes doing this:
Nov 22 17:21:10 sof6pve4 ceph-osd[9500]: 2021-11-22T17:21:10.344+0200 7f7aa463f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way
too early (before 2021-11-22T16:21:10.349781+0200).
When this happens all nodes lose access to storage for a little while.
What I don't get is we are using NTP on each node with the same server.
What could be the issue here?

 

[aaron]

Which PVE version and which NTP client?
Is the NTP server located nearby?

 

[Canadian Pinguin]

Hi aaron,
proxmox-ve: 7.0-2 (running kernel: 5.11.22-7-pve) pve-manager: 7.0-14+1 (running version: 7.0-14+1/08975a4c)
and for the NTP I'm using timedatectl config :

timedatectl
               Local time: Tue 2021-11-23 15:47:21 EET
           Universal time: Tue 2021-11-23 13:47:21 UTC
                 RTC time: Tue 2021-11-23 13:47:21
                Time zone: Europe/Sofia (EET, +0200)
System clock synchronized: yes
              NTP service: active
          RTC in local TZ: no

And i'm using the europe pool ntp.
This is my /etc/systemd/timesyncd.conf :

[Time]
NTP=0.europe.pool.ntp.org 1.europe.pool.ntp.org 2.europe.pool.ntp.org 3.europe.pool.ntp.org

 

[aaron]

Okay, so it's a cluster that was upgraded from 6.4?

The default NTP client on fresh PVE 7 installs has changed and is chrony. If you upgraded from 6.4 you will have to manually install it. https://pve.proxmox.com/wiki/Roadmap#Proxmox_VE_7.0 (Search for "Time Synchronization")
The reason why is because timesyncd would sync the time at boot, but once the nodes were up for longer, the times would drift and could result in the problems you have.

Install chrony and then the situation should get much better. timesyncd will not start once another ntp client is present.

 

[Canadian Pinguin]

Okay, so it's a cluster that was upgraded from 6.4?

No, it's an install made from Debian. But It was Debian 10 at the time as 11 wasn't available so I guess it's the same. I needed to encrypt the local disk so it was the only way.

Ok, I will try chrony.
Thank you!