Proxmox 6.1-2 with Ceph 14.2.5 - does it still need NTP ?

[brucexx]

Just installed Nautilus from scratch and it's been operational for a day with several VMs (4 nodes), 2 Pools. Because I am just testing I left the default timesyncd and for last 24 hours I did not get any clock skews messages and in the log. The time setting are defaults that come with PVE install: 0.debian.pool.ntp.org 1.debian.pool.ntp.org 2.debian.pool.ntp.org 3.debian.pool.ntp.org.

Am I just lucky that I did not get any clock skew or something changed and we don't need now to install NTP and disable timesyncd ?

Thank you

 

[t.lamprecht]

As the by default enabled "systemd-timesyncd.service" now also updates the time during boot, not only on reboot, you should see no more issues in the systemd version shipped in Promxo VE 6, Even with really bad HW RTCs.
The older version avoid doing this to not break systems without a HW realtime clock (e.g., raspberry), but it was always a strange decision to just not do it at all for everyone, instead of checking if a HW RTC is present and always update once the clock drifted to much.
Anyway, as that behaviour is "fixed" now systemd-timesyncd should be enough for most.

 

[brucexx]

Thank you for explanation.

Is it still best practice for Ceph to use local NTP source and not x.debian.pool.ntp.org that comes with time synced.service ?

Also I remember that the NTP was doing peering between nodes following this post: https://forum.proxmox.com/threads/proxmoxve-ceph-clock-issue.20684/#post-153248 , are the nodes synchronizing time between each other too using systems-timesyncd.service ? ...or maybe there is some other mechanism that is keeping the time synchronization between nodes.

Thx

 

[t.lamprecht]

Is it still best practice for Ceph to use local NTP source and not x.debian.pool.ntp.org that comes with time synced.service ?

What do you mean with local source? A local GPS or other GNSS device, which can feed a local NTP server with very high quality of time measurement? No that's not required.

If you mean a local server which "caches" the synchronization to the NTP.org pool for feeding the whole LAN then it depends.
It can be a good practice to do in general, as it reduces load on the NTP.org time pool servers without, normally, bringing any real disadvantage for local clients. If you have quite a few servers running locally I'd say it'd be a good idea to add a local NTP server with fallback to the public available ones, to avoid issues on maintenance of said NTP "relay" or any other downtime of it, scheduled or unscheduled.

FYI: The "x.debian.pool.ntp.org" are OK, they use the most widely used time-server pool wordlwide. NTP.org just requires distributions to prefix the domain with a distro specific names, so they can better schedule and load balance, and know how much usage the service gets from one specific Distribution. Proxmox still uses the Debian one as the related packages all come directly from Debian. In the future we may want to see if it makes sense to allocate our own NTP.org namespace, but for know this works OK.

 

[brucexx]

yes , I meant local server/servers which cache the NTP.org pool or Debian pool. I happen to have 2 NTP server across our subnets that can serve that purpose. I used them before to provide time for regular NTP while disabling the Systemd's time sync service.

I just added more VMs and still don't see any clock skew, that makes me happy that it works out of the box.

As I add more VMs I will update so far so good.

Thank you

 

[brucexx]

Sadly reporting clock skew with the default time settings. Our ceph cluster is still in testing, so limited production. We got clock skew on 2 out of 4 nodes on the 14th so 4 days after we started the cluster. It lasted only for 29 sec till the Health check cleared but it did happen. Will have to look into the NTP again.

Thank you

 

[AllanM]

I switched to NIST time servers in boulder (not far from where I live), restarted/updated my time sync service on all nodes one after the next. All my clock skew issues went away after that.

I also see the timesync updates in syslog on node reboots now, which also likely helps a lot. Good change!

 

[brucexx]

Tried local network NTP source, with two local NTP servers but got clock skew after 3 days of running. At this point I will be disabling systemd time services and going with regular ntpd as I used to do.

thx