Chrony vs systemd-timesyncd on PVE 7.2

[jsabater]

Hey, everyone!

SOME CONTEXT

I have a production 3-node Proxmox 7.2 cluster which has been running very fine for a few months now. Each node was installed on top of a Debian 11 Bullseye via the procedure explained in the Install Proxmox VE on Debian 11 Bullseye wiki page. It only runs LinuX Containers (LXC, no VMs) with Debian 11 and Ubuntu 20.04 (and one Debian 9 because... life, heh).

A few days ago I realised that a number of errors had been happening for almost two weeks, all related to RRDC. Just a few examples would be:

pmxcfs[727]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-storage/proxmox1/local: -1
pmxcfs[727]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-storage/proxmox1/pbs1: -1
pmxcfs[729]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-vm/105

After digging in these forums for a couple of hours, I managed to sort the problem out via a restart of systemd-timesyncd, pvedaemon, pveproxy, pvestatd and pve-cluster [1] in each node and a few reboots, but without having to delete the RRDC cache which, if I understood correctly, would be the relatively harmless last resort.

[1] Later I found out that restarting systemd-timesyncd and pvestatd should have been enough.

THE QUESTION

Among many other pages [2], one that I visited was the Time Synchronization wiki page. This is the one that led me to the systemd-timesyncd restarts, but it also led me to think why this scary event had happened in the first place and whether I should uninstall systemd-timesyncd and install Chrony instead. From the wiki page:

As of Proxmox VE 7, chrony is used as the default NTP daemon, while Proxmox VE 6 uses systemd-timesyncd.

Also:

If you upgrade your system to Proxmox VE 7, it is recommended that you manually install either chrony, ntp or openntpd.

This Chrony vs systemd-timesync thread on Stack Exchange does a pretty good job IMHO describing the scenario in which systemd-timesyncd should suffice, which kind of suits my context as I use the Hetzner's NTP servers:

~ # cat /etc/systemd/timesyncd.conf.d/hetzner.conf
[Time]
NTP=ntp1.hetzner.de ntp2.hetzner.com ntp3.hetzner.net

So my question would be: Am I fine running systemd-timesyncd in my three hosts/nodes, or should I switch to Chrony? If so, would you think it should be added to the Install Proxmox VE on Debian 11 Bullseye wiki page as a "must do"? Or should I be fine with systemd-timesyncd and I should wait and see if this happens again, as it should not?

Thanks in advance for your feedback.

[2] https://forum.proxmox.com/threads/rrdc-time-spam.56011/, https://forum.proxmox.com/threads/solved-rrdc-update-error.16256/ and, mainly, https://forum.proxmox.com/threads/notice-rrdc-rrd-update-error.103473/

 

[LnxBil]

A lot of text, but what was the problem in the end? Was it time offset? Normally the implementation does not matter, what matters is that you monitor your machines and counteract if something is not working correctly, despite the used time synchronisation technique. I used all of them over the years and did not get any problems with either one of them while continously monitoring the system.

 

[jsabater]

Yes, apparently the problem was time offset.

Anyway, how do you monitor whether your servers are correcly synchronised? I was looking forward to finding some information about it in the Datacenter: Cluster menu option of the WebGUI, or perhaps in the Summary, but there is none.

 

[PigLover]

Problems of time sync usually occur when one or more of your hosts selects the “wrong” sync source, or there is a problem with that sync source. System-timesyncd uses SNTP it only tracks a single source and if the hosts in your cluster using systemd-timesyncd lock onto different sources and something happens to one of those sources then things get goofy. Full NTP implementations like Chrony or ntpc generally don’t get hammered by this because the one outlier will get dropped when evaluating the pool responses. But even with these tools stuff happens.

Bottom line, for most cluster operations and Ceph, it is much more important that all of your hosts AGREE on what time it is than having them precisely know the correct time. So its likely much more important that you get your sources right than whether you choose systemd-timesyncd vs chrony vs ntpd.

Suggested best practice - establish a time source for your cluster. A host (or very small group of hosts) running an NTP server/client like ntpd or chrony and set it up to sync to an appropriate outside source pool. If you do set up more than one reference source for redundancy take pains to make sure they will agree on what time it is (declare one of them primary and set it to sync to an eternal pool and set the other(s) to use your primary as first choice and then fall back to the same pool the primary is using) . I use the NTP service embedded in my primary router for the primary time source, but that might not always be a good choice depending on your config (e.g., might not work in hosted environments). Then set up each machine in your cluster to always use this NTP server for their primary (or even only) time source. It really doesn’t matter if they use systemd-timesyncd or chrony or ntpc to do this as they will all be syncing to a single (they all get the same answers) local (very low latency) time source.

If you do this all your time drift worries will melt away into the night…

 

[LnxBil]

Anyway, how do you monitor whether your servers are correcly synchronised?

e.g. Nagios/Icinga with local agents like this:

root@proxmox6 ~ > /usr/lib/nagios/plugins/check_ntp_time -H pool.ntp.org
NTP OK: Offset -0.001314550638 secs|offset=-0.001315s;60.000000;120.000000;

Depending on the used (S|)NTP implementation, you may need more checks.