PVE Cluster time sync questions

R

rdfamily

New Member
Apr 18, 2025
25
2
3
I have a 3 node cluster and modified the chrony.conf file on all the PVEs to use the "x.us.pool.ntp.org" servers. I restarted the chronyd dameon on all and still have a time difference of 3 mins on the third PVE from the 1st & 2nd. how can i verify that the npt servers are used. I did a (journalctl --sice -1h -u chrony) but it does not show the ntp server used. I also ran the (timedatectl set-ntp true) command.
root@perko3:~# journalctl --since -1h -u chrony
Jan 12 15:04:05 perko3 chronyd[2502]: chronyd exiting
Jan 12 15:04:05 perko3 systemd[1]: Stopping chrony.service - chrony, an NTP client/server...
Jan 12 15:04:05 perko3 systemd[1]: chrony.service: Deactivated successfully.
Jan 12 15:04:05 perko3 systemd[1]: Stopped chrony.service - chrony, an NTP client/server.
Jan 12 15:04:05 perko3 systemd[1]: chrony.service: Consumed 1.865s CPU time.
Jan 12 15:04:05 perko3 systemd[1]: Starting chrony.service - chrony, an NTP client/server...
Jan 12 15:04:05 perko3 chronyd[2293790]: chronyd version 4.3 starting (+CMDMON +NTP +REFCLOCK +RTC +PRIVDR>
Jan 12 15:04:05 perko3 chronyd[2293790]: Frequency -9.341 +/- 6.872 ppm read from /var/lib/chrony/chrony.d>
Jan 12 15:04:05 perko3 chronyd[2293790]: Using right/UTC timezone to obtain leap second data
Jan 12 15:04:05 perko3 chronyd[2293790]: Loaded seccomp filter (level 1)
Jan 12 15:04:05 perko3 systemd[1]: Started chrony.service - chrony, an NTP client/server.
Jan 12 15:06:41 perko3 chronyd[2293790]: chronyd exiting
Jan 12 15:06:41 perko3 systemd[1]: Stopping chrony.service - chrony, an NTP client/server...
Jan 12 15:06:41 perko3 systemd[1]: chrony.service: Deactivated successfully.
Jan 12 15:06:41 perko3 systemd[1]: Stopped chrony.service - chrony, an NTP client/server.
Jan 12 15:06:41 perko3 systemd[1]: Starting chrony.service - chrony, an NTP client/server...
Jan 12 15:06:41 perko3 chronyd[2298113]: chronyd version 4.3 starting (+CMDMON +NTP +REFCLOCK +RTC +PRIVDR>
Jan 12 15:06:41 perko3 chronyd[2298113]: Frequency -9.341 +/- 6.872 ppm read from /var/lib/chrony/chrony.d>
Jan 12 15:06:41 perko3 chronyd[2298113]: Using right/UTC timezone to obtain leap second data
Jan 12 15:06:41 perko3 chronyd[2298113]: Loaded seccomp filter (level 1)
Jan 12 15:06:41 perko3 systemd[1]: Started chrony.service - chrony, an NTP client/server.
Jan 12 15:10:59 perko3 chronyd[2298113]: chronyd exiting
Jan 12 15:10:59 perko3 systemd[1]: Stopping chrony.service - chrony, an NTP client/server...
Jan 12 15:10:59 perko3 systemd[1]: chrony.service: Deactivated successfully.
Jan 12 15:10:59 perko3 systemd[1]: Stopped chrony.service - chrony, an NTP client/server.
Jan 12 15:10:59 perko3 systemd[1]: Starting chrony.service - chrony, an NTP client/server...
Jan 12 15:10:59 perko3 chronyd[2304827]: chronyd version 4.3 starting (+CMDMON +NTP +REFCLOCK +RTC +PRIVDR>
Jan 12 15:10:59 perko3 chronyd[2304827]: Frequency -9.341 +/- 6.872 ppm read from /var/lib/chrony/chrony.d>
Jan 12 15:10:59 perko3 chronyd[2304827]: Using right/UTC timezone to obtain leap second data
Jan 12 15:10:59 perko3 chronyd[2304827]: Loaded seccomp filter (level 1)
Jan 12 15:10:59 perko3 systemd[1]: Started chrony.service - chrony, an NTP client/server.

So what can I do to get the time on all 3 servers synced.
 
The command "chronyc sources" will show what sources it is synced to, or if it is not synced.

Example output:

Code: 
~$ chronyc sources

MS Name/IP address         Stratum Poll Reach LastRx Last sample            
===============================================================================
^+ 2600:3c02::f03c:94ff:fe5>     3  10   377   422  -2499us[-2499us] +/-   80ms
^+ ntp-62b.lbl.gov               2  10   377   21m    +49ms[  +49ms] +/-  108ms
^+ 2607:f1c0:f054:5e00::2        2  10   377   462  +2689us[+2689us] +/-  138ms
^* s2-a.time.mci1.us.rozint>     2  10   377   881  +5174us[+4861us] +/-   46ms

The MS column shows the sync status for each server, where ^* is the one it is currently using. Poll column shows the time between polls as a power of 2 (e.g. these are all 2^10 or 1024). The Reach column is octal with one bit set or cleared for the last 8 polls. So if that is not 377 it means chrony is having problems reaching that server. See "man chronyc" for more info on this output.

Most likely you either have a typo in the config file or a networking problem (firewall? gateway? DNS?) that is preventing chrony from reaching the servers.
 
Last edited:
I do have the 3 PVE servers themselves isolated on a local LAN in a 192.168.1.x while the VM's are on a 10.0.x.x LAN with internet access through a firewall appliance. how would I add access to the NTP servers? do I need to setup a VM to act as local NTP client/server with access to both networks. What do you recommend.
 
Hi, I have setup my WIN2008 AD vm as an NTP server since it has internet access. how do I configure PVE to use it as the NTP server for all three PVEs.
 
You should just be able to use a "server" line instead of "pool" in /etc/chrony/chrony.conf. It might be helpful to read the chrony and chrony.conf man pages...

server <ip-of-the-vm> iburst

The thing is, though, that this setup seems kind of brittle to me. Your cluster time will depend on that VM. This could cause issues with initial sync when restarting the cluster because it is hard to control the boot order of the machines. Plus unless this VM has HA enabled, it will go down along with the server it is on, which could be an issue during failure scenarios.

It would be better, in my opinion, to set up a small Linux VM to do NTP. You can do an Alpine Linux VM in a couple hundred MB of RAM and disk. Security harden that VM and make it the dual-homed one. That is a much smaller attack surface than a Windows AD.

But I wouldn't do that either. What I would do is allow limited Internet access for the 10.0.xx LAN via the router/firewall. Make the firewall your NTP server for both networks and use the firewall to control access. The main reason I would do this is to make it simpler to update the hypervisor. I think keeping up with updates is important enough to consider in your network design.

I have worked in secure environments where the network was air-gapped. In that case the admins had local mirrors set up for updates, which they had to keep up-to-date manually. The NTP solution there was a GPS receiver with NTP support, like the ones Brandywine sells. That is the kind of thing you need to do if you really want to isolate that 10.1.x.x network.
 
  • Like
Reactions: Johannes S and leesteken
Is there no other (high-available) server in your LAN that can run a time service and "mandate" what time it is? Other local servers without internet probably also need some kind of time synchronization between each other. Then you can make all your PVE nodes sync with that server and they will agree on what time it is, even if it might be a little off with the rest of the world.

EDIT: Maybe there is a router that can serve the time?
 
Last edited:
  • Like
Reactions: Johannes S
You must log in or register to reply here.
Top Bottom