PVE Cluster time sync questions

[rdfamily]

I have a 3 node cluster and modified the chrony.conf file on all the PVEs to use the "x.us.pool.ntp.org" servers. I restarted the chronyd dameon on all and still have a time difference of 3 mins on the third PVE from the 1st & 2nd. how can i verify that the npt servers are used. I did a (journalctl --sice -1h -u chrony) but it does not show the ntp server used. I also ran the (timedatectl set-ntp true) command.
root@perko3:~# journalctl --since -1h -u chrony
Jan 12 15:04:05 perko3 chronyd[2502]: chronyd exiting
Jan 12 15:04:05 perko3 systemd[1]: Stopping chrony.service - chrony, an NTP client/server...
Jan 12 15:04:05 perko3 systemd[1]: chrony.service: Deactivated successfully.
Jan 12 15:04:05 perko3 systemd[1]: Stopped chrony.service - chrony, an NTP client/server.
Jan 12 15:04:05 perko3 systemd[1]: chrony.service: Consumed 1.865s CPU time.
Jan 12 15:04:05 perko3 systemd[1]: Starting chrony.service - chrony, an NTP client/server...
Jan 12 15:04:05 perko3 chronyd[2293790]: chronyd version 4.3 starting (+CMDMON +NTP +REFCLOCK +RTC +PRIVDR>
Jan 12 15:04:05 perko3 chronyd[2293790]: Frequency -9.341 +/- 6.872 ppm read from /var/lib/chrony/chrony.d>
Jan 12 15:04:05 perko3 chronyd[2293790]: Using right/UTC timezone to obtain leap second data
Jan 12 15:04:05 perko3 chronyd[2293790]: Loaded seccomp filter (level 1)
Jan 12 15:04:05 perko3 systemd[1]: Started chrony.service - chrony, an NTP client/server.
Jan 12 15:06:41 perko3 chronyd[2293790]: chronyd exiting
Jan 12 15:06:41 perko3 systemd[1]: Stopping chrony.service - chrony, an NTP client/server...
Jan 12 15:06:41 perko3 systemd[1]: chrony.service: Deactivated successfully.
Jan 12 15:06:41 perko3 systemd[1]: Stopped chrony.service - chrony, an NTP client/server.
Jan 12 15:06:41 perko3 systemd[1]: Starting chrony.service - chrony, an NTP client/server...
Jan 12 15:06:41 perko3 chronyd[2298113]: chronyd version 4.3 starting (+CMDMON +NTP +REFCLOCK +RTC +PRIVDR>
Jan 12 15:06:41 perko3 chronyd[2298113]: Frequency -9.341 +/- 6.872 ppm read from /var/lib/chrony/chrony.d>
Jan 12 15:06:41 perko3 chronyd[2298113]: Using right/UTC timezone to obtain leap second data
Jan 12 15:06:41 perko3 chronyd[2298113]: Loaded seccomp filter (level 1)
Jan 12 15:06:41 perko3 systemd[1]: Started chrony.service - chrony, an NTP client/server.
Jan 12 15:10:59 perko3 chronyd[2298113]: chronyd exiting
Jan 12 15:10:59 perko3 systemd[1]: Stopping chrony.service - chrony, an NTP client/server...
Jan 12 15:10:59 perko3 systemd[1]: chrony.service: Deactivated successfully.
Jan 12 15:10:59 perko3 systemd[1]: Stopped chrony.service - chrony, an NTP client/server.
Jan 12 15:10:59 perko3 systemd[1]: Starting chrony.service - chrony, an NTP client/server...
Jan 12 15:10:59 perko3 chronyd[2304827]: chronyd version 4.3 starting (+CMDMON +NTP +REFCLOCK +RTC +PRIVDR>
Jan 12 15:10:59 perko3 chronyd[2304827]: Frequency -9.341 +/- 6.872 ppm read from /var/lib/chrony/chrony.d>
Jan 12 15:10:59 perko3 chronyd[2304827]: Using right/UTC timezone to obtain leap second data
Jan 12 15:10:59 perko3 chronyd[2304827]: Loaded seccomp filter (level 1)
Jan 12 15:10:59 perko3 systemd[1]: Started chrony.service - chrony, an NTP client/server.

So what can I do to get the time on all 3 servers synced.

 

[BobhWasatch]

The command "chronyc sources" will show what sources it is synced to, or if it is not synced.

Example output:

~$ chronyc sources

MS Name/IP address         Stratum Poll Reach LastRx Last sample            
===============================================================================
^+ 2600:3c02::f03c:94ff:fe5>     3  10   377   422  -2499us[-2499us] +/-   80ms
^+ ntp-62b.lbl.gov               2  10   377   21m    +49ms[  +49ms] +/-  108ms
^+ 2607:f1c0:f054:5e00::2        2  10   377   462  +2689us[+2689us] +/-  138ms
^* s2-a.time.mci1.us.rozint>     2  10   377   881  +5174us[+4861us] +/-   46ms

The MS column shows the sync status for each server, where ^* is the one it is currently using. Poll column shows the time between polls as a power of 2 (e.g. these are all 2^10 or 1024). The Reach column is octal with one bit set or cleared for the last 8 polls. So if that is not 377 it means chrony is having problems reaching that server. See "man chronyc" for more info on this output.

Most likely you either have a typo in the config file or a networking problem (firewall? gateway? DNS?) that is preventing chrony from reaching the servers.

 

[rdfamily]

I do have the 3 PVE servers themselves isolated on a local LAN in a 192.168.1.x while the VM's are on a 10.0.x.x LAN with internet access through a firewall appliance. how would I add access to the NTP servers? do I need to setup a VM to act as local NTP client/server with access to both networks. What do you recommend.