chronyd issues

[ilia987]

I have a cluster of around 20 nodes, Proxmox 8.0.4 +ceph (on 4 of them)

all have the same configuration of chronyd. and most of the time everything works.
recently (in the past month) i started to have issues with time sync. usually it comes after a power failure, some servers are out of synced from 2-20 hours behind

rebooting does not help, ipmi\bios time is correct.

running

chronyc -m 'burst 3/3' 'makestep 0.1 3'

fix it for short time and then it jumps back in time
chronyc sources -v

.-- Source mode  '^' = server, '=' = peer, '#' = local clock.
 / .- Source state '*' = current best, '+' = combined, '-' = not combined,
| /             'x' = may be in error, '~' = too variable, '?' = unusable.
||                                                 .- xxxx [ yyyy ] +/- zzzz
||      Reachability register (octal) -.           |  xxxx = adjusted offset,
||      Log2(Polling interval) --.      |          |  yyyy = measured offset,
||                                \     |          |  zzzz = estimated error.
||                                 |    |           \
MS Name/IP address         Stratum Poll Reach LastRx Last sample               
===============================================================================
^? 45.150.108.5                  0   6   377     -     +0ns[   +0ns] +/-    0ns
^? 114.30.92.30                  2   6   377    25  -57340s[-57340s] +/-  236ms
^? time.cloudflare.com           3   6   377    35  -57340s[-57340s] +/-   30ms
^? 202.4.186.250                 3   6   377    37  -57340s[-57340s] +/-  209ms
^? x.ns.gin.ntt.net              0   6   377     -     +0ns[   +0ns] +/-    0ns
^? 061239100017.ctinets.com      1   6   377    39  -57340s[-57340s] +/-  124ms
^? twtpe2-ntp-002.aaplimg.c>     1   6   277    48  -57340s[-57340s] +/-  162ms
^? time.cloudflare.com           3   6   377    54  -57340s[-57340s] +/-   30ms
^? 139.199.215.251               0   6   377     -     +0ns[   +0ns] +/-    0ns
^? ec2-51-16-235-6.il-centr>     3   6   373     1  -57340s[-57340s] +/- 9122us
^? ec2-51-17-20-53.il-centr>     3   6   377     7  -57340s[-57340s] +/- 7996us
^? 236.99.217.185.interhost>     3   6   377     9  -57340s[-57340s] +/-   41ms
^? ec2-51-16-77-36.il-centr>     3   6   377    13  -57340s[-57340s] +/- 7286us
^? 158.207.interhost.co.il       4   6   337    13  -57340s[-57340s] +/-   35ms
^? 129.159.140.221               3   6   377    18  -57340s[-57340s] +/- 7513us

chronyc tracking

Reference ID    : 00000000 ()
Stratum         : 0
Ref time (UTC)  : Thu Jan 01 00:00:00 1970
System time     : 0.000000000 seconds slow of NTP time
Last offset     : -57340.101562500 seconds
RMS offset      : 51097.007812500 seconds
Frequency       : 19.303 ppm fast
Residual freq   : +0.000 ppm
Skew            : 0.000 ppm
Root delay      : 1.000000000 seconds
Root dispersion : 1.000000000 seconds
Update interval : 0.0 seconds
Leap status     : Not synchronised

 

[bbgeek17]

Take a look at MS column next to each one of your sources. Note how it starts with ^?.
Looking at the "man chronyc":

               •   ? indicates a source which is not considered to be selectable for synchronisation for other reasons
 (e.g. unreachable, not synchronised, or does not have enough measurements).

https://serverfault.com/questions/1047184/chrony-time-server-not-synchronised

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[ilia987]

Take a look at MS column next to each one of your sources. Note how it starts with ^?.
Looking at the "man chronyc":

               •   ? indicates a source which is not considered to be selectable for synchronisation for other reasons
 (e.g. unreachable, not synchronised, or does not have enough measurements).

https://serverfault.com/questions/1047184/chrony-time-server-not-synchronised

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

i came across it , did not help.


But i think i found a reason, still not sure if it caused it.
all Proxmox nodes have chrony services and should sync by it.
i have two more dedicated servers DC, DC backup
and the backup had a time issue, after it fixed most of the issues resolved

But i dont understand why each proxmox node take the time from the DC (only connection to the DC address is set at DNS ip) and why it conflicts with chronyd