[SOLVED] wrong Time wrong Time synchronisation

maxprox · Jun 17, 2019

Hello,
We have a very strange Problem on two same new Serverinstallations:
The three Times form hwclock, date and NTP are changing more or less, for which I can not see any reason.

Hardware (two servers with the same Supermicro board):
2x a new Supermicro Board X11SCM-F with the last BIOS only 4 weeks old (may 2019).
1x Intel Xeon Intel(R) Xeon(R) E-2124 CPU @ 3.30GHz 1x Intel(R) Xeon(R) E-2136 CPU @ 3.30GHz
with 64 GB ECC RAM and ZFS Raid10 on 4x 4TB SATA disks.

Software:
New installation: pve-manager/5.4-6/ (running kernel: 4.15.18-15-pve)

Have a look at the difference by date and the ntp time. Yesterday we have some minutes differences:

Code:

root@electroprox ~ # date && rdate -4npu ptbtime2.ptb.de
Mon Jun 17 12:10:36 CEST 2019
Mon Jun 17 12:10:53 CEST 2019
root@electroprox ~ # date && rdate -4npu ptbtime2.ptb.de
Mon Jun 17 12:11:21 CEST 2019
Mon Jun 17 12:11:32 CEST 2019
root@electroprox ~ # date && rdate -4npu ptbtime2.ptb.de
Mon Jun 17 12:11:22 CEST 2019
Mon Jun 17 12:11:34 CEST 2019
root@electroprox ~ # date && rdate -4npu ptbtime2.ptb.de
Mon Jun 17 12:11:25 CEST 2019
Mon Jun 17 12:11:37 CEST 2019
root@electroprox ~ # date && rdate -4npu ptbtime2.ptb.de
Mon Jun 17 12:11:29 CEST 2019
Mon Jun 17 12:11:44 CEST 2019
root@electroprox ~ # date && rdate -4npu ptbtime2.ptb.de
Mon Jun 17 12:11:30 CEST 2019
Mon Jun 17 12:11:46 CEST 2019
...

root@electroprox ~ # date && rdate -4npu ptbtime2.ptb.de
Mon Jun 17 13:23:20 CEST 2019
Mon Jun 17 13:23:20 CEST 2019
root@electroprox ~ # date && rdate -4npu ptbtime2.ptb.de
Mon Jun 17 13:23:33 CEST 2019
Mon Jun 17 13:23:48 CEST 2019

What is here goin on?

We worked with a new Installation and „chrony” as ntp-client, than we purge chrony and activate the build in systemd-timesyncd.service.

Any Ideas for this strange behaviour?
regards,
maxprox

aaron · Jun 17, 2019

In the output you included the time difference stays at around 12 to 17 seconds. But then it is only about a minute overall.

Code:

root@electroprox ~ # date && rdate -4npu ptbtime2.ptb.de
Mon Jun 17 13:23:20 CEST 2019
Mon Jun 17 13:23:20 CEST 2019
root@electroprox ~ # date && rdate -4npu ptbtime2.ptb.de
Mon Jun 17 13:23:33 CEST 2019
Mon Jun 17 13:23:48 CEST 2019

I assume you synced the time before you ran the second part and it drifted 15 seconds right away?
How is it over a longer period? Differences up to several minutes like you said?
Do the logs of " chrony" or now "systemd-timesyncd" give any hints that something might be wrong?

Code:

journalctl -u systemd-timesyncd

Out of curiosity how does the output of "hwclock" compare to "date" and "rdate"?

maxprox · Jun 17, 2019

Hello Aaron,
thanks for your reply.
it may also be important:
- our Firewall has a "deny all" rule, for time sync only the default udp port 123 is open.
- the time synchronization in Proxmox is only for itself

We changed from chrony to the time synchronization like described in the wiki and get the output like shown in the wiki:

Code:

journalctl -u systemd-timesyncd
-- Logs begin at Mon 2019-06-17 12:06:57 CEST, end at Mon 2019-06-17 22:43:00 CEST. --
Jun 17 14:05:58 electroprox systemd[1]: Starting Network Time Synchronization...
Jun 17 14:05:58 electroprox systemd[1]: Started Network Time Synchronization.
Jun 17 12:06:57 electroprox systemd-timesyncd[1164]: Synchronized to time server 144.76.60.190:123 (2.de.pool.ntp.org).
Jun 17 22:40:13 electroprox systemd[1]: Stopping Network Time Synchronization...
Jun 17 22:40:13 electroprox systemd[1]: Stopped Network Time Synchronization.
Jun 17 22:40:13 electroprox systemd[1]: Starting Network Time Synchronization...
Jun 17 22:40:13 electroprox systemd[1]: Started Network Time Synchronization.
Jun 17 22:40:35 electroprox systemd-timesyncd[9816]: Synchronized to time server 91.202.42.83:123 (0.de.pool.ntp.org).

But with: 'timedatectl status' the line "NTP synchronized:" say "no"
Is this an error? And how do I fix that?:

Code:

root@electroprox ~ # timedatectl status
      Local time: Mon 2019-06-17 22:41:45 CEST
  Universal time: Mon 2019-06-17 20:41:45 UTC
        RTC time: Mon 2019-06-17 20:41:50
       Time zone: Europe/Berlin (CEST, +0200)
 Network time on: yes
NTP synchronized: no
 RTC in local TZ: no

For now the three Times differs around 10 to 30 seconds, since we changed frome chrony to systemd-timesyncd.
(Yesterday, the day one after the first and new Installation, between 10 seconds and more than 10 minutes)

Code:

root@electroprox ~ # hwclock -r &&  date && rdate -4npuv ptbtime2.ptb.de
2019-06-17 22:33:23.136741+0200
Mon Jun 17 22:33:15 CEST 2019
Mon Jun 17 22:33:24 CEST 2019
rdate: adjust local clock by 8.382084 seconds
root@electroprox ~ # hwclock -r &&  date && rdate -4npuv ptbtime2.ptb.de
2019-06-17 22:33:51.386750+0200
Mon Jun 17 22:33:28 CEST 2019
Mon Jun 17 22:33:52 CEST 2019
rdate: adjust local clock by 23.300647 seconds
...
root@electroprox ~ # date &&  hwclock -r && rdate -4npuv ptbtime2.ptb.de
Tue Jun 18 00:21:54 CEST 2019
2019-06-18 00:22:21.949514+0200
Tue Jun 18 00:22:22 CEST 2019
rdate: adjust local clock by 26.973072 seconds
...
root@electroprox ~ # date &&  hwclock -r && rdate -4npuv ptbtime2.ptb.de
Tue Jun 18 00:44:38 CEST 2019
2019-06-18 00:44:48.339619+0200
Tue Jun 18 00:44:49 CEST 2019
rdate: adjust local clock by 10.072192 seconds
## 9h later:
Tue Jun 18 09:15:14 CEST 2019 
2019-06-18 09:15:32.480303+0200 
Tue Jun 18 09:15:33 CEST 2019 
rdate: adjust local clock by 17.944355 seconds

What I do not understand is the changeover from synchronized Time to not synchronized in both ways?
The other question for me is the dependence from this three Times on the RTC on the Supermicro Board: is it possible that the Problem is caused by a Mainboard RTC bug?
Or are this times independently by the RTC at the moment the system is online?

>> Do the logs of " chrony" or now "systemd-timesyncd" give any hints that something might be wrong?
yesterday with chrony we got the following error in the syslog:

Code:

 Can't synchronise: no selectable sources
Jun 17 03:03:46 electroprox chronyd[28066]: Selected source 192.53.103.104
Jun 17 03:03:46 electroprox chronyd[28066]: System clock wrong by 277.091317 seconds, adjustment started
Jun 17 03:03:49 electroprox chronyd[28066]: Can't synchronise: no selectable sources
Jun 17 03:04:51 electroprox chronyd[28066]: Selected source 192.53.103.104
Jun 17 03:04:51 electroprox chronyd[28066]: System clock wrong by 71.913281 seconds, adjustment started
Jun 17 03:05:56 electroprox chronyd[28066]: Can't synchronise: no selectable sources
Jun 17 03:30:44 electroprox chronyd[28066]: Source 192.53.103.104 replaced with 2001:638:610:be01::104
...
Jun 17 04:49:38 electroprox chronyd[28066]: Selected source 134.76.249.102
Jun 17 04:49:38 electroprox chronyd[28066]: System clock wrong by 671.097973 seconds, adjustment started
Jun 17 04:50:43 electroprox chronyd[28066]: System clock wrong by 67.275050 seconds, adjustment started
Jun 17 04:51:48 electroprox chronyd[28066]: System clock wrong by 68.105236 seconds, adjustment started
Jun 17 04:52:52 electroprox chronyd[28066]: System clock wrong by 66.676199 seconds, adjustment started
Jun 17 04:53:57 electroprox chronyd[28066]: Can't synchronise: no selectable sources
Jun 17 05:01:19 electroprox chronyd[28066]: Source 141.30.228.4 replaced with 88.198.12.78
Jun 17 05:04:33 electroprox chronyd[28066]: Selected source 88.198.12.78
Jun 17 05:04:33 electroprox chronyd[28066]: System clock wrong by 719.442923 seconds, adjustment started
Jun 17 05:04:43 electroprox chronyd[28066]: Can't synchronise: no selectable sources

With the now running timesyncd I can not see any error:

Code:

cat /var/log/syslog | grep -E -i '(ntp|timesyncd)'
Jun 17 12:06:57 electroprox systemd-timesyncd[1164]: Synchronized to time server 144.76.60.190:123 (2.de.pool.ntp.org).
Jun 17 22:40:35 electroprox systemd-timesyncd[9816]: Synchronized to time server 91.202.42.83:123 (0.de.pool.ntp.org).

regards,
maxprox

aaron · Jun 18, 2019

Well, this is quite curious.
What really strikes me is that the HW clock is in perfect sync with the NTP time while the system clock is off.

As far as I know the clock of the running system is not coupled to the HW clock, only on boot to get the initial time.

Regarding NTP sync; I don't know if timesyncd will log errors if there are some connectivity issues. Since chrony did report some problems it might be worth taking a look at the NTP traffic itself.

Can you please run

Code:

tcpdump udp port 123

and restart "systemd-timesyncd" in another shell so it will do a first sync?
This way we should see if it does get an answer from the timeserver or if the firewall is interfering.

For more detailed output, even the NTP data you can run

Code:

tcpdump udp port 123 -t -vvv

You might need to install tcpdump first

Code:

apt install tcpdump

maxprox · Jun 18, 2019

Thank you for pointing that out. However, it's not because the connections all work without problems.

Code:

root@electroprox ~ # tcpdump udp port 123 -t 
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on vmbr0, link-type EN10MB (Ethernet), capture size 262144 bytes
IP electroprox.maidhof.lan.38457 > dc.peter.llan.ntp: NTPv4, Client, length 48
IP dc.peter.llan.ntp > electroprox.maidhof.lan.38457: NTPv4, Server, length 48
IP electroprox.maidhof.lan.52991 > dc.peter.llan.ntp: NTPv4, Client, length 48
IP dc.peter.llan.ntp > electroprox.maidhof.lan.52991: NTPv4, Server, length 48

We have also looked into the logs of our NTP server and the answer is also good.

aaron · Jun 18, 2019

Your NTP servers changed in your answers which makes it a bit tough for me trying to pin point where the cause is.

What is your current config of timesyncd?
What is the log output of timesyncd when you restart it? Please use the following syntax is it will also show the systemd message when it stops and starts the service.

Code:

journalctl -u systemd-timesyncd

maxprox · Jun 18, 2019

Code:

root@electroprox ~ # cat /etc/systemd/timesyncd.conf
...
[Time]
NTP=172.21.8.200  ## our local NTP Server (Debian Samba4 DC)
FallbackNTP=0.de.pool.ntp.org 1.de.pool.ntp.org 2.de.pool.ntp.org 3.de.pool.ntp.org
##NTP=0.de.pool.ntp.org 1.de.pool.ntp.org 2.de.pool.ntp.org 3.de.pool.ntp.org
#FallbackNTP=0.debian.pool.ntp.org 1.debian.pool.ntp.org 2.debian.pool.ntp.org 3.debian.pool.ntp.org

Code:

root@electroprox ~ # journalctl -u systemd-timesyncd
-- Logs begin at Mon 2019-06-17 12:06:57 CEST, end at Tue 2019-06-18 14:16:00 CEST. --
Jun 17 14:05:58 electroprox systemd[1]: Starting Network Time Synchronization...
Jun 17 14:05:58 electroprox systemd[1]: Started Network Time Synchronization.
Jun 17 12:06:57 electroprox systemd-timesyncd[1164]: Synchronized to time server 144.76.60.190:123 (2.de.pool.ntp.org).
Jun 17 22:40:13 electroprox systemd[1]: Stopping Network Time Synchronization...
Jun 17 22:40:13 electroprox systemd[1]: Stopped Network Time Synchronization.
Jun 17 22:40:13 electroprox systemd[1]: Starting Network Time Synchronization...
Jun 17 22:40:13 electroprox systemd[1]: Started Network Time Synchronization.
Jun 17 22:40:35 electroprox systemd-timesyncd[9816]: Synchronized to time server 91.202.42.83:123 (0.de.pool.ntp.org).
Jun 18 01:05:26 electroprox systemd[1]: Stopping Network Time Synchronization...
Jun 18 01:05:26 electroprox systemd[1]: Stopped Network Time Synchronization.
Jun 18 01:05:26 electroprox systemd[1]: Starting Network Time Synchronization...
Jun 18 01:05:26 electroprox systemd[1]: Started Network Time Synchronization.
Jun 18 01:05:37 electroprox systemd-timesyncd[7630]: Synchronized to time server 88.198.12.78:123 (0.de.pool.ntp.org).
Jun 18 10:23:01 electroprox systemd[1]: Stopping Network Time Synchronization...
Jun 18 10:23:01 electroprox systemd[1]: Stopped Network Time Synchronization.
Jun 18 10:23:01 electroprox systemd[1]: Starting Network Time Synchronization...
Jun 18 10:23:01 electroprox systemd[1]: Started Network Time Synchronization.
Jun 18 10:23:17 electroprox systemd-timesyncd[15754]: Synchronized to time server 172.21.8.200:123 (172.21.8.200).
Jun 18 10:23:43 electroprox systemd[1]: Stopping Network Time Synchronization...
Jun 18 10:23:43 electroprox systemd[1]: Stopped Network Time Synchronization.
Jun 18 10:23:43 electroprox systemd[1]: Starting Network Time Synchronization...
Jun 18 10:23:43 electroprox systemd[1]: Started Network Time Synchronization.
Jun 18 10:24:05 electroprox systemd-timesyncd[16190]: Synchronized to time server 172.21.8.200:123 (172.21.8.200).
Jun 18 10:25:43 electroprox systemd[1]: Stopping Network Time Synchronization...
Jun 18 10:25:43 electroprox systemd[1]: Stopped Network Time Synchronization.
Jun 18 10:25:43 electroprox systemd[1]: Starting Network Time Synchronization...
Jun 18 10:25:43 electroprox systemd[1]: Started Network Time Synchronization.
Jun 18 10:26:10 electroprox systemd-timesyncd[17150]: Synchronized to time server 172.21.8.200:123 (172.21.8.200).
Jun 18 10:26:14 electroprox systemd[1]: Stopping Network Time Synchronization...
Jun 18 10:26:14 electroprox systemd[1]: Stopped Network Time Synchronization.
Jun 18 10:26:14 electroprox systemd[1]: Starting Network Time Synchronization...
Jun 18 10:26:14 electroprox systemd[1]: Started Network Time Synchronization.
Jun 18 10:26:16 electroprox systemd-timesyncd[17194]: Synchronized to time server 172.21.8.200:123 (172.21.8.200).
Jun 18 10:31:50 electroprox systemd[1]: Stopping Network Time Synchronization...
Jun 18 10:31:50 electroprox systemd[1]: Stopped Network Time Synchronization.
Jun 18 10:31:50 electroprox systemd[1]: Starting Network Time Synchronization...
Jun 18 10:31:50 electroprox systemd[1]: Started Network Time Synchronization.
Jun 18 10:31:59 electroprox systemd-timesyncd[19581]: Synchronized to time server 172.21.8.200:123 (172.21.8.200).
Jun 18 10:39:44 electroprox systemd[1]: Stopping Network Time Synchronization...
Jun 18 10:39:44 electroprox systemd[1]: Stopped Network Time Synchronization.
Jun 18 10:39:44 electroprox systemd[1]: Starting Network Time Synchronization...
Jun 18 10:39:44 electroprox systemd[1]: Started Network Time Synchronization.
Jun 18 10:39:51 electroprox systemd-timesyncd[23100]: Synchronized to time server 172.21.8.200:123 (172.21.8.200).
Jun 18 10:50:03 electroprox systemd[1]: Stopping Network Time Synchronization...
Jun 18 10:50:03 electroprox systemd[1]: Stopped Network Time Synchronization.
Jun 18 10:50:03 electroprox systemd[1]: Starting Network Time Synchronization...
Jun 18 10:50:03 electroprox systemd[1]: Started Network Time Synchronization.
Jun 18 10:50:17 electroprox systemd-timesyncd[27709]: Synchronized to time server 172.21.8.200:123 (172.21.8.200).
Jun 18 10:51:34 electroprox systemd[1]: Stopping Network Time Synchronization...
lines 1-50

aaron · Jun 18, 2019

Thank you for the detailed log.
What does "timedatectl status" show right after you restart timesyncd and after the "Synchronized to time server..." line is logged?

maxprox · Jun 18, 2019

aaron said:
Thank you for the detailed log.
What does "timedatectl status" show right after you restart timesyncd and after the "Synchronized to time server..." line is logged?

Restart, without an answer:

Code:

root@electroprox ~ # systemctl restart systemd-timesyncd.service

timedatectl status:

Code:

root@electroprox ~ # timedatectl status
      Local time: Tue 2019-06-18 15:48:51 CEST
  Universal time: Tue 2019-06-18 13:48:51 UTC
        RTC time: Tue 2019-06-18 13:48:58
       Time zone: Europe/Berlin (CEST, +0200)
 Network time on: yes
NTP synchronized: no
 RTC in local TZ: no

systemctl status systemd-timesyncd

Code:

root@electroprox ~ # systemctl status systemd-timesyncd
● systemd-timesyncd.service - Network Time Synchronization
   Loaded: loaded (/lib/systemd/system/systemd-timesyncd.service; enabled; vendor preset: enabled)
  Drop-In: /lib/systemd/system/systemd-timesyncd.service.d
           └─disable-with-time-daemon.conf
   Active: active (running) since Tue 2019-06-18 15:48:33 CEST; 31s ago
     Docs: man:systemd-timesyncd.service(8)
 Main PID: 21816 (systemd-timesyn)
   Status: "Synchronized to time server 172.21.8.200:123 (172.21.8.200)."
    Tasks: 2 (limit: 4915)
   Memory: 980.0K
      CPU: 12ms
   CGroup: /system.slice/systemd-timesyncd.service
           └─21816 /lib/systemd/systemd-timesyncd

Jun 18 15:48:33 electroprox systemd[1]: Starting Network Time Synchronization...
Jun 18 15:48:33 electroprox systemd[1]: Started Network Time Synchronization.
Jun 18 15:48:45 electroprox systemd-timesyncd[21816]: Synchronized to time server 172.21.8.200:123 (172.21.8.200).

Here I cannot see any fault. Never mind if we work with a local NTP Server or one from the Internet.

In parallel I have open a "case" by Supermicro, with the question focused to the Mainboard RTC and BIOS,
but this does not helped for now.

aaron · Jun 18, 2019

Well, I am pretty much at a loss now. There is one last thing that might work. But since I cannot replicate your situation I cannot say for sure.

Can you please run the following command:

Code:

timedatectl set-ntp true

maxprox · Jun 18, 2019

Hi Aaron,
I thank you for following this (fucking) issue!

in the past, this command we try more than once with:
timedatectl set-ntp true
timedatectl set-ntp on
timedatectl set-ntp 1
but it only change the line:
System clock synchronized: no to System clock synchronized: yes
an not the line:
NTP service: no (to yes or active)

On one of the two servers, now we have installed for testing a clean Debian Buster (bare metal),
there we get the same curiosity result

Code:

root@debianTEST:~# date &&  hwclock -r && rdate -4npuv ptbtime2.ptb.de
Di 18. Jun 16:23:09 CEST 2019
2019-06-18 16:24:09.104787+02:00
Tue Jun 18 16:24:12 CEST 2019
rdate: adjust local clock by 61.621529 seconds
root@debianTEST:~#

Here the second last line by the command timedatectl status we get "active" under proxmox we get here "no".
But whatever the result is the same.
On the clean Debian Buster (TEST) the status and its output is different, but I think it has nothing to do with our problem:

Code:

root@debianTEST:~# timedatectl set-ntp true
root@debianTEST:~# timedatectl status
               Local time: Di 2019-06-18 16:33:47 CEST
           Universal time: Di 2019-06-18 14:33:47 UTC
                 RTC time: Di 2019-06-18 14:33:45
                Time zone: Europe/Berlin (CEST, +0200)
System clock synchronized: yes
              NTP service: active
          RTC in local TZ: no
root@debianTEST:~# timedatectl set-ntp false
root@debianTEST:~# timedatectl status
               Local time: Di 2019-06-18 16:34:01 CEST
           Universal time: Di 2019-06-18 14:34:01 UTC
                 RTC time: Di 2019-06-18 14:34:32
                Time zone: Europe/Berlin (CEST, +0200)
System clock synchronized: yes
              NTP service: inactive
          RTC in local TZ: no

aaron · Jun 18, 2019

I am a bit confused. Did the "timedatectl set-ntp true" did help or not because "System clock synchronized" is only shown on Buster as far as I can tell. Did the "NTP synchronized" line change to yes on the server with Proxmox on it?

The output of "timedatectl status" changed in Buster as I found out as well. There is the new option "timedatectl show" which does seem to show some of the infos that used to be shown by "status" in Stretch.

"timedatectl status" on a PVE 5.4 host (stretch):

Code:

# timedatectl status
      Local time: Tue 2019-06-18 16:50:04 CEST
  Universal time: Tue 2019-06-18 14:50:04 UTC
        RTC time: Tue 2019-06-18 14:50:04
       Time zone: Europe/Vienna (CEST, +0200)
 Network time on: yes
NTP synchronized: yes
 RTC in local TZ: no

"timedatectl status" on a Buster host:

Code:

# timedatectl status
               Local time: Tue 2019-06-18 16:37:08 CEST
           Universal time: Tue 2019-06-18 14:37:08 UTC
                 RTC time: Tue 2019-06-18 14:37:08
                Time zone: Europe/Vienna (CEST, +0200)
System clock synchronized: yes
              NTP service: active
          RTC in local TZ: no

"timedatectl show" on Buster:

Code:

# timedatectl show  
Timezone=Europe/Vienna
LocalRTC=no
CanNTP=yes
NTP=yes
NTPSynchronized=yes
TimeUSec=Tue 2019-06-18 16:51:50 CEST
RTCTimeUSec=Tue 2019-06-18 16:51:50 CEST

maxprox · Jun 18, 2019

On our Debian Buster Test System we get:

Code:

root@debianTEST:~# timedatectl show
root@debianTEST:~# timedatectl show
Timezone=Europe/Berlin
LocalRTC=no
CanNTP=yes
NTP=yes
NTPSynchronized=yes
TimeUSec=Tue 2019-06-18 17:16:14 CEST
RTCTimeUSec=Tue 2019-06-18 17:05:19 CEST

everything like it should, but wiht more then the same problem, here we have minutes different:

Code:

root@debianTEST:~# systemctl restart systemd-timesyncd.service
root@debianTEST:~# date &&  hwclock -r && rdate -4npuv ptbtime2.ptb.de
Di 18. Jun 17:16:01 CEST 2019
2019-06-18 17:04:37.761224+02:00
Tue Jun 18 17:16:04 CEST 2019
rdate: adjust local clock by 1.905862 seconds

On the Proxmox 5.4 System we get this shown in the last posts,
the timedatectl status:
NTP synchronized: no
here stay always, and with
systemctl status systemd-timesyncd
we get "
"Synchronized to time server 172.21.8.200:123 (172.21.8.200)."

But on this both servers time is switching around,
In the same LAN we have two more Proxmox 5.4-6 Installations (since many month) and there we never have any problem with the time. With date && hwclock -r && rdate we maximum get one second different.

maxprox · Jun 18, 2019

aaron said:
I am a bit confused. Did the "timedatectl set-ntp true" did help or not because "System clock synchronized" is only shown on Buster as far as I can tell. Did the "NTP synchronized" line change to yes on the server with Proxmox on it?

No,
not on the Proxmox installaiton. The command "timedatectl set-ntp true" only change the output line before: "
from "Network time on: no" to Network time on: yes.
But the output line "NTP synchronized: no" stay always / ever:

Code:

root@electroprox ~ # timedatectl status
      Local time: Tue 2019-06-18 16:32:45 CEST
  Universal time: Tue 2019-06-18 14:32:45 UTC
        RTC time: Tue 2019-06-18 14:33:14
       Time zone: Europe/Berlin (CEST, +0200)
 Network time on: yes
NTP synchronized: no
 RTC in local TZ: no

root@electroprox ~ # timedatectl set-ntp false

root@electroprox ~ # timedatectl status
      Local time: Tue 2019-06-18 16:32:51 CEST
  Universal time: Tue 2019-06-18 14:32:51 UTC
        RTC time: Tue 2019-06-18 14:33:29
       Time zone: Europe/Berlin (CEST, +0200)
 Network time on: no
NTP synchronized: no
 RTC in local TZ: no

and vice versa.

maxprox · Jun 18, 2019

again I have a look at the Firewall rules and LOGs - I see nothing that help.
try several NTP Server in the /etc/systemd/timesyncd.conf
and have a look at the tcpdump (represented below).
Restart the timesyncd many times and so on,
but with the problem nothing changed ...

Code:

root@electroprox ~ # tcpdump udp port 123 -t -v
IP (tos 0x0, ttl 1, id 14172, offset 0, flags [none], proto UDP (17), length 76)
    172.21.8.20.ntp > 172.21.8.255.ntp: NTPv4, length 48
        Broadcast, Leap indicator:  (0), Stratum 2 (secondary reference), poll 7 (128s), precision -7
        Root Delay: 0.049942, Root dispersion: 0.007736, Reference-ID: chronos2.gwdg.de
          Reference Timestamp:  3769863337.629999956 (2019/06/18 18:15:37)
          Originator Timestamp: 0.000000000
          Receive Timestamp:    0.000000000
          Transmit Timestamp:   3769863337.629999956 (2019/06/18 18:15:37)
            Originator - Receive Timestamp:  0.000000000
            Originator - Transmit Timestamp: 3769863337.629999956 (2019/06/18 18:15:37)
IP (tos 0x10, ttl 64, id 40116, offset 0, flags [DF], proto UDP (17), length 76)
    electroprox.maidhof.lan.38296 > ptbtime2.ptb.de.ntp: NTPv4, length 48
        Client, Leap indicator:  (0), Stratum 0 (unspecified), poll 0 (1s), precision 0
        Root Delay: 0.000000, Root dispersion: 0.000000, Reference-ID: (unspec)
          Reference Timestamp:  0.000000000
          Originator Timestamp: 0.000000000
          Receive Timestamp:    0.000000000
          Transmit Timestamp:   3769863340.071097020 (2019/06/18 18:15:40)
            Originator - Receive Timestamp:  0.000000000
            Originator - Transmit Timestamp: 3769863340.071097020 (2019/06/18 18:15:40)
IP (tos 0x0, ttl 50, id 36095, offset 0, flags [DF], proto UDP (17), length 76)
    ptbtime2.ptb.de.ntp > electroprox.maidhof.lan.38296: NTPv4, length 48
        Server, Leap indicator:  (0), Stratum 1 (primary reference), poll 2 (4s), precision -18
        Root Delay: 0.000000, Root dispersion: 0.000228, Reference-ID: PTB^@
          Reference Timestamp:  3769863356.045554646 (2019/06/18 18:15:56)
          Originator Timestamp: 3769863340.071097020 (2019/06/18 18:15:40)
          Receive Timestamp:    3769863362.941940260 (2019/06/18 18:16:02)
          Transmit Timestamp:   3769863362.942211401 (2019/06/18 18:16:02)
            Originator - Receive Timestamp:  +22.870843240
            Originator - Transmit Timestamp: +22.871114381
IP (tos 0x0, ttl 64, id 42975, offset 0, flags [DF], proto UDP (17), length 76)
    electroprox.maidhof.lan.34423 > ptbtime2.ptb.de.ntp: NTPv4, length 48
        Client, Leap indicator:  (0), Stratum 0 (unspecified), poll 0 (1s), precision 0
        Root Delay: 0.000000, Root dispersion: 0.000000, Reference-ID: (unspec)
          Reference Timestamp:  0.000000000
          Originator Timestamp: 0.000000000
          Receive Timestamp:    0.000000000
          Transmit Timestamp:   2187754936.538524215 (2105/06/06 14:10:32)
            Originator - Receive Timestamp:  0.000000000
            Originator - Transmit Timestamp: 2187754936.538524215 (2105/06/06 14:10:32)
IP (tos 0x0, ttl 50, id 52501, offset 0, flags [DF], proto UDP (17), length 76)
    ptbtime2.ptb.de.ntp > electroprox.maidhof.lan.34423: NTPv4, length 48
        Server, Leap indicator:  (0), Stratum 1 (primary reference), poll 2 (4s), precision -18
        Root Delay: 0.000000, Root dispersion: 0.000198, Reference-ID: PTB^@
          Reference Timestamp:  3769863380.045580076 (2019/06/18 18:16:20)
          Originator Timestamp: 2187754936.538524215 (2105/06/06 14:10:32)
          Receive Timestamp:    3769863386.044879844 (2019/06/18 18:16:26)
          Transmit Timestamp:   3769863386.045272511 (2019/06/18 18:16:26)
            Originator - Receive Timestamp:  +1582108449.506355628
            Originator - Transmit Timestamp: +1582108449.506748296

maxprox · Jun 18, 2019

Next Test:
Now I Install with the same Proxmox 5.4 USB Boot stick on an old Backup-System-PC
....
everything is running as it shuld, the time synchronization works with the same (copy & past) /etc/systemd/timesyncd.conf
Now the output line: "NTP synchronized: yes" shows "YES"
The clock adjust is by 0,0..... seconds.

Now I will wipe the Proxmox 5.4 ZFS Raid10 complete and will install the same Proxmox .5.4 one more time to spy out if it is a Supermicro Mainboard (hardware) issue or not.

udo · Jun 18, 2019

Hi,
do you have any settings in the bios depends on clock source?
How are the performance/powersave settings?

Any difference with full performance mode?

Udo

maxprox · Jun 18, 2019

okay, Now I install the same Proxmox 5.4 on the concernd Supermcro X11SCM-F Server,
(but not on the wiped 4x4TB ZFSRaid10, on a separate SATA SSD)
After the fresh installation I copy & past again the same: /etc/systemd/timesyncd.conf
On this fresh Proxmox: proxmox-ve: 5.4-1 (running kernel: 4.15.18-12-pve)
I changed nothing, but:

Code:

apt update
apt install rdate
root@pve:~# nano /etc/systemd/timesyncd.conf  ## copy & past NTP config
##  then:
root@pve:~# service systemd-timesyncd restart

And I get the same issue as before

Code:

root@pve:~# timedatectl status
      Local time: Tue 2019-06-18 20:56:36 CEST
  Universal time: Tue 2019-06-18 18:56:36 UTC
        RTC time: Tue 2019-06-18 18:56:51
       Time zone: Europe/Berlin (CEST, +0200)
 Network time on: yes
NTP synchronized: no
 RTC in local TZ: no
root@pve:~# date &&  hwclock -r && rdate -4npuv ptbtime2.ptb.de
Tue Jun 18 20:57:31 CEST 2019
2019-06-18 20:57:52.762019+0200
Tue Jun 18 20:57:53 CEST 2019
rdate: adjust local clock by 20.934942 seconds
root@pve:~# date &&  hwclock -r && rdate -4npuv ptbtime2.ptb.de
Tue Jun 18 20:57:57 CEST 2019
2019-06-18 20:57:58.589932+0200
Tue Jun 18 20:57:59 CEST 2019
rdate: adjust local clock by 1.356120 seconds
root@pve:~# date &&  hwclock -r && rdate -4npuv ptbtime2.ptb.de
Tue Jun 18 20:58:09 CEST 2019
2019-06-18 20:58:15.605490+0200
Tue Jun 18 20:58:16 CEST 2019
rdate: adjust local clock by 6.532718 seconds

maxprox · Jun 18, 2019

udo said:
Hi,
do you have any settings in the bios depends on clock source?
How are the performance/powersave settings?

Any difference with full performance mode?

Udo

Hi Udo,

No I don't think so,
In the BIOS the only thing I changed -after an BIOS and IPMI Update- was the Boot order.
Do you have a special / concrete setting for witch I have to look for in the BIOS?
regards,
maxprox

Stoiko Ivanov · Jun 19, 2019

Hi,

On a hunch I found the following links for a potential explanation:
* https://unix.stackexchange.com/questions/377621/clock-instability-in-debian-9-stretch

Could you please post the output of:
* `dmesg`
* `journalctl -b` (after a fresh boot)

Additionally - does changing the kernel-commanline as explained in the article above mitigate the problem?

You could also try to disable hpet in the bios - seems the option might be called "Enable multimedia timer" (https://www.supermicro.com/support/faqs/faq.cfm?faq=5732)

Also as @udo suggested - try changing the BIOS settings affecting the CPU-clock scale-down (usually found in the BIOS either at CPU-settings or PowerSaving Settings) - https://www.supermicro.com/support/faqs/faq.cfm?faq=21555

Hope this helps!

[SOLVED] wrong Time wrong Time synchronisation

Renowned Member

Proxmox Staff Member

Renowned Member

Proxmox Staff Member

Renowned Member

Proxmox Staff Member

Renowned Member

Proxmox Staff Member

Renowned Member

Proxmox Staff Member

Renowned Member

Proxmox Staff Member

Renowned Member

Renowned Member

Renowned Member

Renowned Member

Distinguished Member

Renowned Member

Renowned Member

Proxmox Staff Member