Linux system time temporally jumps

[auranext]

Hello,

Last Saturday I saw for the first time a strange system time changing behavior in a PROMOX 5.4 servers
At 05:15 systemd report in syslog a time change to 14 Jun 07:38
HA components force node reboot after system watchdog failed
After reboot system and hwclock are good and cluster is working good
The syslogs do not refer to an ntp synchronization, the time change seems to be produced by a local behavior
The server is a HPE DL360Gen9
At 05:15 system load was safe

someone have experiencing this case ?

Aug 15 05:15:15 CLIB05PVE2 systemd[1]: Started Session 933769 of user root. Aug 15 05:15:15 CLIB05PVE2 systemd[1]: Started Session 933770 of user root. Aug 15 05:15:20 CLIB05PVE2 snmpd[27738]: error on subcontainer 'ia_addr' insert (-1) Aug 15 05:15:20 CLIB05PVE2 snmpd[27738]: error on subcontainer 'ia_addr' insert (-1) Aug 15 05:15:20 CLIB05PVE2 snmpd[27738]: error on subcontainer 'ia_addr' insert (-1) Aug 15 05:15:50 CLIB05PVE2 snmpd[27738]: error on subcontainer 'ia_addr' insert (-1) Aug 15 05:15:50 CLIB05PVE2 snmpd[27738]: error on subcontainer 'ia_addr' insert (-1) Aug 15 05:15:50 CLIB05PVE2 snmpd[27738]: error on subcontainer 'ia_addr' insert (-1) [B]Jun 14 07:38:51 CLIB05PVE2 systemd[1]: Time has been changed Jun 14 07:38:51 CLIB05PVE2 systemd[8360]: Time has been changed[/B] Jun 14 07:38:51 CLIB05PVE2 systemd[1]: apt-daily-upgrade.timer: Adding 13min 37.592597s random time. Jun 14 07:38:51 CLIB05PVE2 systemd[1]: apt-daily.timer: Adding 5h 52min 29.550868s random time. Jun 14 07:38:51 CLIB05PVE2 systemd[1]: pve-daily-update.timer: Adding 1h 39min 7.752208s random time. Jun 14 07:38:52 CLIB05PVE2 systemd[1]: Started Proxmox VE replication runner. Jun 14 07:38:58 CLIB05PVE2 CRON[9581]: (root) CMD (/usr/sbin/pve-ultron-resetme.pl -SR 2175) Jun 14 07:39:00 CLIB05PVE2 postfix/pickup[30559]: 01CE11E4E07: uid=0 from=<root> Jun 14 07:39:00 CLIB05PVE2 postfix/cleanup[18234]: 01CE11E4E07: message-id=<20200614053900.01CE11E4E07@CLIB05PVE2.auracloud.int> Jun 14 07:39:00 CLIB05PVE2 postfix/qmgr[19057]: warning: backward time jump detected -- slewing clock Jun 14 07:39:00 CLIB05PVE2 postfix/qmgr[19057]: 01CE11E4E07: from=<root@CLIB05PVE2.auracloud.int>, size=719, nrcpt=1 (queue active) Jun 14 07:39:00 CLIB05PVE2 CRON[10165]: (root) CMD (/usr/sbin/pve-ultron-resetme.pl -SR 2175) Jun 14 07:39:00 CLIB05PVE2 pmxcfs[5158]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-node/CLIB05PVE2: -1 Jun 14 07:39:00 CLIB05PVE2 pmxcfs[5158]: [status] notice: RRD update error /var/lib/rrdcached/db/pve2-node/CLIB05PVE2: /var/lib/rrdcached/db/pve2-node/CLIB05PVE2: illegal attempt to update using time 1592113140 when last update time is 1597461312 (minimum one second step)

thanks

 

[eoinkim]

@auranext Is this using rsyslog package? If so, what is the version of it? I have a similar issue on Ubuntu server. I am wondering if this is related - https://bugs.launchpad.net/ubuntu/+source/rsyslog/+bug/1429427

 

[auranext]

Hi,

thak you for your reply
I think this is not related
On my understanding rsyslog bug do not change system clock but just put wrong timestamp in syslog data lines

 

[Aristo]

Hi,

a bit late but I wanted to add my experience here.

Some weeks ago I had exactly the same issue:

Jan 13 14:26:13 xxx postfix/smtp[1393282]: connect to xxx[xxx]:25: Connection refused Jan 13 14:26:13 xxx postfix/smtp[1393282]: 3856B41100: to=<xxx>, relay=none, delay=389122, delays=389091/0.02/30/0, dsn=4.4.1, status=deferred (connect to [xxx]:25: Connection refused) Jan 13 14:24:19 xxx systemd[1]: Time has been changed Jan 13 14:24:19 xxx systemd[1]: apt-daily-upgrade.timer: Adding 52min 53.271989s random time. Jan 13 14:24:19 xxx systemd[1]: pve-daily-update.timer: Adding 3h 59min 32.182236s random time. Jan 13 14:24:19 xxx systemd[1]: apt-daily.timer: Adding 3h 17min 52.660783s random time. Jan 13 14:24:22 xxx pmxcfs[6385]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-node/xxx: -1 Jan 13 14:24:22 xxx pmxcfs[6385]: [status] notice: RRD update error /var/lib/rrdcached/db/pve2-node/xxx: /var/lib/rrdcached/db/pve2-node/xxx: illegal attempt to update using time 1610544262 when last update time is 1610544280 (minimum one second step) Jan 13 14:24:22 xxx pmxcfs[6385]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-vm/203: -1 Jan 13 14:24:22 xxx pmxcfs[6385]: [status] notice: RRD update error /var/lib/rrdcached/db/pve2-vm/203: /var/lib/rrdcached/db/pve2-vm/203: illegal attempt to update using time 1610544262 when last update time is 1610544270 (minimum one second step)

Unfortunately it was already after the end of support for 5.4 so the answer to my ticket was more or less: "We don't know what it could be, please update to 6.3"

We did the update some days later and now I am curious when and if it happens again. The trust got some scratches here

Hardware is: HPE DL380 Gen10

 

[auranext]

Hi,

Apparently this is a problem that occurs very rarely.
for several months we monitor CPU time, hw and system clocks, everything is perfectly synchronized, no drift.
It is unlikely but possible that it is a broken NTP server coupled with the fact that the node allows an unjustified time jump.
Time jumps are clearly BAD for running systems
what are the source NTPs configured on your node (before upgrade to 6.3) ?

 

[Aristo]

That's basically our big issue which we are going to solve soon.
At the moment there are two Domain Controllers which are acting as NTP servers BUT they are both on VMs so it's a really bad solution.

We plan for a change already. What is your NTP setup?

 

[spirit]

We use chrony at work , in proxmox host + vms for client, and chrony as servers on physical machines to avoid clock drift.

3 ntp servers, to have quorum.

Also, on your proxmox nodes, disable energy features (dynamic freq, cores shutdown,...), this help a lot to have stable clock.

 

[pvps1]

That's basically our big issue which we are going to solve soon.
At the moment there are two Domain Controllers which are acting as NTP servers BUT they are both on VMs so it's a really bad solution.

We plan for a change already. What is your NTP setup?

ntp on (redundant, non virtualized) firewall. they sync to ntp pool.

internal firewall only allowed ntp source

 

[auranext]

@spirit
will chrony prevent suspicious time jumps if a broken NTP server returns an erroneous date ?

 

[spirit]

@spirit
will chrony prevent suspicious time jumps if a broken NTP server returns an erroneous date ?

if you have 3 ntp server, yes. it'll check diff between 3 nodes, and majority will win.