Dell R7525 System clock was stepped

[adresner]

I have a Dell R7525. It operates perfectly normal with Windows. I can go into the System Bios, set the time, set the time in iDrac, and the system time in Windows and everything is fine.

However, when I install Proxmox, the system time changes to 7 time zones earlier than the one I am in. I have been working on this for 2 weeks. You name it, I tried it. The problem only happens when I install Proxmox. Tried to install earlier version but get stuck at

Just a note, last troubleshooting was installing with no internet connection.. so everything is cool til I connect to the network.. then instantly I time travel.

Dec 03 00:04:35 hq-pv1 pvedaemon[4080]: <root@pam> successful auth for user 'root@pam'
Dec 03 00:04:39 hq-pv1 chronyd[3747]: Selected source 171.103.24.86 (2.debian.pool.ntp.org)
Dec 02 17:04:50 hq-pv1 chronyd[3747]: System clock wrong by -25189.002426 seconds
Dec 02 17:04:50 hq-pv1 systemd-journald[2435]: Time jumped backwards, rotating.
Dec 02 17:04:50 hq-pv1 chronyd[3747]: System clock was stepped by -25189.002426 seconds
Dec 02 17:04:50 hq-pv1 chronyd[3747]: System clock TAI offset set to 37 seconds

Why is this going on? How to resolve it please?

 

[adresner]

I see that there is some configuration issue because this Dell Server is compatible with Linux, which is looking for UTC time?

Here is some extra output:

root@hq-pv1:/etc/chrony# date
Mon Dec 2 08:34:11 PM +07 2024

root@hq-pv1:/etc/chrony# timedatectl status
Local time: Mon 2024-12-02 20:34:26 +07
Universal time: Mon 2024-12-02 13:34:26 UTC
RTC time: Mon 2024-12-02 13:34:26
Time zone: Asia/Bangkok (+07, +0700)
System clock synchronized: yes
NTP service: active
RTC in local TZ: no
root@hq-pv1:/etc/chrony#

root@hq-pv1:/etc/chrony# journalctl --since -1h -u chrony
Dec 02 23:53:35 hq-pv1 systemd[1]: Starting chrony.service - chrony, an NTP client/server...
Dec 02 23:53:35 hq-pv1 chronyd[3841]: chronyd version 4.3 starting (+CMDMON +NTP +REFCLOCK +RTC +PRIVDROP +SCFILTER +SIGND +ASYNCDNS +NTS +SECHASH +IPV6 -DEBU>
Dec 02 23:53:35 hq-pv1 chronyd[3841]: Using right/UTC timezone to obtain leap second data
Dec 02 23:53:35 hq-pv1 chronyd[3841]: Loaded seccomp filter (level 1)
Dec 02 23:53:35 hq-pv1 systemd[1]: Started chrony.service - chrony, an NTP client/server.
Dec 02 23:54:05 hq-pv1 chronyd[3841]: chronyd exiting
Dec 02 23:54:05 hq-pv1 systemd[1]: Stopping chrony.service - chrony, an NTP client/server...
Dec 02 23:54:05 hq-pv1 systemd[1]: chrony.service: Deactivated successfully.
Dec 02 23:54:05 hq-pv1 systemd[1]: Stopped chrony.service - chrony, an NTP client/server.
-- Boot be235ae5901f4a53b310f6a5015460fd --
Dec 02 23:58:13 hq-pv1 systemd[1]: Starting chrony.service - chrony, an NTP client/server...
Dec 02 23:58:13 hq-pv1 chronyd[3711]: chronyd version 4.3 starting (+CMDMON +NTP +REFCLOCK +RTC +PRIVDROP +SCFILTER +SIGND +ASYNCDNS +NTS +SECHASH +IPV6 -DEBU>
Dec 02 23:58:13 hq-pv1 chronyd[3711]: Frequency 0.000 +/- 1000000.000 ppm read from /var/lib/chrony/chrony.drift
Dec 02 23:58:13 hq-pv1 chronyd[3711]: Using right/UTC timezone to obtain leap second data
Dec 02 23:58:13 hq-pv1 chronyd[3711]: Loaded seccomp filter (level 1)
Dec 02 23:58:13 hq-pv1 systemd[1]: Started chrony.service - chrony, an NTP client/server.
Dec 02 23:58:58 hq-pv1 chronyd[3711]: Selected source 147.50.17.138 (2.debian.pool.ntp.org)
-- Boot b5c5e0d97f124493a1e7997e4d3af70c --
Dec 03 00:04:00 hq-pv1 systemd[1]: Starting chrony.service - chrony, an NTP client/server...
Dec 03 00:04:00 hq-pv1 chronyd[3747]: chronyd version 4.3 starting (+CMDMON +NTP +REFCLOCK +RTC +PRIVDROP +SCFILTER +SIGND +ASYNCDNS +NTS +SECHASH +IPV6 -DEBU>
Dec 03 00:04:00 hq-pv1 chronyd[3747]: Frequency 0.000 +/- 1000000.000 ppm read from /var/lib/chrony/chrony.drift
Dec 03 00:04:00 hq-pv1 chronyd[3747]: Using right/UTC timezone to obtain leap second data
Dec 03 00:04:00 hq-pv1 chronyd[3747]: Loaded seccomp filter (level 1)
Dec 03 00:04:00 hq-pv1 systemd[1]: Started chrony.service - chrony, an NTP client/server.
Dec 03 00:04:39 hq-pv1 chronyd[3747]: Selected source 171.103.24.86 (2.debian.pool.ntp.org)
-- Boot f6b972c2acdd4020bd71ad70cd5ef496 --
Dec 03 02:18:59 hq-pv1 systemd[1]: Starting chrony.service - chrony, an NTP client/server...
Dec 03 02:18:59 hq-pv1 chronyd[3728]: chronyd version 4.3 starting (+CMDMON +NTP +REFCLOCK +RTC +PRIVDROP +SCFILTER +SIGND +ASYNCDNS +NTS +SECHASH +IPV6 -DEBU>
Dec 03 02:18:59 hq-pv1 chronyd[3728]: Frequency -7.752 +/- 0.145 ppm read from /var/lib/chrony/chrony.drift
Dec 03 02:18:59 hq-pv1 chronyd[3728]: Using right/UTC timezone to obtain leap second data
Dec 03 02:18:59 hq-pv1 chronyd[3728]: Loaded seccomp filter (level 1)
Dec 03 02:18:59 hq-pv1 systemd[1]: Started chrony.service - chrony, an NTP client/server.
Dec 03 02:19:38 hq-pv1 chronyd[3728]: Selected source 202.12.97.45 (2.debian.pool.ntp.org)
-- Boot a3003248fe7f483ea7568c3c76ae111e --
Dec 03 02:34:00 hq-pv1 systemd[1]: Starting chrony.service - chrony, an NTP client/server...
Dec 03 02:34:00 hq-pv1 chronyd[3718]: chronyd version 4.3 starting (+CMDMON +NTP +REFCLOCK +RTC +PRIVDROP +SCFILTER +SIGND +ASYNCDNS +NTS +SECHASH +IPV6 -DEBU>
Dec 03 02:34:00 hq-pv1 chronyd[3718]: Frequency -7.752 +/- 0.145 ppm read from /var/lib/chrony/chrony.drift
Dec 03 02:34:00 hq-pv1 chronyd[3718]: Using right/UTC timezone to obtain leap second data
Dec 03 02:34:00 hq-pv1 chronyd[3718]: Loaded seccomp filter (level 1)
Dec 03 02:34:00 hq-pv1 systemd[1]: Started chrony.service - chrony, an NTP client/server.
Dec 03 02:34:39 hq-pv1 chronyd[3718]: Selected source 171.103.24.86 (2.debian.pool.ntp.org)
Dec 03 02:34:39 hq-pv1 chronyd[3718]: System clock wrong by -25191.929899 seconds
Dec 02 19:34:47 hq-pv1 chronyd[3718]: System clock was stepped by -25191.929899 seconds
Dec 02 19:34:47 hq-pv1 chronyd[3718]: System clock TAI offset set to 37 seconds
Dec 02 19:43:00 hq-pv1 chronyd[3718]: chronyd exiting
Dec 02 19:43:00 hq-pv1 systemd[1]: Stopping chrony.service - chrony, an NTP client/server...
Dec 02 19:43:00 hq-pv1 systemd[1]: chrony.service: Deactivated successfully.
Dec 02 19:43:00 hq-pv1 systemd[1]: Stopped chrony.service - chrony, an NTP client/server.
Dec 02 19:43:00 hq-pv1 systemd[1]: Starting chrony.service - chrony, an NTP client/server...
Dec 02 19:43:00 hq-pv1 chronyd[8366]: chronyd version 4.3 starting (+CMDMON +NTP +REFCLOCK +RTC +PRIVDROP +SCFILTER +SIGND +ASYNCDNS +NTS +SECHASH +IPV6 -DEBU>
Dec 02 19:43:00 hq-pv1 chronyd[8366]: Frequency -6.988 +/- 1.665 ppm read from /var/lib/chrony/chrony.drift
Dec 02 19:43:00 hq-pv1 chronyd[8366]: Using right/UTC timezone to obtain leap second data
Dec 02 19:43:00 hq-pv1 chronyd[8366]: Loaded seccomp filter (level 1)
Dec 02 19:43:00 hq-pv1 systemd[1]: Started chrony.service - chrony, an NTP client/server.
Dec 02 19:43:05 hq-pv1 chronyd[8366]: Selected source 147.50.17.138 (2.debian.pool.ntp.org)
Dec 02 19:43:05 hq-pv1 chronyd[8366]: System clock TAI offset set to 37 seconds
Dec 02 19:45:32 hq-pv1 chronyd[8366]: chronyd exiting
Dec 02 19:45:32 hq-pv1 systemd[1]: Stopping chrony.service - chrony, an NTP client/server...
Dec 02 19:45:32 hq-pv1 systemd[1]: chrony.service: Deactivated successfully.
Dec 02 19:45:32 hq-pv1 systemd[1]: Stopped chrony.service - chrony, an NTP client/server.
Dec 02 19:45:32 hq-pv1 systemd[1]: Starting chrony.service - chrony, an NTP client/server...
Dec 02 19:45:32 hq-pv1 chronyd[9141]: chronyd version 4.3 starting (+CMDMON +NTP +REFCLOCK +RTC +PRIVDROP +SCFILTER +SIGND +ASYNCDNS +NTS +SECHASH +IPV6 -DEBU>
Dec 02 19:45:32 hq-pv1 chronyd[9141]: Frequency -7.001 +/- 2.665 ppm read from /var/lib/chrony/chrony.drift
Dec 02 19:45:32 hq-pv1 chronyd[9141]: Using right/UTC timezone to obtain leap second data
Dec 02 19:45:32 hq-pv1 chronyd[9141]: Loaded seccomp filter (level 1)
Dec 02 19:45:32 hq-pv1 systemd[1]: Started chrony.service - chrony, an NTP client/server.
Dec 02 19:45:38 hq-pv1 chronyd[9141]: Selected source 162.159.200.123 (0.asia.pool.ntp.org)
Dec 02 19:45:38 hq-pv1 chronyd[9141]: System clock TAI offset set to 37 seconds
Dec 02 19:46:01 hq-pv1 chronyd[9141]: chronyd exiting
Dec 02 19:46:01 hq-pv1 systemd[1]: Stopping chrony.service - chrony, an NTP client/server...
Dec 02 19:46:01 hq-pv1 systemd[1]: chrony.service: Deactivated successfully.
Dec 02 19:46:01 hq-pv1 systemd[1]: Stopped chrony.service - chrony, an NTP client/server.
-- Boot 3874d70603e84b9a865ebaf7059f041d --
Dec 03 02:50:19 hq-pv1 systemd[1]: Starting chrony.service - chrony, an NTP client/server...
Dec 03 02:50:19 hq-pv1 chronyd[3814]: chronyd version 4.3 starting (+CMDMON +NTP +REFCLOCK +RTC +PRIVDROP +SCFILTER +SIGND +ASYNCDNS +NTS +SECHASH +IPV6 -DEBU>
Dec 03 02:50:19 hq-pv1 chronyd[3814]: Frequency -7.001 +/- 2.665 ppm read from /var/lib/chrony/chrony.drift
Dec 03 02:50:19 hq-pv1 chronyd[3814]: Using right/UTC timezone to obtain leap second data
Dec 03 02:50:19 hq-pv1 chronyd[3814]: Loaded seccomp filter (level 1)
Dec 03 02:50:19 hq-pv1 systemd[1]: Started chrony.service - chrony, an NTP client/server.
Dec 03 02:50:31 hq-pv1 chronyd[3814]: Selected source 162.159.200.123 (0.asia.pool.ntp.org)
Dec 03 02:50:31 hq-pv1 chronyd[3814]: System clock wrong by -25192.359877 seconds
Dec 02 19:50:39 hq-pv1 chronyd[3814]: System clock was stepped by -25192.359877 seconds
Dec 02 19:50:39 hq-pv1 chronyd[3814]: System clock TAI offset set to 37 seconds
Dec 02 20:12:12 hq-pv1 chronyd[3814]: Selected source 103.147.22.149 (0.asia.pool.ntp.org)

And I see these in my logs, which seems to be related:

Dec 02 20:57:12 hq-pv1 pmxcfs[3818]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-node/hq-pv1: -1
Dec 02 20:57:12 hq-pv1 pmxcfs[3818]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-storage/hq-pv1/local: -1
Dec 02 20:57:12 hq-pv1 pmxcfs[3818]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-storage/hq-pv1/local-zfs: -1
Dec 02 20:57:22 hq-pv1 pmxcfs[3818]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-node/hq-pv1: -1
Dec 02 20:57:22 hq-pv1 pmxcfs[3818]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-storage/hq-pv1/local-zfs: -1
Dec 02 20:57:22 hq-pv1 pmxcfs[3818]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-storage/hq-pv1/local: -1
Dec 02 20:57:32 hq-pv1 pmxcfs[3818]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-node/hq-pv1: -1
Dec 02 20:57:32 hq-pv1 pmxcfs[3818]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-storage/hq-pv1/local-zfs: -1
Dec 02 20:57:32 hq-pv1 pmxcfs[3818]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-storage/hq-pv1/local: -1
Dec 02 20:57:42 hq-pv1 pmxcfs[3818]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-node/hq-pv1: -1
Dec 02 20:57:42 hq-pv1 pmxcfs[3818]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-storage/hq-pv1/local-zfs: -1
Dec 02 20:57:42 hq-pv1 pmxcfs[3818]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-storage/hq-pv1/local: -1
Dec 02 20:57:52 hq-pv1 pmxcfs[3818]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-node/hq-pv1: -1
Dec 02 20:57:52 hq-pv1 pmxcfs[3818]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-storage/hq-pv1/local: -1
Dec 02 20:57:52 hq-pv1 pmxcfs[3818]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-storage/hq-pv1/local-zfs: -1
Dec 02 20:58:02 hq-pv1 pmxcfs[3818]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-node/hq-pv1: -1
Dec 02 20:58:02 hq-pv1 pmxcfs[3818]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-storage/hq-pv1/local: -1
Dec 02 20:58:02 hq-pv1 pmxcfs[3818]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-storage/hq-pv1/local-zfs: -1
Dec 02 20:58:12 hq-pv1 pmxcfs[3818]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-node/hq-pv1: -1
Dec 02 20:58:12 hq-pv1 pmxcfs[3818]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-storage/hq-pv1/local: -1
Dec 02 20:58:12 hq-pv1 pmxcfs[3818]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-storage/hq-pv1/local-zfs: -1
Dec 02 20:58:22 hq-pv1 pmxcfs[3818]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-node/hq-pv1: -1
Dec 02 20:58:22 hq-pv1 pmxcfs[3818]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-storage/hq-pv1/local: -1
Dec 02 20:58:22 hq-pv1 pmxcfs[3818]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-storage/hq-pv1/local-zfs: -1
Dec 02 20:58:32 hq-pv1 pmxcfs[3818]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-node/hq-pv1: -1
Dec 02 20:58:32 hq-pv1 pmxcfs[3818]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-storage/hq-pv1/local-zfs: -1
Dec 02 20:58:32 hq-pv1 pmxcfs[3818]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-storage/hq-pv1/local: -1
Dec 02 20:58:42 hq-pv1 pmxcfs[3818]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-node/hq-pv1: -1
Dec 02 20:58:42 hq-pv1 pmxcfs[3818]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-storage/hq-pv1/local-zfs: -1
Dec 02 20:58:42 hq-pv1 pmxcfs[3818]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-storage/hq-pv1/local: -1
Dec 02 20:58:52 hq-pv1 pmxcfs[3818]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-node/hq-pv1: -1
Dec 02 20:58:52 hq-pv1 pmxcfs[3818]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-storage/hq-pv1/local: -1
Dec 02 20:58:52 hq-pv1 pmxcfs[3818]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-storage/hq-pv1/local-zfs: -1
Dec 02 20:59:02 hq-pv1 pmxcfs[3818]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-node/hq-pv1: -1
Dec 02 20:59:02 hq-pv1 pmxcfs[3818]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-storage/hq-pv1/local-zfs: -1
Dec 02 20:59:02 hq-pv1 pmxcfs[3818]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-storage/hq-pv1/local: -1
Dec 02 20:59:12 hq-pv1 pmxcfs[3818]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-node/hq-pv1: -1
Dec 02 20:59:12 hq-pv1 pmxcfs[3818]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-storage/hq-pv1/local: -1
Dec 02 20:59:12 hq-pv1 pmxcfs[3818]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-storage/hq-pv1/local-zfs: -1
Dec 02 20:59:22 hq-pv1 pmxcfs[3818]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-node/hq-pv1: -1
Dec 02 20:59:22 hq-pv1 pmxcfs[3818]: [status] notice: RRDC update error /var/lib/rrdcache

 

[BobhWasatch]

I have a Dell R7525. It operates perfectly normal with Windows. I can go into the System Bios, set the time, set the time in iDrac, and the system time in Windows and everything is fine.

Windows wants the BIOS clock to be on local time, Linux wants it on UTC. This is an historical artifact from MS-DOS that still infects Windows.

However, when I install Proxmox, the system time changes to 7 time zones earlier than the one I am in. I have been working on this for 2 weeks. You name it, I tried it. The problem only happens when I install Proxmox. Tried to install earlier version but get stuck at

Yes, because you set the BIOS clock to local time instead of UTC. Just let chrony set the time and then use "hwclock -w" to set the BIOS clock to match and it will be fine.

 

[adresner]

Thank you for the reply.

If I understand you correctly.

Leave the System Time alone.

Inside iDRAC, set the time zone to UTC (what about NTP?) ?

Then run command: hwclock -w ?

 

[BobhWasatch]

Yes, set the IDRAC time to UTC and it will be fine (after a reboot...the IDRAC/BIOS clock is only read on startup). Or you could let chrony step the clock to the correct time and then use "hwclock -w" to write the system time to the hardware clock. You don't need both. This assumes that you have set the time zone correctly when you installed the system.

I don't know what you mean about NTP. It always operates in UTC anyway as does Linux itself. The time the system shows you is controlled by the timezone setting in timedatectl but that is just for display purposes. Internally, things like file times are stored as UTC and converted for display.

 

[adresner]

Thank you again. I made a fresh install and followed your instructions. Not seeing those notifications in the logs.

However, I'm back to where I started. Timer issues in Windows VMs. My other Dell R730, this does not happen. BIOS settings are the same.

The upload time is UTC, the VM is Thailand Time.

Under the VM options, I tried to change the "Use Local time for RTC" - Default Setting for Windows and No, same problem.

Something still going on.. I have promox running for many years on many servers, never seen this before. Any suggestions?

 

[BobhWasatch]

The Windows time (in the taskbar) seems to be correct, or at least roughly matches the PVE time. The upload time in the Geekbench page could come from anywhere. Like maybe it comes from the server you upload to.

ETA: Windows versions I have used all want the "RTC is local time" setting. There is a registry setting to change it to expect UTC. If that is set then you need to use the other RTC setting for UTC. There's not much more to it than that.

 

[adresner]

So I think this has something to do with the way Epyc 2 (maybe other generations as well) handle AVX2.
If you take a moment and look at my benchmark:
https://browser.geekbench.com/v6/cpu/9315394
you can see the error referring to AVX2.
This article:
https://www.anandtech.com/show/14694/amd-rome-epyc-2nd-gen/2
Seems to reference this?

I have a supermicro server with a gen 2 epyc and getting the same error.

So probably just something I will have to learn to live with.. going away from Intel to AMD, I knew something strange would come up like this. Any thoughts?