timedatectl (systemd-timesyncd, systemd-timedated) failed to start

[opsflexyz]

In one of our clusters we have a server where the systemd-timesyncd and systemd-timedated service are failing.

When using the timedatectl command we get a timeout error:

timedatectl status

Failed to query server: Failed to activate service 'org.freedesktop.timedate1': timed out (service_start_timeout=25000ms)

systemd-timesyncd:

● systemd-timesyncd.service - Network Time Synchronization
   Loaded: loaded (/lib/systemd/system/systemd-timesyncd.service; enabled; vendor preset: enabled)
  Drop-In: /usr/lib/systemd/system/systemd-timesyncd.service.d
           └─disable-with-time-daemon.conf
   Active: activating (start) since Thu 2021-07-01 10:38:39 CEST; 35s ago
     Docs: man:systemd-timesyncd.service(8)
 Main PID: 42153 ((imesyncd))
    Tasks: 1 (limit: 9830)
   Memory: 632.0K
   CGroup: /system.slice/systemd-timesyncd.service
           └─42153 (imesyncd)

Jul 01 10:38:39 node100 systemd[1]: Starting Network Time Synchronization...


Jul 01 11:07:13 node100 systemd[1]: systemd-timesyncd.service: Service has no hold-off time (RestartSec=0), scheduling restart.
Jul 01 11:07:13 node100 systemd[1]: systemd-timesyncd.service: Scheduled restart job, restart counter is at 134.
Jul 01 11:07:13 node100 systemd[1]: Stopped Network Time Synchronization.

systemd-timedated:

● systemd-timedated.service - Time & Date Service
   Loaded: loaded (/lib/systemd/system/systemd-timedated.service; static; vendor preset: enabled)
   Active: failed (Result: timeout) since Thu 2021-07-01 10:01:32 CEST; 37min ago
     Docs: man:systemd-timedated.service(8)
           man:localtime(5)
           https://www.freedesktop.org/wiki/Software/systemd/timedated
  Process: 20929 ExecStart=/lib/systemd/systemd-timedated (code=killed, signal=TERM)
 Main PID: 20929 (code=killed, signal=TERM)

Jul 01 10:00:02 node100 systemd[1]: Starting Time & Date Service...
Jul 01 10:01:32 node100 systemd[1]: systemd-timedated.service: Start operation timed out. Terminating.
Jul 01 10:01:32 node100 systemd[1]: systemd-timedated.service: Main process exited, code=killed, status=15/TERM
Jul 01 10:01:32 node100 systemd[1]: systemd-timedated.service: Failed with result 'timeout'.
Jul 01 10:01:32 node100 systemd[1]: Failed to start Time & Date Service.

proxmox-ve: 6.4-1 (running kernel: 5.4.106-1-pve)
pve-manager: 6.4-5 (running version: 6.4-5/6c7bf5de)
pve-kernel-5.4: 6.4-1
pve-kernel-helper: 6.4-1
pve-kernel-5.4.106-1-pve: 5.4.106-1
pve-kernel-5.4.73-1-pve: 5.4.73-1
ceph-fuse: 12.2.11+dfsg1-2.1+b1
corosync: 3.1.2-pve1
criu: 3.11-3
glusterfs-client: 5.5-3
ifupdown: not correctly installed
ifupdown2: 3.0.0-1+pve3
ksm-control-daemon: 1.3-1
libjs-extjs: 6.0.1-10
libknet1: 1.20-pve1
libproxmox-acme-perl: 1.0.8
libproxmox-backup-qemu0: 1.0.3-1
libpve-access-control: 6.4-1
libpve-apiclient-perl: 3.1-3
libpve-common-perl: 6.4-2
libpve-guest-common-perl: 3.1-5
libpve-http-server-perl: 3.2-1
libpve-storage-perl: 6.4-1
libqb0: 1.0.5-1
libspice-server1: 0.14.2-4~pve6+1
lvm2: 2.03.02-pve4
lxc-pve: 4.0.6-2
lxcfs: 4.0.6-pve1
novnc-pve: 1.1.0-1
proxmox-backup-client: 1.1.5-1
proxmox-mini-journalreader: 1.1-1
proxmox-widget-toolkit: 2.5-3
pve-cluster: 6.4-1
pve-container: 3.3-5
pve-docs: 6.4-1
pve-edk2-firmware: 2.20200531-1
pve-firewall: 4.1-3
pve-firmware: 3.2-2
pve-ha-manager: 3.1-1
pve-i18n: 2.3-1
pve-qemu-kvm: 5.2.0-6
pve-xtermjs: 4.7.0-3
qemu-server: 6.4-2
smartmontools: 7.2-pve2
spiceterm: 3.1-1
vncterm: 1.6-2
zfsutils-linux: 2.0.4-pve1

Yesterday everything was working well and this morning with out ansible run when the command "timedatctl" was issued this gave a error.

 

[Moayad]

may you try to restart the timedated service and see if that helps systemctl restart systemd-timedated otherwise check the Syslog in that node for any error message

 

[opsflexyz]

Thanks for your reply!

This is the log for the service:

Jul 01 12:22:16 node100 systemd[1]: systemd-timedated.service: Start operation timed out. Terminating.
Jul 01 12:22:16 node100 systemd[1]: systemd-timedated.service: Main process exited, code=killed, status=15/TERM
-- Subject: Unit process exited
-- Defined-By: systemd
-- Support: https://www.debian.org/support
--
-- An ExecStart= process belonging to unit systemd-timedated.service has exited.
--
-- The process' exit code is 'killed' and its exit status is 15.

Jul 01 12:22:16 node100 systemd[1]: systemd-timedated.service: Failed with result 'timeout'.
-- Subject: Unit failed
-- Defined-By: systemd
-- Support: https://www.debian.org/support
--
-- The unit systemd-timedated.service has entered the 'failed' state with result 'timeout'.

Jul 01 12:22:16 node100 systemd[1]: Failed to start Time & Date Service.
-- Subject: A start job for unit systemd-timedated.service has failed
-- Defined-By: systemd
-- Support: https://www.debian.org/support
--
-- A start job for unit systemd-timedated.service has finished with a failure.
--
-- The job identifier is 1748973 and the job result is failed.

Jul 01 12:22:16 node100 audit[1]: SERVICE_START pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=systemd-timedated comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=failed'

It is getting timed out but it i cannot find on what...

 

[Moayad]

Did restarting the node not help?

can you run timedatectl status during the journalctl -f and send the output.

 

[opsflexyz]

We cannot restart the node at this moment. This is the only node with nvme local storage. So we cannot move the vm's off of this node.

Here is the output of timedatectl status

Jul 01 14:39:00 node100 dbus-daemon[8534]: [system] Activating via systemd: service name='org.freedesktop.timedate1' unit='dbus-org.freedesktop.timedate1.service' requested by ':1.19631' (uid=0 pid=2241 comm="timedatectl status " label="unconfined")
Jul 01 14:39:00 node100 systemd[1]: Starting Time & Date Service...
Jul 01 14:39:00 node100 systemd[1]: pvesr.service: Succeeded.
Jul 01 14:39:00 node100 systemd[1]: Started Proxmox VE replication runner.
Jul 01 14:39:00 node100 audit[1]: SERVICE_START pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=pvesr comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
Jul 01 14:39:00 node100 audit[1]: SERVICE_STOP pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=pvesr comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
Jul 01 14:39:01 node100 systemd[1]: systemd-timesyncd.service: Start operation timed out. Terminating.
Jul 01 14:39:01 node100 systemd[1]: systemd-timesyncd.service: Main process exited, code=killed, status=15/TERM
Jul 01 14:39:01 node100 systemd[1]: systemd-timesyncd.service: Failed with result 'timeout'.
Jul 01 14:39:01 node100 systemd[1]: Failed to start Network Time Synchronization.
Jul 01 14:39:01 node100 audit[1]: SERVICE_START pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=systemd-timesyncd comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=failed'
Jul 01 14:39:01 node100 systemd[1]: systemd-timesyncd.service: Service has no hold-off time (RestartSec=0), scheduling restart.
Jul 01 14:39:01 node100 systemd[1]: systemd-timesyncd.service: Scheduled restart job, restart counter is at 275.
Jul 01 14:39:01 node100 systemd[1]: Stopped Network Time Synchronization.
Jul 01 14:39:01 node100 audit[1]: SERVICE_START pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=systemd-timesyncd comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
Jul 01 14:39:01 node100 audit[1]: SERVICE_STOP pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=systemd-timesyncd comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
Jul 01 14:39:01 node100 systemd[1]: Starting Network Time Synchronization...
Jul 01 14:39:01 node100 audit: NETFILTER_CFG table=filter family=7 entries=0
Jul 01 14:39:04 node100 pveproxy[3381]: worker exit
Jul 01 14:39:04 node100 pveproxy[10061]: worker 3381 finished
Jul 01 14:39:04 node100 pveproxy[10061]: starting 1 worker(s)
Jul 01 14:39:04 node100 pveproxy[10061]: worker 2733 started
Jul 01 14:39:07 node100 pvestatd[9952]: status update time (5.936 seconds)
Jul 01 14:39:25 node100 dbus-daemon[8534]: [system] Failed to activate service 'org.freedesktop.timedate1': timed out (service_start_timeout=25000ms)

"Jul 01 14:39:01 node100 systemd[1]: systemd-timesyncd.service: Scheduled restart job, restart counter is at 275."

Seems to be restarting indefinetly...

 

[ph0x]

Although it's not a fix per se, but since I switched to chrony I've never had an issue with time again.
Maybe it's worth trying.

 

[zisco]

Although it's not a fix per se, but since I switched to chrony I've never had an issue with time again.
Maybe it's worth trying.

I have a similar problem here - is there a documentation available on how to switch to chrony?