systemd-timedated starting every minute filling up the journal

[pflirsich]

On one of our three nodes I observe the following:

Apr 14 10:12:46 proxmox3 dbus-daemon[3026]: [system] Activating via systemd: service name='org.freedesktop.timedate1' unit='dbus-org.freedesktop.timedate1.service' requested by ':1.457' (uid=33 pid=136911 comm="timedatectl show --property=Timezone --value>
Apr 14 10:12:46 proxmox3 systemd[1]: Starting systemd-timedated.service - Time & Date Service...
Apr 14 10:12:46 proxmox3 systemd[1]: Started systemd-timedated.service - Time & Date Service.
Apr 14 10:12:46 proxmox3 dbus-daemon[3026]: [system] Successfully activated service 'org.freedesktop.timedate1'
Apr 14 10:13:16 proxmox3 systemd[1]: systemd-timedated.service: Deactivated successfully.
Apr 14 10:13:46 proxmox3 dbus-daemon[3026]: [system] Activating via systemd: service name='org.freedesktop.timedate1' unit='dbus-org.freedesktop.timedate1.service' requested by ':1.459' (uid=33 pid=137683 comm="timedatectl show --property=Timezone --value>
Apr 14 10:13:46 proxmox3 systemd[1]: Starting systemd-timedated.service - Time & Date Service...
Apr 14 10:13:46 proxmox3 systemd[1]: Started systemd-timedated.service - Time & Date Service.
Apr 14 10:13:46 proxmox3 dbus-daemon[3026]: [system] Successfully activated service 'org.freedesktop.timedate1'
Apr 14 10:14:16 proxmox3 systemd[1]: systemd-timedated.service: Deactivated successfully.

Every minute at exactly the 46th second something (probably pveproxy I'm guessing) is querying the timezone. If I do systemctl stop pveproxy the log messages stop. If I start the service again the log messages continue. Also it doesn't matter when I stop the service, the query remains at exactly the 46th second.
After exactly 30s the service "deactivated successfully". This goes on and on filling my journal.

Spoiler: Packet versions

proxmox-ve: 9.1.0 (running kernel: 6.17.13-2-pve)
pve-manager: 9.1.7 (running version: 9.1.7/16b139a017452f16)
proxmox-kernel-helper: 9.0.4
proxmox-kernel-6.17: 6.17.13-2
proxmox-kernel-6.17.13-2-pve-signed: 6.17.13-2
proxmox-kernel-6.17.2-1-pve-signed: 6.17.2-1
amd64-microcode: 3.20251202.1~bpo13+1
ceph-fuse: 19.2.3-pve2
corosync: 3.1.10-pve2
criu: 4.1.1-1
frr-pythontools: 10.4.1-1+pve1
ifupdown2: 3.3.0-1+pmx12
ksm-control-daemon: 1.5-1
libjs-extjs: 7.0.0-5
libproxmox-acme-perl: 1.7.1
libproxmox-backup-qemu0: 2.0.2
libproxmox-rs-perl: 0.4.1
libpve-access-control: 9.0.6
libpve-apiclient-perl: 3.4.2
libpve-cluster-api-perl: 9.1.1
libpve-cluster-perl: 9.1.1
libpve-common-perl: 9.1.9
libpve-guest-common-perl: 6.0.2
libpve-http-server-perl: 6.0.5
libpve-network-perl: 1.2.5
libpve-rs-perl: 0.11.4
libpve-storage-perl: 9.1.1
libspice-server1: 0.15.2-1+b1
lvm2: 2.03.31-2+pmx1
lxc-pve: 6.0.5-4
lxcfs: 6.0.4-pve1
novnc-pve: 1.6.0-3
proxmox-backup-client: 4.1.5-1
proxmox-backup-file-restore: 4.1.5-1
proxmox-backup-restore-image: 1.0.0
proxmox-firewall: 1.2.1
proxmox-kernel-helper: 9.0.4
proxmox-mail-forward: 1.0.2
proxmox-mini-journalreader: 1.6
proxmox-offline-mirror-helper: 0.7.3
proxmox-widget-toolkit: 5.1.9
pve-cluster: 9.1.1
pve-container: 6.1.2
pve-docs: 9.1.2
pve-edk2-firmware: 4.2025.05-2
pve-esxi-import-tools: 1.0.1
pve-firewall: 6.0.4
pve-firmware: 3.18-2
pve-ha-manager: 5.1.3
pve-i18n: 3.7.0
pve-qemu-kvm: 10.1.2-7
pve-xtermjs: 5.5.0-3
qemu-server: 9.1.6
smartmontools: 7.4-pve1
spiceterm: 3.4.1
swtpm: 0.8.0+pve3
vncterm: 1.9.1
zfsutils-linux: 2.4.1-pve1

My other two nodes are still running 8.4.17 with Linux 6.8.12-4-pve (2024-11-06T15:04Z) I didn't rebooted them after upgrade but I dont think the other node would cause this "issue"? This node joined the cluster with 9.1.7.

I recreated the cluster in a PVE Test-Setup and tried to match the configuration as close as possible. The Log-Spam doesn't happen there.

Furthermore I did the following observation: If I open the timezone settings in the WebUI the log messages doesn't appear either. When I close the settings the log messages appear again. In the following excerpt you can see the gap between 11:03:33 and 11:06:46 (yet again the 46th second).

Apr 14 11:03:33 proxmox3 systemd[1]: Starting systemd-timedated.service - Time & Date Service...
Apr 14 11:03:33 proxmox3 systemd[1]: Started systemd-timedated.service - Time & Date Service.
Apr 14 11:03:33 proxmox3 dbus-daemon[3026]: [system] Successfully activated service 'org.freedesktop.timedate1'
Apr 14 11:06:16 proxmox3 systemd[1]: systemd-timedated.service: Deactivated successfully.
Apr 14 11:06:46 proxmox3 dbus-daemon[3026]: [system] Activating via systemd: service name='org.freedesktop.timedate1' unit='dbus-org.freedesktop.timedate1.service' requested by ':1.703' (uid=33 pid=179327 comm="timedatectl show --property=Timezone --value" label="unconfined")
Apr 14 11:06:46 proxmox3 systemd[1]: Starting systemd-timedated.service - Time & Date Service...

I found this Commit (Fix #7175) in the Github Repo but cannot really judge if it has something to do with this (maybe just cosmetic) issue.

Now my question is: Is this really desired behavior or is this a bug? Did someone else observed this and know how to "fix" it?

 

[pershakov.av]

I raised the level in the /etc/systemd/journald.conf to MaxLevelSyslog=info and restarted the service systemctl restart systemd-journald

 

[pflirsich]

Unfortunately this doesn't work for me.

 

[leesteken]

I have never seen this but I switched to chrony (which is the new default since PVE 7) quite some time ago: https://pve.proxmox.com/pve-docs/pve-admin-guide.html#_time_synchronization

EDIT: Looks like this is not a fix or work-around, unfortunately.

 

[mauzeris]

After last PVE update(9.1.7), I got same spam messages, Im using chrony as mentioned above. Weird.

 

[pflirsich]

Yep, I'm using chrony too as this was a fresh installation. Guess we just have to wait for a fix.

 

[leesteken]

I believe this service sets the time once during startup. Maybe just disable it, as the chrony service will sync time anyway? I still have no idea why it does start every minute. Any scripts installed? Or any other monitoring tooling that continuously checks the services and sees that it is not running (which is normal as it just does it thing and stops)?

 

[mauzeris]

I have tried to disable it, and got this:

root@pve:~# systemctl disable systemd-timedated.service
The unit files have no installation config (WantedBy=, RequiredBy=, UpheldBy=,
Also=, or Alias= settings in the [Install] section, and DefaultInstance= for
template units). This means they are not meant to be enabled or disabled using systemctl.

Possible reasons for having these kinds of units are:
• A unit may be statically enabled by being symlinked from another unit's
  .wants/, .requires/, or .upholds/ directory.
• A unit's purpose may be to act as a helper for some other unit which has
  a requirement dependency on it.
• A unit may be started when needed via activation (socket, path, timer,
  D-Bus, udev, scripted systemctl call, ...).
• In case of template units, the unit is meant to be enabled with some
  instance name specified.

edit: ok, I tried to mask it, and now i got spam of:

Apr 17 11:10:40 pve dbus-daemon[1218]: [system] Activating via systemd: service name='org.freedesktop.timedate1' unit='dbus-org.freedesktop.timedate1.service' requested by ':1.2316' (uid=33 pid=502369 comm="timedatectl show --property=Timezone --value" label="unconfined")
Apr 17 11:10:40 pve dbus-daemon[1218]: [system] Activation via systemd failed for unit 'dbus-org.freedesktop.timedate1.service': Unit systemd-timedated.service is masked.

 

[leesteken]

Makes sense, as it is a oneshot service. Do you have anything setup or installed that might be trying to start it every minute (as it looks like it is not running, which is actually fine)?

 

[mauzeris]

Makes sense, as it is a oneshot service. Do you have anything setup or installed that might be trying to start it every minute (as it looks like it is not running, which is actually fine)?

Its a single node, pretty basic setup, I only monitoring it with Zabbix

 

[pflirsich]

Its a single node

Now things are getting interesting.

I just deleted one node from my cluster to set it up with new hardware and new boot configuration. When it wasn't connected to the cluster the log-spam did not happen.

As soon as I joined the cluster the log-spam started.

I have some Zabbix monitoring active and can check if disabling this changes anything.

 

[pflirsich]

I am using the template Proxmox VE by HTTP in Zabbix. Disabling it stops the log-spam.

The template has an item Time which uses the API : https://{$PVE.URL.HOST}:{$PVE.URL.PORT}/api2/json/nodes/proxmox1/time
It has a default polling interval of 1m.

So I would say increasing the interval should help at first.

Obviously this is only my solution since I don't know if that applies to you as well.

Maybe I have to update the template if an update is available. I will check this next week.

 

[leesteken]

Maybe Zabbix should not monitor that (oneshot) service, as it's not supposed to be running all the time? Maybe report this to Zabbix?