Hello from blanch. alsoAh.. Dublin Blanch here alsoThis would have been handy several hours ago
systemd 100% cpu hang?
- Thread starter rechena
- Start date
Can you hear the THANK YOU comin all the way from Meath?
I made little experiment. I had 2nd proxmox 7.3 running. This one has access to pve-enterprise repository. I applied all the updates on this system (+reboot). This one has been also affected by the Europe/Dublin timezone issue.
I switched to Etc timezone, and problem immediately goes away.
I tried putting Europe/Dublin timezone back, and the problem re-appears.
This must be some Celtic curse
I switched to Etc timezone, and problem immediately goes away.
I tried putting Europe/Dublin timezone back, and the problem re-appears.
This must be some Celtic curse
LOL. He must have planted this logical bomb into systemd code. How clever of him ...
Thank you so much! I’ve been pulling my hair out all day! I owe you one. Clontarf/Dublin.
same issue for me, hello from LouthRegards from Greystones
just tried changing to the etc time zone and it worked! Based in Dublin, and had issues with 3 servers (the Intel nuc just wouldn't boot... even rebuilt it last time... will try it again in the morning). When I did get back into the boxes, did an apt update, and there is an update for tzdata... wonder does that fix it? Haven't been brave enough to try it yet... nearly 3am and don't want to break much before bed...
This could be the CEST Austria Daylight Savings change tonight considering the workaround.
https://www.visitingvienna.com/visitorinfo/time/
Proxmox is based out of Vienna, Austria.
A Timing glitch.
I just tried changing timezone on my system to Etc, and problem went away.
https://www.visitingvienna.com/visitorinfo/time/
Proxmox is based out of Vienna, Austria.
A Timing glitch.
I just tried changing timezone on my system to Etc, and problem went away.
Last edited:
This is possible. BTW, I have all the updates applied, PVE 7.4-3, etc. If I switch to Europe/Dublin TZ, problem manifests itself immediately.
Yeah, I can reproduce this, i.e. the following hangs and spawns a process eating up 100% CPU time of one core:
But it works if one either sets the TZ to something else (e.g.,
Shortest workaround for now is using UTC or some other timezone. Note that using UTC for servers has its advantages in general, as it has no daylight-saving time "jumps" (but for the sake of completeness: it has still leap second adjustments, but those might be abolished too).
The root cause here seems to be that Ireland is IIRC the only one that uses summer time as its standard time and winter time as their non-daylight-saving time so to say.
We'll look into a more permanent fix.
TZ=Europe/Dublin faketime 2023-03-26 systemd-analyze calendar --iterations=5 'Sun *-*-* 01:00:00'But it works if one either sets the TZ to something else (e.g.,
TZ=Europe/Vienna or TZ=Etc/UTC or advances the by faketime simulated day by one, to 2023-03-27.Shortest workaround for now is using UTC or some other timezone. Note that using UTC for servers has its advantages in general, as it has no daylight-saving time "jumps" (but for the sake of completeness: it has still leap second adjustments, but those might be abolished too).
The root cause here seems to be that Ireland is IIRC the only one that uses summer time as its standard time and winter time as their non-daylight-saving time so to say.
We'll look into a more permanent fix.
Last edited:
Another hello from Wicklow here, we must invest in a DataCentre...haha, that's mad.
That's worked for me as well.
I did a bit of more investigation. It seem to me the problem manifests itself only from 25th March 2023 and earlier.
When I first encountered issue with systemd utilizing 100% of CPU resources, the version of tzdata DEB package was 2021a-1+deb11u8. As of now (2023-03-26), rebooting Proxmox system with Europe/Dublin timezone set, resolves the problem (at least for me). I tested moving time 1 days/2 days backward, and the problem is triggered immediately. I also updatedd tzdata package to latest version 2021a-1+deb11u9, and I can also trigger the problem by moving time backward.
Last experiment was to use Dublin's timezone file from different distribution; I chose Rocky Linux 9.1 (tzdata-2022g-1.el9_1.noarch). Interestingly, the problem isn't present when different tzdata file is used; moving time backward + using RockyLinux 9.1 tzdata file (Europe/Dublin TZ) doesn't trigger the problem.
Below screenshot depicts some differences in the content of tzdata file.
What a bizzare problem indeed.
When I first encountered issue with systemd utilizing 100% of CPU resources, the version of tzdata DEB package was 2021a-1+deb11u8. As of now (2023-03-26), rebooting Proxmox system with Europe/Dublin timezone set, resolves the problem (at least for me). I tested moving time 1 days/2 days backward, and the problem is triggered immediately. I also updatedd tzdata package to latest version 2021a-1+deb11u9, and I can also trigger the problem by moving time backward.
Last experiment was to use Dublin's timezone file from different distribution; I chose Rocky Linux 9.1 (tzdata-2022g-1.el9_1.noarch). Interestingly, the problem isn't present when different tzdata file is used; moving time backward + using RockyLinux 9.1 tzdata file (Europe/Dublin TZ) doesn't trigger the problem.
Below screenshot depicts some differences in the content of tzdata file.
What a bizzare problem indeed.
Attachments
+1 from Dublin... that's solved it for me, great find! Was just about to prep for a rebuild after a few hours of deadends...
For me it began after a successful boot yesterday morning (25th), VMs were working fine all day, then the host became unresponsive to commands when shutting down around 7pm.
All my other various installs run UTC, but the proxmox machine has been stable for 16+ months since it was built so I never had a reason to check it.
For me it began after a successful boot yesterday morning (25th), VMs were working fine all day, then the host became unresponsive to commands when shutting down around 7pm.
All my other various installs run UTC, but the proxmox machine has been stable for 16+ months since it was built so I never had a reason to check it.
FYI, I uploaded a newer systemd package with a backported fix for this in version
The fix is relatively small and targeted and fixes my reproducer here.
247.3-7+1-pmx11u1 to the pvetest repository.The fix is relatively small and targeted and fixes my reproducer here.