systemd 100% cpu hang?

[orfeous]

Same issue here in south Dublin :-/

 

[t.lamprecht]

Same issue here in south Dublin :-/

If I could get some additional confirmation that installing the mentioned systemd version helps (might require one reboot) then I'd be open to moving the package along to the no-subscription repository, as it really is not that big of a change there.

FYI, I uploaded a newer systemd package with a backported fix for this in version 247.3-7+1-pmx11u1 to the pvetest repository.
The fix is relatively small and targeted and fixes my reproducer here.

 

[orfeous]

I'll try, i'm just trying to restore my "access" to my hosts. It takes like 15-20 min to get into ssh

 

[arbenp]

From west wicklow, thanks for the fix!!, that command worked for me immediately and fixed the hang
ln -sf /usr/share/zoneinfo/Etc /etc/localtime

 

[Shardanoid]

I just tried changing timezone on my system to Etc, and problem went away.

ln -sf /usr/share/zoneinfo/Etc /etc/localtime

I had this system operating in Europe/Dublin timezone. Maybe they issue has something to do with upcoming daylight saving, etc.

After changing the timezone, I rebooted the machine. After reboot, all works fine.

Hello from Sligo!
I just created an account here to thank you!
Have we got a Discord channel for Ireland Proxmox admins?

slán

 

[tiernano]

Hello from Sligo!
I just created an account here to thank you!
Have we got a Discord channel for Ireland Proxmox admins?

slán

Just created one now! not sure if this is allowed, but hopefully it is... https://discord.gg/NbKU75Ka

 

[3m3k]

Just found this thread. In the middle of apt full-upgrade now...will try this when it's done. Dublin-Swords


... 1h later and it's alive !!! Wielke dzieki @gwojcieszczuk

 

[t.lamprecht]

Thanks, didn't want to

FYI, I uploaded a newer systemd package with a backported fix for this in version 247.3-7+1-pmx11u1 to the pvetest repository.
The fix is relatively small and targeted and fixes my reproducer here.

Also, available on no-subscription now; while this year it probably won't affect or help anyone anymore, there's a year after that and Proxmox VE 7.x will still be supported in a year.

 

[crowbarz]

Before finding this thread (and also spending hours figuring out why systemd was at 100% CPU on my Proxmox host) I added the Debian `bullseye-backports` repo and installed `systemd==252.5-2~bpo11+1` and dependencies, and this also resolved the issue.

I'm on the `enterprise` repo, is there an expected timeline for the `247.3-7+1-pmx11u1` patch release to be pushed there? I would prefer to move back to a more standard configuration. Alternatively, is adding `no-subscription` or `test` repos and cherrypicking this update the right way to pull in this patch when on `enterprise`? I am unsure whether adding either of those repos will cause the whole system to be upgraded to the newer versions there, also ISTR having `no-subscription` enabled caused Proxmox to complain at every login.

 

[t.lamprecht]

I'm on the `enterprise` repo, is there an expected timeline for the `247.3-7+1-pmx11u1` patch release to be pushed there?

Yes, probably quite soon in the next days; we just want to ensure that there's no other regression from this patch, even if unlikely.

 

[t.lamprecht]

Alternatively, is adding `no-subscription` or `test` repos and cherrypicking this update the right way to pull in this patch when on `enterprise`?

Yes, and definitively the smaller change than pulling a relatively huge jump in major versions from backports.

I am unsure whether adding either of those repos will cause the whole system to be upgraded to the newer versions there, also ISTR having `no-subscription` enabled caused Proxmox to complain at every login.

You can do targeted updates:

1. Add the no-subscription repository:
2. Run apt update to refresh the index of available packages
3. Run apt full-upgrade systemd
4. Remove/Disable the no-subscription repository again.
5. Run apt update to refresh again

 

[b.miller]

This is the most wholesome thread I've ever read. It's like an Ireland morning roster!

+1 Wicklow

Thanks to support for the speedy reply to my ticket. Nothing like coming into work to find problems on a Monday morning. Though I prefer a simple cup of coffee to wake me up.

 

[dconnx]

Hello from Glen of the Downs, Wicklow. I don't know what to say - today I lost two v7 Proxmox servers (in 2 different datacentres - EquinixDB1 and Servecentric Dublin south & north) - they were really messed up and unresponsive with systemd using 100% of one CPU core). Another pile of servers are on v6 and have no problems. Because of this thread I've managed to get in and change the timezone away from Europe/Dublin and I'm back in (and thank goodness I didn't have to go to the datacentres). I'm honestly speechless (and it makes me like AWS more again). Derek

 

[cregganna]

Hi, Galway here.

Just spent hours and hours on this - even bash-completion was hanging and I couldn't retrain myself not to use the tab key.

apt purge bash-completion
dpkg-reconfigure tzdata # Set to UTC

And reboot, now all OK.

Systemd is getting a good verbal kicking from me though.

 

[mciantar]

Hi, Swords here,

I am happy to have found this post as I have been hours trying to identify what is wrong. I happened to be adding a node to the cluster, so I thought something got corrupted with that. Then it turned out to be this exact problem.

Thanks all!

 

[t.lamprecht]

FYI, the fix in systemd is now widely available on all our repositories, so if you got the update to systemd 247.3-7+1-pmx11u1 or newer your systemd should not run into this specific weird issue anymore.

You can easily (and safely) try this reproducer to confirm (needs the faketime package installed):

TZ=Europe/Dublin faketime 2023-03-26 systemd-analyze calendar --iterations=5 'Sun *-*-* 01:00:00'

On affected systems it will hang, but as it isn't done by the init PID 1 itself it won't cause such wide blockage and nasty effects as through a systemd timer – simply git CTRL + C to abort it. On systems with the fix it will output the next five dates it found and exit, normally all together requiring less than a second to do so.

We also forwarded this to Debian upstream so that they can hopefully include this fix in one of their next point releases too.

 

[Nathanp]

Galway Proxmox homelab here.

Correct me if I am wrong, but the best procedure for upgrading seems to be to switch to the Etc timezone, run the apt full-upgrade systemd and then switch back to the Europe/Dublin timezone. I ran the apt full-upgrade command without changing the timezone and it took over an hour with many package configuration failures. Once it finally finished (with unconfigured packages and package upgrade failures) I changed to the Etc timezone, reran the apt full-upgrade systemd which finished the package configurations and had no installation failures and then set the timezone back to Europe/Dublin and all was good. Just trying to make this transition as smooth as possible for of the Irish Proxmox users affected by this strange bug.

 

[t.lamprecht]

Galway Proxmox homelab here.

Correct me if I am wrong, but the best procedure for upgrading seems to be to switch to the Etc timezone, run the apt full-upgrade systemd and then switch back to the Europe/Dublin timezone. I ran the apt full-upgrade command without changing the timezone and it took over an hour with many package configuration failures. Once it finally finished (with unconfigured packages and package upgrade failures) I changed to the Etc timezone, reran the apt full-upgrade systemd which finished the package configurations and had no installation failures and then set the timezone back to Europe/Dublin and all was good. Just trying to make this transition as smooth as possible for of the Irish Proxmox users affected by this strange bug.

Yes, if your system is still overloaded and bogged down due to this bug it's better to first apply the workaround by changing the server to UTC, and then doing the upgrade. If you want to change back to Europe/Dublin is then up to you or your company/customer/clients needs - as it can have some advantage of running a server with UTC (no time "jumps" twice a year due to daylight saving times).

 

[nahog]

Regards from Greystones

Wow! Fellow Greystoninan here. Same problem! Fix worked like a charm!

 

[Dobby90]

Thanks from TIpperary came back from holidays to server not running due to power cut, After hours of puling hair out I found your fix