Error 500: unable to fork worker in GUI, no SSH access, Login loop

[jjmyers]

Hello ---

Need some help troubleshooting an issue. Proxmox Virtual Environment 4.4-15/7599e35a.

This was a sudden issue where our only VM is completely down, all other CT are grayed out in the GUI but are up and running. We are unable to SSH in as connection "ssh_exchange_identification: Connection closed by remote host" and can not access via a direct server console either. So we can't grab a syslog or any error logs. Obviously the server is locked-up, but has been running fine for 2 years (we actually have an upgrade path on our schedule near-term).

Since we can only access the environment via GUI, when clicking any action, including console, restart, stop:
"Error 500: unable to fork worker - Resource temporarily unavailable at /usr/share/perl5/PVE/RESTEnvironment.pm line 404."

When clicking some storage devices:
"could not activate storage 'local-zfs', zfs error: open3: fork failed: Resource temporarily unavailable at /usr/share/perl5/PVE/Tools.pm line 411. (500)"

We have plenty of RAM, Storage, CPU, resources, but have had an issue with one finicky CT recently, but primarily was specific to the CT environment.

Before we to have to hard boot or even restart PVE, as these are staging/production instances (again 95% of our CTs are running), we thought we'd throw out our error in case some else has seen this issue. That being said we may have to go into rescue mode tonight to continue our investigation.

 

[t.lamprecht]

Need some help troubleshooting an issue. Proxmox Virtual Environment 4.4-15/7599e35a.

FYI: EOL since mid 2018, I'd update very soon, great if you have that scheduled.

Since we can only access the environment via GUI, when clicking any action, including console, restart, stop:
"Error 500: unable to fork worker - Resource temporarily unavailable at /usr/share/perl5/PVE/RESTEnvironment.pm line 404."

When clicking some storage devices:
"could not activate storage 'local-zfs', zfs error: open3: fork failed: Resource temporarily unavailable at /usr/share/perl5/PVE/Tools.pm line 411. (500)"

We have plenty of RAM, Storage, CPU, resources, but have had an issue with one finicky CT recently, but primarily was specific to the CT environment.

fork fails, i.e., a new process cannot be created.. That's something one does not sees every day for sure..

So I guess ther's way to many processes, maybe zombies left over or the CT just got really crazy.
Either you've run into ulimit limits, or PID exhaustion, hard to say..

Anyway, check if you can set ulimit, but if you have no console then you'll have a hard time to do that..

Reboot would work for sure, not sure if you can get out of this in another way...
Did you try directly on the server, i.e., keyboard connected directly and logging in via non-graphical tty?

One Idea I've got left:
You could maybe try to connect into running CTs and stop one of them, this could reduce process count so that you can login into the host and kill the problematic CT. But that all could fail with the same reason too, best thing would be to have a way telling CTs to stop without the need of a new process...

 

[jjmyers]

Thanks so much for the reply -- That upgrade is on its way for sure. We'll be at it next weekend.

After many attempts to access CTs directly with no avail, we believe a corrupted network config on one CT may have been the culprit. After a reluctant hard boot, we were able to get everything back up and running, migrated a dead CT to a new one all was well in the world again. We'll be monitoring closely and if we are able track down the actual issue we'll post for future posterity.