Error 500: unable to fork worker in GUI, no SSH access, Login loop

jjmyers

Member
Oct 5, 2019
2
0
21
51
Hello ---

Need some help troubleshooting an issue. Proxmox Virtual Environment 4.4-15/7599e35a.

This was a sudden issue where our only VM is completely down, all other CT are grayed out in the GUI but are up and running. We are unable to SSH in as connection "ssh_exchange_identification: Connection closed by remote host" and can not access via a direct server console either. So we can't grab a syslog or any error logs. Obviously the server is locked-up, but has been running fine for 2 years (we actually have an upgrade path on our schedule near-term).

Since we can only access the environment via GUI, when clicking any action, including console, restart, stop:
"Error 500: unable to fork worker - Resource temporarily unavailable at /usr/share/perl5/PVE/RESTEnvironment.pm line 404."

When clicking some storage devices:
"could not activate storage 'local-zfs', zfs error: open3: fork failed: Resource temporarily unavailable at /usr/share/perl5/PVE/Tools.pm line 411. (500)"

We have plenty of RAM, Storage, CPU, resources, but have had an issue with one finicky CT recently, but primarily was specific to the CT environment.

Before we to have to hard boot or even restart PVE, as these are staging/production instances (again 95% of our CTs are running), we thought we'd throw out our error in case some else has seen this issue. That being said we may have to go into rescue mode tonight to continue our investigation.
 
Need some help troubleshooting an issue. Proxmox Virtual Environment 4.4-15/7599e35a.

FYI: EOL since mid 2018, I'd update very soon, great if you have that scheduled.


Since we can only access the environment via GUI, when clicking any action, including console, restart, stop:
"Error 500: unable to fork worker - Resource temporarily unavailable at /usr/share/perl5/PVE/RESTEnvironment.pm line 404."

When clicking some storage devices:
"could not activate storage 'local-zfs', zfs error: open3: fork failed: Resource temporarily unavailable at /usr/share/perl5/PVE/Tools.pm line 411. (500)"

We have plenty of RAM, Storage, CPU, resources, but have had an issue with one finicky CT recently, but primarily was specific to the CT environment.

fork fails, i.e., a new process cannot be created.. That's something one does not sees every day for sure..

So I guess ther's way to many processes, maybe zombies left over or the CT just got really crazy.
Either you've run into ulimit limits, or PID exhaustion, hard to say..

Anyway, check if you can set ulimit, but if you have no console then you'll have a hard time to do that..

Reboot would work for sure, not sure if you can get out of this in another way...
Did you try directly on the server, i.e., keyboard connected directly and logging in via non-graphical tty?

One Idea I've got left:
You could maybe try to connect into running CTs and stop one of them, this could reduce process count so that you can login into the host and kill the problematic CT. But that all could fail with the same reason too, best thing would be to have a way telling CTs to stop without the need of a new process...
 
  • Like
Reactions: jjmyers
Thanks so much for the reply -- That upgrade is on its way for sure. We'll be at it next weekend.

After many attempts to access CTs directly with no avail, we believe a corrupted network config on one CT may have been the culprit. After a reluctant hard boot, we were able to get everything back up and running, migrated a dead CT to a new one all was well in the world again. We'll be monitoring closely and if we are able track down the actual issue we'll post for future posterity.
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!