it's the weirdest thing, I ran:
:~# lsof -K | grep inotify | wc -l
93
Which is hardly enough watches to saturate the system. I then shutdown all containers except the one experiencing issues. It still wouldn't come up in the Proxmox console or allow SSH in.
I noticed on the affected system SSH is in an inactive/dead state despite being set to enabled.
Trying to start it took ages, but when it did start the Console came up in Proxmox UI and I was able to SSH in.
Only differences with the affected container vs other working containers are:
1. It's using the ID of a container which I previously deleted.
2. It's Debian 9, other containers are Ubuntu 18.04.
I've just setup an Ununtu 18.04 container with identical config and same software as the affected container, everything works perfectly.
Here I was pushing myself to use Debian more vs Ubuntu but yikes, got bit. I'm curious what the issue might be since I'd prefer to use Debian long term over Ubuntu. Any ideas?