Insecure directory in $ENV{PATH} while running with -T switch at

[iwyllpa]

I have Proxmox VE 7.4-3 on a Dell R730XD server. Inside Proxmox, I have an LXC container for Cockpit, an LXC container for Jellyfin, and an LXC container to run Docker using Docker Compose and Portainer.

A few hours ago, we were watching something from Jellyfin when it suddenly stopped working. I tried accessing the Jellyfin webui from a laptop, and I could not connect, so I restarted the LXC container. That didn't work, so I shutdown all LXC containers and started them back up again, one by one. Still Jellyfin didn't work, so I restarted the Proxmox host. When it came back up, I went into the Proxmox WebUI and tried to start the LXC containers up, but each gives an error and won't start:

Error is:
TASK ERROR: command 'lxc-start --version' failed: Insecure directory in $ENV{PATH} while running with -T switch at /usr/lib/x86_64-linux-gnu/perl-base/IPC/Open3.pm line 178.

Same error no matter which LXC container I try to start.

So I tried to open the Proxmox node shell to see what is going on, but that also gives an error:
TASK ERROR: command '/usr/bin/termproxy 5900 --path /nodes/proxmox --perm Sys.Console -- /bin/login -f root' failed: Insecure directory in $ENV{PATH} while running with -T switch at /usr/lib/x86_64-linux-gnu/perl-base/IPC/Open3.pm line 178.

I tried to SSH in on the Proxmox node's IP from my laptop using PuTTY, but I get a Connection Refused error.

I tried to reboot the Proxmox node from the Reboot button within the WebUI, and I get an erorr:
Insecure directory in $ENV{PATH} while running with -T switch at /usr/share/perl5/PVE/API2/Nodes.pm line 569. (500)

I did some Googling, and found:
https://forum.proxmox.com/threads/insecure-directory-in-env-path-while-running-with-t.27153/ Suggests file permissions are the issue
https://forum.proxmox.com/threads/unable-to-start-pvedeamon-after-update.75568/ Suggests apt dist-upgrade could fix things

I went to the server, plugged in a keyboard and monitor, and it was showing a line to log in, so I logged in as root and ran apt dist-upgrade, and that all ran smoothly. I then rebooted (just in case, to finish the upgrading) and when the server finished rebooting, it no longer has an option for me to log in, it just says:

----------------------------
Proxmox Virtual Environment. Please use your web browser to server - connect to:
168.4.218:8006/
---------------------------
(That is me typing in what I am seeing from my laptop, because I obviously can't Ctrl+C/Ctrl+V from my server right now)
I have no idea how to log back in so I can keep trying other possible solutions. So apparently the old monitor that I plugged in to my server was cutting off some of the text on the sides, so it was asking me to log in to the cli, I just couldn't see it.

So I can access the command line from the server itself, but I still cannot login via SSH or through the Proxmox WebUI and into the Shell. That means I can't find any way to copy and paste from the server's command line to this laptop where I'm writing this post.

Other posts that might help (if I can get logged back in to the command line from the server itself):
https://forum.proxmox.com/threads/proxmox-not-restart.45980/page-2#post-218905 Permissions seem to be the issue
https://forum.proxmox.com/threads/pveproxy-compilation-failed.42997/page-2#post-206398 Specifically, change permissions of one of the path in script /usr/share/perl5/PVE/Report.pm. I have no idea how to try that, even if I could log in.

I know that some extra diagnostic info from my Proxmox node could be helpful, but first I need to know how to log in. Currently, I can access the Proxmox WebUI, and it is now Virtual Environment 7.4-17. But I still get errors when I try to access the shell, start an LXC container, or reboot.

I ran pveversion -r, but I can't copy and paste the results. Here are some of the results:
proxmox-ve 7.4-1 (running kernel: 5.15.126-1-pve)
pve-manager: 7.4-17 (running version 7.4-17/513c62be)
pve-kernel-5.15: 7.4-7

What can I try? Before I go messing around with permissions, I would like to know what exactly I should try.

 

[iwyllpa]

Update: Some kind people in the unofficial Discord server have tried to help me out, but we have not managed to solve the issue. I was at least able to get some LXC containers up and running, including my Docker host and Cockpit containers, so I am copying all important files using the network shares through Cockpit and copying/pasting all my Docker Compose files from Portainer and I am going to do a complete re-install and then I'll copy the files back.

In case this helps anyone in the future, here is the command that managed to get an LXC up and running: ```lxc-start -n 101 -F -l DEBUG -o /tmp/lxc-101.log```

And in order to get SSH up and running again, I had to chmod the permissions a bit because the error was complaining they were all 777, This is the command used: ```chmod -R 600 /etc/ssh```

 

[LnxBil]

And in order to get SSH up and running again, I had to chmod the permissions a bit because the error was complaining they were all 777, This is the command used: ```chmod -R 600 /etc/ssh```

That is not good. What about other files in /etc? They should NOT be 777 ...

 

[iwyllpa]

That is not good. What about other files in /etc? They should NOT be 777 ...

Yeah, not good, indeed! I think what happened was that around a month or so ago, I was fighting with permissions trying to get some folders on a ZFS pool set up so that I could access them from within Docker containers and also as network folders on Windows machines, and some command I typed in to modify the permissions of a folder somehow ended up with the wrong folder, and it changed a bunch of permissions that I really shouldn't have changed. However, I had never turned off my Proxmox server node since then (it had about 2 months of uptime before I restarted it) and apparently that restart I did as part of my attempts to fix one LXC container ended up finally making all those changed permissions 'catch up with me', so to speak. So yeah, I have copied everything off that I think will be important, and I'm going to do a full re-install. On the bright side, I had been meaning to make a few modifications now that I have been using this setup for a few months. For example, I want to swap out the 2.5 inch SSD I'm using on the server for a higher capacity one. I also think I'll change up some of the spinning rust drives I have in there and set aside some for backups on a separate machine.

 

[LnxBil]

Yes, that's what I suspected. I was also called to help to fix a couple of machines by accidentially chowning stuff with a "wrongfully placed space and an asterisk". It's easier than you think by just using a similar machine and copying the permissions from there. You can use find to create the correct commands on a healthy system and run them on a compromised one.