PVEPROXY Not Working: can't open lock - Permission denied

[Teapot]

Ive read a few posts here about this issue, but none seem to be my issue. Here is what happened. We installed MegaRaid MSM management software so I could monitor / maintain the built in RAID. Then, we had a problem, so I uninstalled it. After that I lost the GUI for Proxmox. When I try to start pveproxy it throws an error. Here is the paste from systemctl status pveproxy.service:

In journal -xn -- can't open lock '/var/run/pveproxy/pveproxy.pid.lock' - Permission denied

 

[t.lamprecht]

. We installed MegaRaid MSM management software so I could monitor / maintain the built in RAID. Then, we had a problem, so I uninstalled it.

Sounds strange, we do not depend anyhow in anyway on MegaRaid MSM management software packages.. What where the steps for install/remove?

In journal -xn -- can't open lock '/var/run/pveproxy/pveproxy.pid.lock' - Permission denied

Did you tried to manually start pveproxy as root? It seems like the directory permissions got changed and thus the unpriviledged running pveproxy.service cannot create it's lock file..

Please post the output from following commands ran as root:

ps aux | grep pveproxy
ls -al /var/run/pveproxy

 

[Teapot]

1-) dpkg -i megaraid-storage-manager_15.03.01-00_all.deb
dpkg --install lib-utils2_1.00-07_all.deb
https://forum.proxmox.com/threads/megaraid-storage-manager-on-proxmox.13852/

2-)Yes I tried as a root service pveproxy start. But failed.

3-)ps aux | grep pveproxy output

ps aux | grep pveproxy
root     28379  0.0  0.0   6072  2512 pts/5    S+   18:27   0:00 grep pveproxy
ps aux | grep pveproxy
root     28554  0.0  0.0   6072   888 pts/5    S+   18:28   0:00 grep pveproxy

root@xx:/etc# ls -al /var/run/pveproxy
total 0
drwx------  2 www-data www-data   40 Jan 11 18:21 .
drw-r--r-- 30 root     root     1420 Jan 11 18:11 ..

 

[t.lamprecht]

1-) dpkg -i megaraid-storage-manager_15.03.01-00_all.deb
dpkg --install lib-utils2_1.00-07_all.deb
https://forum.proxmox.com/threads/megaraid-storage-manager-on-proxmox.13852/

removal would be more interesting.. Could just post:

tail -50 /var/log/apt/history.log

if it happened recently.

2-)Yes I tried as a root service pveproxy start. But failed.

Can you retry with: systemctl start pvedaemon.service pveproxy.service

3-)ps aux | grep pveproxy output

Weird, directory permissions seem OK, pveproxy does not runs in a frozen state or the like either..

if above start does not work post also (besides the apt history): systemctl status pvedaemon.service pveproxy.service

 

[Teapot]

removal would be more interesting.. Could just post:

tail -50 /var/log/apt/history.log

if it happened recently.



Can you retry with: systemctl start pvedaemon.service pveproxy.service


Weird, directory permissions seem OK, pveproxy does not runs in a frozen state or the like either..

if above start does not work post also (besides the apt history): systemctl status pvedaemon.service pveproxy.service

Thank you for reply.
2-)
root@xx:~# systemctl start pvedaemon.service pveproxy.service


Job for pveproxy.service failed because the control process exited with error code.


See "systemctl status pveproxy.service" and "journalctl -xe" for details.




3-)
root@xx:~# ps aux | grep pveproxy output


grep: output: No such file or directory


4-)
● pvedaemon.service - PVE API Daemon


Loaded: loaded (/lib/systemd/system/pvedaemon.service; enabled; vendor preset: enabled)


Active: active (running) since Sat 2020-01-11 18:05:45 +03; 6h ago


Main PID: 18346 (pvedaemon)


Tasks: 4 (limit: 4915)


Memory: 122.6M


CGroup: /system.slice/pvedaemon.service


├─18346 pvedaemon


├─18347 pvedaemon worker


├─18348 pvedaemon worker


└─18349 pvedaemon worker





Jan 11 18:05:44 xx systemd[1]: Starting PVE API Daemon...


Jan 11 18:05:45 xx pvedaemon[18346]: starting server


Jan 11 18:05:45 xx pvedaemon[18346]: starting 3 worker(s)


Jan 11 18:05:45 xx pvedaemon[18346]: worker 18347 started


Jan 11 18:05:45 xx pvedaemon[18346]: worker 18348 started


Jan 11 18:05:45 xx pvedaemon[18346]: worker 18349 started


Jan 11 18:05:45 xx systemd[1]: Started PVE API Daemon.





● pveproxy.service - PVE API Proxy Server


Loaded: loaded (/lib/systemd/system/pveproxy.service; enabled; vendor preset: enabled)


Active: failed (Result: exit-code) since Sun 2020-01-12 00:36:48 +03; 48s ago


Process: 29064 ExecStartPre=/usr/bin/pvecm updatecerts --silent (code=exited, status=0/SUCCESS)


Process: 29077 ExecStart=/usr/bin/pveproxy start (code=exited, status=255/EXCEPTION)





Jan 12 00:36:48 xx systemd[1]: pveproxy.service: Service RestartSec=100ms expired, scheduling restart.


Jan 12 00:36:48 xx systemd[1]: pveproxy.service: Scheduled restart job, restart counter is at 5.


Jan 12 00:36:48 xx systemd[1]: Stopped PVE API Proxy Server.


Jan 12 00:36:48 xx systemd[1]: pveproxy.service: Start request repeated too quickly.


Jan 12 00:36:48 xx systemd[1]: pveproxy.service: Failed with result 'exit-code'.


Jan 12 00:36:48 xx systemd[1]: Failed to start PVE API Proxy Server.

 

[Teapot]

removal would be more interesting.. Could just post:

tail -50 /var/log/apt/history.log

if it happened recently.



Can you retry with: systemctl start pvedaemon.service pveproxy.service


Weird, directory permissions seem OK, pveproxy does not runs in a frozen state or the like either..

if above start does not work post also (besides the apt history): systemctl status pvedaemon.service pveproxy.service

I rebooted server.
Pveproxy working but gui not working.

root@xx:~# service pveproxy status
● pveproxy.service - PVE API Proxy Server
Loaded: loaded (/lib/systemd/system/pveproxy.service; enabled; vendor preset: enabled)
Active: active (running) since Sun 2020-01-12 06:15:19 +03; 1min 22s ago
Process: 5099 ExecStartPre=/usr/bin/pvecm updatecerts --silent (code=exited, status=0/SUCCESS)
Process: 5127 ExecStart=/usr/bin/pveproxy start (code=exited, status=0/SUCCESS)
Main PID: 5146 (pveproxy)
Tasks: 4 (limit: 4915)
Memory: 125.9M
CGroup: /system.slice/pveproxy.service
├─5146 pveproxy
├─6253 pveproxy worker
├─6254 pveproxy worker
└─6255 pveproxy worker

Jan 12 06:16:39 xx pveproxy[5146]: worker 6189 finished
Jan 12 06:16:39 xx pveproxy[5146]: starting 2 worker(s)
Jan 12 06:16:39 xx pveproxy[5146]: worker 6254 started
Jan 12 06:16:39 xx pveproxy[5146]: worker 6255 started
Jan 12 06:16:39 xx pveproxy[6254]: Can't stat /usr/share/javascript: Permission denied
at /usr/share/perl5/PVE/APIServer/AnyEvent.pm line 1759.
Jan 12 06:16:39 xx pveproxy[6254]: Can't stat /usr/share/fonts/truetype/glyphicons: Permission denied
at /usr/share/perl5/PVE/APIServer/AnyEvent.pm line 1759.
Jan 12 06:16:39 xx pveproxy[6255]: Can't stat /usr/share/javascript: Permission denied
at /usr/share/perl5/PVE/APIServer/AnyEvent.pm line 1759.
Jan 12 06:16:39 xx pveproxy[6255]: Can't stat /usr/share/fonts/truetype/glyphicons: Permission denied
at /usr/share/perl5/PVE/APIServer/AnyEvent.pm line 1759.
Jan 12 06:16:39 xx pveproxy[6254]: Can't locate AnyEvent/Log.pm: /usr/local/lib/x86_64-linux-gnu/perl/5.28.1/AnyEvent/Log.pm: Permission denied at /usr/lib/x86_64-linux-gnu/perl5/5.28/AnyEvent.pm line 1331.
Jan 12 06:16:39 xx pveproxy[6255]: Can't locate AnyEvent/Log.pm: /usr/local/lib/x86_64-linux-gnu/perl/5.28.1/AnyEvent/Log.pm: Permission denied at /usr/lib/x86_64-linux-gnu/perl5/5.28/AnyEvent.pm line 1331.

 

[Teapot]

Here is the permission

root@xx:~# ls -lah /usr/share/javascript/
total 32K
drwxr-xr-x   8 root root 4.0K Sep 22 03:11 .
drwxr-xr-x 135 root root 4.0K Jan 11 14:30 ..
drwxr-xr-x   6 root root 4.0K Sep 22 03:11 bootstrap
drwxr-xr-x   5 root root 4.0K Sep 22 03:11 extjs
lrwxrwxrwx   1 root root   21 Jul  5  2018 font-awesome -> ../fonts-font-awesome
drwxr-xr-x   2 root root 4.0K Sep 22 03:11 jquery
drwxr-xr-x   4 root root 4.0K Dec  7 05:15 proxmox-widget-toolkit
drwxr-xr-x   3 root root 4.0K Sep 22 03:11 sphinxdoc
drwxr-xr-x   2 root root 4.0K Sep 22 03:11 underscore

 

[t.lamprecht]

Your removal of the megaraid package surely did the damange, so it'd be still good to know how you tried to remove it..
You have missing perl modules, strange permission issues, ...

You maybe can come out of this be reinstalling a bunch of packages.. As a starter go with:
apt install --reinstall pve-manager libpve-http-server-perl libjs-extjs proxmox-widget-toolkit libpve-common-perl

 

[WonderBeta]

I've reproduced the issue, and finally found that all folders at / have lost execute permission.
Simply add x permission to them, and the problem should be fixed:

chmod +x /*

And thanks t.lamprecht for reminding me,
there is a bit more permissions need to edit:

chmod 0700 /root
chmod 1755 /tmp

REMEMBER TO CHECK EVERY FOLDER AND FILE IN / TO ENSURE THEIR PERMISSIONS ARE CORRECT AND SECURE, NOT ONLY TWO OF THEM MENTIONED ABOVE.
Hope this helps.

 

[t.lamprecht]

Thansk for sharing what helped you in such a case, a few comments for others though.

I've reproduced the issue, and finally found that all folders at / have lost execute permission.
Simply add x permission to them, and the problem should be fixed:

chmod +x /*

Hope this helps.

First, the symptoms can have various causes, this doesn't need to be the single source of such issues and directories do not just loose there execution permission, so try to remember what was done before the problem started, it's probably related to the actual cause.

Second, with that you gave non-root user permission to read the directory contents of /root (the super-users home directory), that could be deemed as information leak or worse, depending on what files are actually located in /root.

I'd recommend removing that extra permission:

chmod go-x /root

Often you should also be able to get away by reinstalling the package that provides the base hierarchy, normally safer too:

apt install --reinstall base-files

 

[WonderBeta]

Thansk for sharing what helped you in such a case, a few comments for others though.

First, the symptoms can have various causes, this doesn't need to be the single source of such issues and directories do not just loose there execution permission, so try to remember what was done before the problem started, it's probably related to the actual cause.

Second, with that you gave non-root user permission to read the directory contents of /root (the super-users home directory), that could be deemed as information leak or worse, depending on what files are actually located in /root.

I'd recommend removing that extra permission:

chmod go-x /root

Often you should also be able to get away by reinstalling the package that provides the base hierarchy, normally safer too:

apt install --reinstall base-files

Thanks for reminding me that, and I've edited the comment above.

Here's what I've done and the issue occurs:
MSM is provided as csh script and rpm package, can be downloaded here:
17.05.02.01_MSM_Linux-x64.zip
First, I installed csh and alien (for converting rpm to deb) by apt.
Then, I've edited RunRPM.sh(in that archive), replaced rpm -ivh xxx.rpm / rpm -Uvh xxx.rpm to alien -iv xxx.rpm --script,and deleted rpm -i --test.
Run install.csh.
Then I've removed those packages by dpkg --remove xxx.
At this point I've found that issue, but I don't know which step that actually cause the problem.
Thanks again for reminding me security problems and a better way to fix the problem.

 

[gleep52]

wow - I can't believe I found this post after it was made years ago. I too just tried to install the megaraid management software last night by using alien to convert the rpms and then dpkg to install the deb files alien created. Sadly, my GUI went poof after I installed, even though the full install failed because I didn't have java installed. I installed java, then realized my gui was no longer working and I only had ssh access. I uninstalled MSM since I figured it was causing the issue - but then things got worse and I lost ssh access as well. With some help from chatgpt I got my GUI back, but not ssh still. (Even after purging ssh and reinstalling it, reconfiguring the sshd_config file, etc.)

I first noticed issues when trying to do apt update with some permissions issues - I think the /tmp directory?

I've got 4 containers and 14 VMs on this system. I have GUI access but no SSH access. SSH service refuses to start. I've attempted to follow the instructions here but perhaps something has changed since I'm on pve 8.3 now?

I've tried to reinstall the base files as well as the line:

apt install --reinstall pve-manager libpve-http-server-perl libjs-extjs proxmox-widget-toolkit libpve-common-perl

nothing will install since the tmp folder seems to be broken. "chmod 1755 /tmp" doesn't seem to fix the issue. I'm a heavy windows admin and have been playing with proxmox in my off time for the past few months. I've dabled with ubuntu for a good decade or more and raspberry pis - but nothing really concerning permissions and heavy file structure changes.

At this point, I need to reinstall my proxmox host I'm guessing - but what's the easiest way to bring the containers and VMs back without copying all the data? My VM's HDDs are on an nvme pool, my OS is on a different zraid pool mirror for booting.

Can I copy just the /etc/pve/qemu-server/*.conf files somewhere and restore those after reinstalling proxmox?

Obviously if there is an easier route to restoring permissions and not restarting would be ideal so I'm posting here in this thread as a dual post - for help, but also a warning to users who might find this prior to installing MSM and realizing it's a bad idea lol

 

[t.lamprecht]

I've got 4 containers and 14 VMs on this system. I have GUI access but no SSH access. SSH service refuses to start. I've attempted to follow the instructions here but perhaps something has changed since I'm on pve 8.3 now?

Can you check the system log, i.e. in the UI under the node panel or using journalctl -b and post anything that sounds like a (related) error message plus some context before/after these lines?

Maybe this error can be fixed in the existing system.

At this point, I need to reinstall my proxmox host I'm guessing - but what's the easiest way to bring the containers and VMs back without copying all the data? My VM's HDDs are on an nvme pool, my OS is on a different zraid pool mirror for booting.

Can I copy just the /etc/pve/qemu-server/*.conf files somewhere and restore those after reinstalling proxmox?

If you ensure that the VM data is on other disks, and you can easily differentiate them during installation to ensure you do not overwrite any production data permanently then yes, that can work – especially if you keep storage names and such identical.

If you want to continue on that road then I still would heavily recommend creating backups of your virtual machines first, and maybe recursively copy the whole /etc folder to some secure location, as most configuration files are in there, so these can help to get your node back like it was again.

 

[gleep52]

with your pointers on the journalctl -b option I noticed all of my private and public keys were all set to 644 instead of 600 for the private keys. After setting those to 600, ssh will start up again. I'm still pretty unsure of the integrity of the machine at this point. Since these files were obviously changed permission wise - I'm unsure what else could be changed or modified over a clean slate system.

I'm also unsure about the /etc folder as it seems that's where the majority of my issues stemmed from originally after the MSM fiasco.

I still am unable to do an apt update command - I get these results:

apt update
Hit:1 http://ftp.us.debian.org/debian bookworm InRelease
Err:1 http://ftp.us.debian.org/debian bookworm InRelease
  Couldn't create temporary file /tmp/apt.conf.OIowPv for passing config to apt-key
Get:2 http://ftp.us.debian.org/debian bookworm-updates InRelease [55.4 kB]
Get:3 http://security.debian.org/debian-security bookworm-security InRelease [48.0 kB]
Err:3 http://security.debian.org/debian-security bookworm-security InRelease
  Couldn't create temporary file /tmp/apt.conf.iuog9r for passing config to apt-key
Hit:4 http://deb.debian.org/debian bookworm InRelease
Err:4 http://deb.debian.org/debian bookworm InRelease
  Couldn't create temporary file /tmp/apt.conf.RNcLKX for passing config to apt-key
Get:5 http://security.debian.org bookworm-security InRelease [48.0 kB]
Get:6 http://deb.debian.org/debian bookworm-updates InRelease [55.4 kB]
Err:5 http://security.debian.org bookworm-security InRelease
  Couldn't create temporary file /tmp/apt.conf.FgEHxJ for passing config to apt-key
Err:6 http://deb.debian.org/debian bookworm-updates InRelease
  Couldn't create temporary file /tmp/apt.conf.HhGd6B for passing config to apt-key
Err:2 http://ftp.us.debian.org/debian bookworm-updates InRelease
  Couldn't create temporary file /tmp/apt.conf.k06Rrg for passing config to apt-key
Get:7 http://download.proxmox.com/debian/pve bookworm InRelease [2,768 B]
Err:7 http://download.proxmox.com/debian/pve bookworm InRelease
  Couldn't create temporary file /tmp/apt.conf.DF64fB for passing config to apt-key
Fetched 55.4 kB in 0s (118 kB/s)
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
All packages are up to date.
W: An error occurred during the signature verification. The repository is not updated and the previous index files will be used. GPG error: http://ftp.us.debian.org/debian bookworm InRelease: Couldn't create temporary file /tmp/apt.conf.OIowPv for passing config to apt-key
W: An error occurred during the signature verification. The repository is not updated and the previous index files will be used. GPG error: http://security.debian.org/debian-security bookworm-security InRelease: Couldn't create temporary file /tmp/apt.conf.iuog9r for passing config to apt-key
W: An error occurred during the signature verification. The repository is not updated and the previous index files will be used. GPG error: http://deb.debian.org/debian bookworm InRelease: Couldn't create temporary file /tmp/apt.conf.RNcLKX for passing config to apt-key
W: An error occurred during the signature verification. The repository is not updated and the previous index files will be used. GPG error: http://security.debian.org bookworm-security InRelease: Couldn't create temporary file /tmp/apt.conf.FgEHxJ for passing config to apt-key
W: An error occurred during the signature verification. The repository is not updated and the previous index files will be used. GPG error: http://deb.debian.org/debian bookworm-updates InRelease: Couldn't create temporary file /tmp/apt.conf.HhGd6B for passing config to apt-key
W: An error occurred during the signature verification. The repository is not updated and the previous index files will be used. GPG error: http://ftp.us.debian.org/debian bookworm-updates InRelease: Couldn't create temporary file /tmp/apt.conf.k06Rrg for passing config to apt-key
W: An error occurred during the signature verification. The repository is not updated and the previous index files will be used. GPG error: http://download.proxmox.com/debian/pve bookworm InRelease: Couldn't create temporary file /tmp/apt.conf.DF64fB for passing config to apt-key
W: Failed to fetch http://deb.debian.org/debian/dists/bookworm/InRelease  Couldn't create temporary file /tmp/apt.conf.RNcLKX for passing config to apt-key
W: Failed to fetch http://deb.debian.org/debian/dists/bookworm-updates/InRelease  Couldn't create temporary file /tmp/apt.conf.HhGd6B for passing config to apt-key
W: Failed to fetch http://security.debian.org/debian-security/dists/bookworm-security/InRelease  Couldn't create temporary file /tmp/apt.conf.iuog9r for passing config to apt-key
W: Failed to fetch http://ftp.us.debian.org/debian/dists/bookworm/InRelease  Couldn't create temporary file /tmp/apt.conf.OIowPv for passing config to apt-key
W: Failed to fetch http://ftp.us.debian.org/debian/dists/bookworm-updates/InRelease  Couldn't create temporary file /tmp/apt.conf.k06Rrg for passing config to apt-key
W: Failed to fetch http://security.debian.org/dists/bookworm-security/InRelease  Couldn't create temporary file /tmp/apt.conf.FgEHxJ for passing config to apt-key
W: Failed to fetch http://download.proxmox.com/debian/pve/dists/bookworm/InRelease  Couldn't create temporary file /tmp/apt.conf.DF64fB for passing config to apt-key
W: Some index files failed to download. They have been ignored, or old ones used instead.

If I set the temp folder to 777 then I can run apt update without issue - but it's these bizarre permission issues that (when they are changed outside of my doing) I'm pretty unsure how to move forward and feel confident about it. If that makes any sense. Perhaps I lived in linux like I have with windows I'd recognize the basic permission issues but as it stands right now - I'm not sure if /tmp should have 777 or if that was a lucky guess.