PVEPROXY Not Working: can't open lock - Permission denied

Teapot

Member
Apr 9, 2019
23
0
21
27
Ive read a few posts here about this issue, but none seem to be my issue. Here is what happened. We installed MegaRaid MSM management software so I could monitor / maintain the built in RAID. Then, we had a problem, so I uninstalled it. After that I lost the GUI for Proxmox. When I try to start pveproxy it throws an error. Here is the paste from systemctl status pveproxy.service:

In journal -xn -- can't open lock '/var/run/pveproxy/pveproxy.pid.lock' - Permission denied
 
. We installed MegaRaid MSM management software so I could monitor / maintain the built in RAID. Then, we had a problem, so I uninstalled it.

Sounds strange, we do not depend anyhow in anyway on MegaRaid MSM management software packages.. What where the steps for install/remove?

In journal -xn -- can't open lock '/var/run/pveproxy/pveproxy.pid.lock' - Permission denied

Did you tried to manually start pveproxy as root? It seems like the directory permissions got changed and thus the unpriviledged running pveproxy.service cannot create it's lock file..

Please post the output from following commands ran as root:

Code:
ps aux | grep pveproxy
ls -al /var/run/pveproxy
 
  • Like
Reactions: Teapot
1-) dpkg -i megaraid-storage-manager_15.03.01-00_all.deb
dpkg --install lib-utils2_1.00-07_all.deb
https://forum.proxmox.com/threads/megaraid-storage-manager-on-proxmox.13852/

2-)Yes I tried as a root service pveproxy start. But failed.

3-)ps aux | grep pveproxy output
Code:
ps aux | grep pveproxy
root     28379  0.0  0.0   6072  2512 pts/5    S+   18:27   0:00 grep pveproxy
ps aux | grep pveproxy
root     28554  0.0  0.0   6072   888 pts/5    S+   18:28   0:00 grep pveproxy

Code:
root@xx:/etc# ls -al /var/run/pveproxy
total 0
drwx------  2 www-data www-data   40 Jan 11 18:21 .
drw-r--r-- 30 root     root     1420 Jan 11 18:11 ..
 
Last edited by a moderator:
1-) dpkg -i megaraid-storage-manager_15.03.01-00_all.deb
dpkg --install lib-utils2_1.00-07_all.deb
https://forum.proxmox.com/threads/megaraid-storage-manager-on-proxmox.13852/

removal would be more interesting.. Could just post:
Code:
tail -50 /var/log/apt/history.log
if it happened recently.

2-)Yes I tried as a root service pveproxy start. But failed.

Can you retry with: systemctl start pvedaemon.service pveproxy.service

3-)ps aux | grep pveproxy output
Weird, directory permissions seem OK, pveproxy does not runs in a frozen state or the like either..

if above start does not work post also (besides the apt history): systemctl status pvedaemon.service pveproxy.service
 
removal would be more interesting.. Could just post:
Code:
tail -50 /var/log/apt/history.log
if it happened recently.



Can you retry with: systemctl start pvedaemon.service pveproxy.service


Weird, directory permissions seem OK, pveproxy does not runs in a frozen state or the like either..

if above start does not work post also (besides the apt history): systemctl status pvedaemon.service pveproxy.service
Thank you for reply.
2-)
root@xx:~# systemctl start pvedaemon.service pveproxy.service


Job for pveproxy.service failed because the control process exited with error code.


See "systemctl status pveproxy.service" and "journalctl -xe" for details.




3-)
root@xx:~# ps aux | grep pveproxy output


grep: output: No such file or directory


4-)
● pvedaemon.service - PVE API Daemon


Loaded: loaded (/lib/systemd/system/pvedaemon.service; enabled; vendor preset: enabled)


Active: active (running) since Sat 2020-01-11 18:05:45 +03; 6h ago


Main PID: 18346 (pvedaemon)


Tasks: 4 (limit: 4915)


Memory: 122.6M


CGroup: /system.slice/pvedaemon.service


├─18346 pvedaemon


├─18347 pvedaemon worker


├─18348 pvedaemon worker


└─18349 pvedaemon worker





Jan 11 18:05:44 xx systemd[1]: Starting PVE API Daemon...


Jan 11 18:05:45 xx pvedaemon[18346]: starting server


Jan 11 18:05:45 xx pvedaemon[18346]: starting 3 worker(s)


Jan 11 18:05:45 xx pvedaemon[18346]: worker 18347 started


Jan 11 18:05:45 xx pvedaemon[18346]: worker 18348 started


Jan 11 18:05:45 xx pvedaemon[18346]: worker 18349 started


Jan 11 18:05:45 xx systemd[1]: Started PVE API Daemon.





● pveproxy.service - PVE API Proxy Server


Loaded: loaded (/lib/systemd/system/pveproxy.service; enabled; vendor preset: enabled)


Active: failed (Result: exit-code) since Sun 2020-01-12 00:36:48 +03; 48s ago


Process: 29064 ExecStartPre=/usr/bin/pvecm updatecerts --silent (code=exited, status=0/SUCCESS)


Process: 29077 ExecStart=/usr/bin/pveproxy start (code=exited, status=255/EXCEPTION)





Jan 12 00:36:48 xx systemd[1]: pveproxy.service: Service RestartSec=100ms expired, scheduling restart.


Jan 12 00:36:48 xx systemd[1]: pveproxy.service: Scheduled restart job, restart counter is at 5.


Jan 12 00:36:48 xx systemd[1]: Stopped PVE API Proxy Server.


Jan 12 00:36:48 xx systemd[1]: pveproxy.service: Start request repeated too quickly.


Jan 12 00:36:48 xx systemd[1]: pveproxy.service: Failed with result 'exit-code'.


Jan 12 00:36:48 xx systemd[1]: Failed to start PVE API Proxy Server.
 
removal would be more interesting.. Could just post:
Code:
tail -50 /var/log/apt/history.log
if it happened recently.



Can you retry with: systemctl start pvedaemon.service pveproxy.service


Weird, directory permissions seem OK, pveproxy does not runs in a frozen state or the like either..

if above start does not work post also (besides the apt history): systemctl status pvedaemon.service pveproxy.service
I rebooted server.
Pveproxy working but gui not working.

root@xx:~# service pveproxy status
● pveproxy.service - PVE API Proxy Server
Loaded: loaded (/lib/systemd/system/pveproxy.service; enabled; vendor preset: enabled)
Active: active (running) since Sun 2020-01-12 06:15:19 +03; 1min 22s ago
Process: 5099 ExecStartPre=/usr/bin/pvecm updatecerts --silent (code=exited, status=0/SUCCESS)
Process: 5127 ExecStart=/usr/bin/pveproxy start (code=exited, status=0/SUCCESS)
Main PID: 5146 (pveproxy)
Tasks: 4 (limit: 4915)
Memory: 125.9M
CGroup: /system.slice/pveproxy.service
├─5146 pveproxy
├─6253 pveproxy worker
├─6254 pveproxy worker
└─6255 pveproxy worker

Jan 12 06:16:39 xx pveproxy[5146]: worker 6189 finished
Jan 12 06:16:39 xx pveproxy[5146]: starting 2 worker(s)
Jan 12 06:16:39 xx pveproxy[5146]: worker 6254 started
Jan 12 06:16:39 xx pveproxy[5146]: worker 6255 started
Jan 12 06:16:39 xx pveproxy[6254]: Can't stat /usr/share/javascript: Permission denied
at /usr/share/perl5/PVE/APIServer/AnyEvent.pm line 1759.
Jan 12 06:16:39 xx pveproxy[6254]: Can't stat /usr/share/fonts/truetype/glyphicons: Permission denied
at /usr/share/perl5/PVE/APIServer/AnyEvent.pm line 1759.
Jan 12 06:16:39 xx pveproxy[6255]: Can't stat /usr/share/javascript: Permission denied
at /usr/share/perl5/PVE/APIServer/AnyEvent.pm line 1759.
Jan 12 06:16:39 xx pveproxy[6255]: Can't stat /usr/share/fonts/truetype/glyphicons: Permission denied
at /usr/share/perl5/PVE/APIServer/AnyEvent.pm line 1759.
Jan 12 06:16:39 xx pveproxy[6254]: Can't locate AnyEvent/Log.pm: /usr/local/lib/x86_64-linux-gnu/perl/5.28.1/AnyEvent/Log.pm: Permission denied at /usr/lib/x86_64-linux-gnu/perl5/5.28/AnyEvent.pm line 1331.
Jan 12 06:16:39 xx pveproxy[6255]: Can't locate AnyEvent/Log.pm: /usr/local/lib/x86_64-linux-gnu/perl/5.28.1/AnyEvent/Log.pm: Permission denied at /usr/lib/x86_64-linux-gnu/perl5/5.28/AnyEvent.pm line 1331.
 
Here is the permission
Code:
root@xx:~# ls -lah /usr/share/javascript/
total 32K
drwxr-xr-x   8 root root 4.0K Sep 22 03:11 .
drwxr-xr-x 135 root root 4.0K Jan 11 14:30 ..
drwxr-xr-x   6 root root 4.0K Sep 22 03:11 bootstrap
drwxr-xr-x   5 root root 4.0K Sep 22 03:11 extjs
lrwxrwxrwx   1 root root   21 Jul  5  2018 font-awesome -> ../fonts-font-awesome
drwxr-xr-x   2 root root 4.0K Sep 22 03:11 jquery
drwxr-xr-x   4 root root 4.0K Dec  7 05:15 proxmox-widget-toolkit
drwxr-xr-x   3 root root 4.0K Sep 22 03:11 sphinxdoc
drwxr-xr-x   2 root root 4.0K Sep 22 03:11 underscore
 
Last edited by a moderator:
Your removal of the megaraid package surely did the damange, so it'd be still good to know how you tried to remove it..
You have missing perl modules, strange permission issues, ...

You maybe can come out of this be reinstalling a bunch of packages.. As a starter go with:
apt install --reinstall pve-manager libpve-http-server-perl libjs-extjs proxmox-widget-toolkit libpve-common-perl
 
I've reproduced the issue, and finally found that all folders at / have lost execute permission.
Simply add x permission to them, and the problem should be fixed:
Bash:
chmod +x /*
And thanks t.lamprecht for reminding me,
there is a bit more permissions need to edit:
Bash:
chmod 0700 /root
chmod 1755 /tmp
REMEMBER TO CHECK EVERY FOLDER AND FILE IN / TO ENSURE THEIR PERMISSIONS ARE CORRECT AND SECURE, NOT ONLY TWO OF THEM MENTIONED ABOVE.
Hope this helps.
 
Last edited:
Thansk for sharing what helped you in such a case, a few comments for others though.
I've reproduced the issue, and finally found that all folders at / have lost execute permission.
Simply add x permission to them, and the problem should be fixed:
Bash:
chmod +x /*
Hope this helps.
First, the symptoms can have various causes, this doesn't need to be the single source of such issues and directories do not just loose there execution permission, so try to remember what was done before the problem started, it's probably related to the actual cause.

Second, with that you gave non-root user permission to read the directory contents of /root (the super-users home directory), that could be deemed as information leak or worse, depending on what files are actually located in /root.

I'd recommend removing that extra permission:
Bash:
chmod go-x /root

Often you should also be able to get away by reinstalling the package that provides the base hierarchy, normally safer too:
Bash:
apt install --reinstall base-files
 
Thansk for sharing what helped you in such a case, a few comments for others though.

First, the symptoms can have various causes, this doesn't need to be the single source of such issues and directories do not just loose there execution permission, so try to remember what was done before the problem started, it's probably related to the actual cause.

Second, with that you gave non-root user permission to read the directory contents of /root (the super-users home directory), that could be deemed as information leak or worse, depending on what files are actually located in /root.

I'd recommend removing that extra permission:
Bash:
chmod go-x /root

Often you should also be able to get away by reinstalling the package that provides the base hierarchy, normally safer too:
Bash:
apt install --reinstall base-files
Thanks for reminding me that, and I've edited the comment above.

Here's what I've done and the issue occurs:
MSM is provided as csh script and rpm package, can be downloaded here:
17.05.02.01_MSM_Linux-x64.zip
First, I installed csh and alien (for converting rpm to deb) by apt.
Then, I've edited RunRPM.sh(in that archive), replaced rpm -ivh xxx.rpm / rpm -Uvh xxx.rpm to alien -iv xxx.rpm --script,and deleted rpm -i --test.
Run install.csh.
Then I've removed those packages by dpkg --remove xxx.
At this point I've found that issue, but I don't know which step that actually cause the problem.
Thanks again for reminding me security problems and a better way to fix the problem.
 
  • Like
Reactions: t.lamprecht

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!