PVE 9.0.5 pveproxy not working, no webgui

K

kamkry18

Member
Mar 26, 2022
2
1
6
Hey folks!
I've upgraded PVE from 8.x (don't remember particular version) to 9.0 without issues. All has been working OK (cluster of 2 nodes) for week or so, but today (after restarting each node) webUI is not responding.
What I've found..:
Bash: 
# systemctl restart pveproxy
Job for pveproxy.service failed because the control process exited with error code.
See "systemctl status pveproxy.service" and "journalctl -xeu pveproxy.service" for details.
root@conductor:~# systemctl restart pvedaemon.service
root@conductor:~# systemctl restart pveproxy
Job for pveproxy.service failed because the control process exited with error code.
See "systemctl status pveproxy.service" and "journalctl -xeu pveproxy.service" for details.
root@conductor:~# systemctl status pveproxy
× pveproxy.service - PVE API Proxy Server
     Loaded: loaded (/usr/lib/systemd/system/pveproxy.service; enabled; preset: enabled)
     Active: failed (Result: exit-code) since Sun 2025-08-17 18:39:54 CEST; 7min ago
 Invocation: 2e93ed9602c94df2a78032e0815ad103
    Process: 89806 ExecStartPre=/usr/bin/pvecm updatecerts --silent (code=exited, status=0/SUCCESS)
    Process: 89812 ExecStart=/usr/bin/pveproxy start (code=exited, status=255/EXCEPTION)
   Mem peak: 143.1M
        CPU: 1.353s

Aug 17 18:39:54 conductor systemd[1]: pveproxy.service: Scheduled restart job, restart counter is at 1.
Aug 17 18:39:54 conductor systemd[1]: pveproxy.service: Start request repeated too quickly.
Aug 17 18:39:54 conductor systemd[1]: pveproxy.service: Failed with result 'exit-code'.
Aug 17 18:39:54 conductor systemd[1]: Failed to start pveproxy.service - PVE API Proxy Server.
Aug 17 18:39:54 conductor systemd[1]: pveproxy.service: Consumed 1.353s CPU time, 143.1M memory peak.

Same problem can be seen for spiceproxy daemon..:

Bash: 
# systemctl status spiceproxy.service
× spiceproxy.service - PVE SPICE Proxy Server
     Loaded: loaded (/usr/lib/systemd/system/spiceproxy.service; enabled; preset: enabled)
     Active: failed (Result: exit-code) since Mon 2025-08-18 07:42:47 CEST; 43min ago
 Invocation: 3c0a1f5cddff488e8930e6c42b695f89
    Process: 2355 ExecStart=/usr/bin/spiceproxy start (code=exited, status=255/EXCEPTION)
   Mem peak: 50.8M
        CPU: 244ms

Aug 18 07:42:47 virtuos systemd[1]: Starting spiceproxy.service - PVE SPICE Proxy Server...
Aug 18 07:42:47 virtuos spiceproxy[2355]: start failed - unable to open lock file '/var/lock/spiceproxy.lck' - Permission denied
Aug 18 07:42:47 virtuos spiceproxy[2355]: start failed - unable to open lock file '/var/lock/spiceproxy.lck' - Permission denied
Aug 18 07:42:47 virtuos systemd[1]: spiceproxy.service: Control process exited, code=exited, status=255/EXCEPTION
Aug 18 07:42:47 virtuos systemd[1]: spiceproxy.service: Failed with result 'exit-code'.
Aug 18 07:42:47 virtuos systemd[1]: Failed to start spiceproxy.service - PVE SPICE Proxy Server.
root@virtuos:~# systemctl status pveproxy.service
× pveproxy.service - PVE API Proxy Server
     Loaded: loaded (/usr/lib/systemd/system/pveproxy.service; enabled; preset: enabled)
     Active: failed (Result: exit-code) since Mon 2025-08-18 08:12:17 CEST; 14min ago
 Invocation: e90558e894a54616a8381611180d7eca
    Process: 9497 ExecStartPre=/usr/bin/pvecm updatecerts --silent (code=exited, status=0/SUCCESS)
    Process: 9500 ExecStart=/usr/bin/pveproxy start (code=exited, status=255/EXCEPTION)
   Mem peak: 142.5M
        CPU: 947ms

more issues found in journalctl..:
Bash: 
Aug 17 18:39:53 conductor systemd[1]: Starting pveproxy.service - PVE API Proxy Server...
░░ Subject: A start job for unit pveproxy.service has begun execution
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░
░░ A start job for unit pveproxy.service has begun execution.
░░
░░ The job identifier is 17930.
Aug 17 18:39:54 conductor pveproxy[89812]: start failed - unable to open lock file '/var/lock/pveproxy.lck' - Permission denied
Aug 17 18:39:54 conductor pveproxy[89812]: start failed - unable to open lock file '/var/lock/pveproxy.lck' - Permission denied
Aug 17 18:39:54 conductor systemd[1]: pveproxy.service: Control process exited, code=exited, status=255/EXCEPTION
░░ Subject: Unit process exited
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░
░░ An ExecStart= process belonging to unit pveproxy.service has exited.
░░
░░ The process' exit code is 'exited' and its exit status is 255.

I don't see /var/lock/pveproxy.lck file being created (with a watchloop) neither when (re)starting pveproxy daemon, nor by manually calling /usr/bin/pveproxy start.
Permissions on /var/lock and /run/lock..:

Bash: 
# ls -lah /var/lock/
total 0
drwxr-xr-x  7 root root  220 Aug 17 17:08 .
drwxr-xr-x 38 root root 1.6K Aug 17 18:39 ..
drwx--x--x  2 root root   40 Aug 17 17:07 lvm
drwxr-xr-x  2 root root   40 Aug 17 17:07 lxc
-rwxr-xr-x  1 root root    0 Aug 17 17:08 pvedaemon.lck
-rwxr-xr-x  1 root root    0 Aug 17 17:07 pvefw.lck
-rwxr-xr-x  1 root root    0 Aug 17 17:07 pvefw-logger.lck
drwxr-xr-x  2 root root   40 Aug 17 17:08 pve-manager
-rwxr-xr-x  1 root root    0 Aug 17 17:08 pvesr.lck
drwxr-xr-x  2 root root  200 Aug 17 17:08 qemu-server
drwxr-xr-x  2 root root   40 Aug 17 17:08 subsys

# ls -lah /run/lock/
total 0
drwxr-xr-x  7 root root  220 Aug 17 17:08 .
drwxr-xr-x 38 root root 1.6K Aug 17 18:39 ..
drwx--x--x  2 root root   40 Aug 17 17:07 lvm
drwxr-xr-x  2 root root   40 Aug 17 17:07 lxc
-rwxr-xr-x  1 root root    0 Aug 17 17:08 pvedaemon.lck
-rwxr-xr-x  1 root root    0 Aug 17 17:07 pvefw.lck
-rwxr-xr-x  1 root root    0 Aug 17 17:07 pvefw-logger.lck
drwxr-xr-x  2 root root   40 Aug 17 17:08 pve-manager
-rwxr-xr-x  1 root root    0 Aug 17 17:08 pvesr.lck
drwxr-xr-x  2 root root  200 Aug 17 17:08 qemu-server
drwxr-xr-x  2 root root   40 Aug 17 17:08 subsys

Any help? All VMs are starting fine, but webUI is not reachable

Bash: 
# curl -vk http://127.0.0.1:8006
*   Trying 127.0.0.1:8006...
* connect to 127.0.0.1 port 8006 from 127.0.0.1 port 50894 failed: Connection refused
* Failed to connect to 127.0.0.1 port 8006 after 0 ms: Could not connect to server
* closing connection #0
curl: (7) Failed to connect to 127.0.0.1 port 8006 after 0 ms: Could not connect to server

root@conductor:~# curl -vk https://127.0.0.1:8006
*   Trying 127.0.0.1:8006...
* connect to 127.0.0.1 port 8006 from 127.0.0.1 port 50896 failed: Connection refused
* Failed to connect to 127.0.0.1 port 8006 after 0 ms: Could not connect to server
* closing connection #0
curl: (7) Failed to connect to 127.0.0.1 port 8006 after 0 ms: Could not connect to server
 
Last edited:
OK, after checking what privileges/accounts are set for other binaries related to either pveproxy or spiceproxy, I've noticed quite an interesting thing.:
Bash: 
# ls -lah /var/run/ | grep 'proxy'
drwx------  2 www-data www-data  120 Aug 18 10:46 pveproxy

That means file permissions inside /var/lock dir aren't (indeed, as logline in journalctl shows) able to do what they're intended to do :)

My fix was to append to /usr/lib/systemd/system/spiceproxy.service and /usr/lib/systemd/system/pveproxy.service following lines.:
Bash: 
[Service]
ExecStartPre=/bin/chmod o+rwx /var/lock
ExecStartPre=/bin/touch /var/lock/spiceproxy.lck
ExecStartPre=/bin/chown www-data:www-data /var/lock/spiceproxy.lck

-- or, for the pveproxy.service file --

[Service]
ExecStartPre=/bin/chmod o+rwx /var/lock
ExecStartPre=/bin/touch /var/lock/pveproxy.lck
ExecStartPre=/bin/chown www-data:www-data /var/lock/pveproxy.lck

and now it is working both after systemctl daemon-reload && systemctl restart pveproxy.service spiceproxy.service and survives the reboot of entire host :)
 
  • Like
Reactions: Swifty.hu
You must log in or register to reply here.
Top Bottom