Suddenly unable to access web UI

M

Maiyannah

New Member
Apr 3, 2026
4
0
1
www.highlandarrow.com
Hi all,
After some time the proxmox UI has become unavalable. It refuses the connection and while it is running, the proxy has nothing in the log.

Things I checked:
  1. Tried both hostname and direct IP address
  2. Restarted pveproxy
  3. Ensured system has up to date APT packages
  4. Tried from multiple POPs.
  5. SSH to the server (this works)
  6. Curl to the URL both local and WAN hangs indefinitely.
  7. All logs I could see looked clean.
Any ideas?

It should be noted the web interface has been perfectly accessible about two weeks ago last I had to do some administrata on the VM end and has been continuously running for a year now (not including some update restarts)
 
If curl hangs rather than refusing outright, pveproxy is likely listening on port 8006 but not completing the TLS handshake. The most common cause after a long uptime is certificate expiry, either the self-signed cert expired or there's a corruption in the PVE cluster certificate store. First verify the port is actually bound:

Code: 
ss -tlnp | grep 8006

If pveproxy appears there, the service is up and listening, something is blocking the handshake, not the service itself.

Check cert expiry:

Code: 
openssl x509 -in /etc/pve/local/pve-ssl.pem -noout -dates

If it's expired, regenerate:

Code: 
pvecm updatecerts --force
systemctl restart pveproxy

Also check pvedaemon, pveproxy depends on it and sometimes the daemon gets stuck without pveproxy logging anything obvious:

Code: 
systemctl status pvedaemon
systemctl restart pvedaemon pveproxy

If the cert is fine, check whether iptables or nftables picked up a rule blocking 8006 (this can happen after kernel updates or if fail2ban is running):

Code: 
iptables -L -n | grep 8006
nft list ruleset | grep 8006

One more place to check: the pveproxy journal rather than the log file, the file often stays quiet while the journal shows TLS errors or connection resets:

Code: 
journalctl -u pveproxy -n 100 --no-pager

Given it was working fine two weeks ago and you've been on continuous uptime for a year, cert expiry is the most likely culprit. PVE's default self-signed certs are valid for 10 years, but if you ever had a custom cert applied it may have a shorter validity.
 
Good shout on certs, it was current however. I did restart it nonetheless, but it had no effect.

Journalctl has nothing at all unusual; it spins up, it makes three workers, and it waits. Thought to check pvedaemon too; same thing.

I don't find any iptables or nft rules affecting 8006.

There were some new versions to upgrade when I hit apt update / upgrade. They installed successfully with no errors.

I tried a restart of the host machine to no effect either.
 
Maiyannah said:
  1. Curl to the URL both local and WAN hangs indefinitely.
Click to expand...
Have you tried 127.0.0.1 ?
It would be helpful if you posted your configuration and commands you run here, rather than just reporting the results.

The output of these commands in text format and encoded with CODE tags is a good start:
pveversion -v
uname -a
uptime
systemctl list-units --failed
systemctl status pve-cluster
systemctl status corosync
systemctl status pvedaemon
systemctl status pveproxy
systemctl status pvestatd
journalctl -b -p err
journalctl -u pve-cluster -b
journalctl -u corosync -b
journalctl -u pvedaemon -b
journalctl -u pveproxy -b
journalctl -u pvestatd -b
pvecm status
pvecm nodes
lsblk
df -h

Cheers

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox
 
So in:

pveversion -v
Code: 
~# pveversion -v
proxmox-ve: 9.1.0 (running kernel: 6.17.4-2-pve)
pve-manager: 9.1.7 (running version: 9.1.7/16b139a017452f16)
proxmox-kernel-helper: 9.0.4
proxmox-kernel-6.17: 6.17.13-2
proxmox-kernel-6.17.13-2-pve-signed: 6.17.13-2
proxmox-kernel-6.17.4-2-pve-signed: 6.17.4-2
proxmox-kernel-6.14: 6.14.11-6
proxmox-kernel-6.14.11-6-pve-signed: 6.14.11-6
ceph-fuse: 19.2.3-pve1
corosync: 3.1.10-pve2
criu: 4.1.1-1
frr-pythontools: 10.4.1-1+pve1
ifupdown: residual config
ifupdown2: 3.3.0-1+pmx12
libjs-extjs: 7.0.0-5
libproxmox-acme-perl: 1.7.0
libproxmox-backup-qemu0: 2.0.2
libproxmox-rs-perl: 0.4.1
libpve-access-control: 9.0.5
libpve-apiclient-perl: 3.4.2
libpve-cluster-api-perl: 9.1.1
libpve-cluster-perl: 9.1.1
libpve-common-perl: 9.1.9
libpve-guest-common-perl: 6.0.2
libpve-http-server-perl: 6.0.5
libpve-network-perl: 1.2.5
libpve-rs-perl: 0.11.4
libpve-storage-perl: 9.1.1
libspice-server1: 0.15.2-1+b1
lvm2: 2.03.31-2+pmx1
lxc-pve: 6.0.5-4
lxcfs: 6.0.4-pve1
novnc-pve: 1.6.0-3
proxmox-backup-client: 4.1.5-1
proxmox-backup-file-restore: 4.1.5-1
proxmox-backup-restore-image: 1.0.0
proxmox-firewall: 1.2.1
proxmox-kernel-helper: 9.0.4
proxmox-mail-forward: 1.0.2
proxmox-mini-journalreader: 1.6
proxmox-offline-mirror-helper: 0.7.3
proxmox-widget-toolkit: 5.1.8
pve-cluster: 9.1.1
pve-container: 6.1.2
pve-docs: 9.1.2
pve-edk2-firmware: not correctly installed
pve-esxi-import-tools: 1.0.1
pve-firewall: 6.0.4
pve-firmware: 3.18-2
pve-ha-manager: 5.1.3
pve-i18n: 3.6.6
pve-qemu-kvm: 10.1.2-7
pve-xtermjs: 5.5.0-3
qemu-server: 9.1.6
smartmontools: 7.4-pve1
spiceterm: 3.4.1
swtpm: 0.8.0+pve3
vncterm: 1.9.1
zfsutils-linux: 2.4.1-pve1

I notice:
pve-edk2-firmware: not correctly installed

I'm not sure what that component is?

Otherwise in systemctl only failed unit appears to be LE certbot. Cert is current though so I dont see how that'd be a problem (at least right now)

I see nothing at all unusual in the logs.

Code: 
# systemctl status pvedaemon
● pvedaemon.service - PVE API Daemon
     Loaded: loaded (/usr/lib/systemd/system/pvedaemon.service; enabled; preset: enabled)
     Active: active (running) since Sun 2026-04-05 01:12:46 UTC; 22min ago
 Invocation: dd57f86dd2f84dcc80ccecc60091f1ba
    Process: 508466 ExecStart=/usr/bin/pvedaemon start (code=exited, status=0/SUCCESS)
   Main PID: 508487 (pvedaemon)
      Tasks: 4 (limit: 154447)
     Memory: 155.1M (peak: 177.3M)
        CPU: 1.736s
     CGroup: /system.slice/pvedaemon.service
             ├─508487 pvedaemon
             ├─508488 "pvedaemon worker"
             ├─508489 "pvedaemon worker"
             └─508490 "pvedaemon worker"

Apr 05 01:12:45 montreal systemd[1]: Starting pvedaemon.service - PVE API Daemon...
Apr 05 01:12:46 montreal pvedaemon[508487]: starting server
Apr 05 01:12:46 montreal pvedaemon[508487]: starting 3 worker(s)
Apr 05 01:12:46 montreal pvedaemon[508487]: worker 508488 started
Apr 05 01:12:46 montreal pvedaemon[508487]: worker 508489 started
Apr 05 01:12:46 montreal pvedaemon[508487]: worker 508490 started
Apr 05 01:12:46 montreal systemd[1]: Started pvedaemon.service - PVE API Daemon.

Code: 
# systemctl status pveproxy
● pveproxy.service - PVE API Proxy Server
     Loaded: loaded (/usr/lib/systemd/system/pveproxy.service; enabled; preset: enabled)
     Active: active (running) since Sun 2026-04-05 01:16:57 UTC; 18min ago
 Invocation: 1be5ddc4b4f24f64b4f15658b776783d
    Process: 509229 ExecStartPre=/usr/bin/pvecm updatecerts --silent (code=exited, status=0/SUCCESS)
    Process: 509233 ExecStart=/usr/bin/pveproxy start (code=exited, status=0/SUCCESS)
    Process: 509258 ExecStartPost=sh -c [ ! -e /var/log/pveam.log ] && /usr/bin/pveupdate (code=exited, status=1/FAILURE)
   Main PID: 509253 (pveproxy)
      Tasks: 4 (limit: 154447)
     Memory: 161.9M (peak: 184.3M)
        CPU: 2.345s
     CGroup: /system.slice/pveproxy.service
             ├─509253 pveproxy
             ├─509254 "pveproxy worker"
             ├─509255 "pveproxy worker"
             └─509256 "pveproxy worker"

Apr 05 01:16:55 montreal systemd[1]: Starting pveproxy.service - PVE API Proxy Server...
Apr 05 01:16:57 montreal pveproxy[509233]: Using '/etc/pve/local/pveproxy-ssl.pem' as certificate for the web interface.
Apr 05 01:16:57 montreal pveproxy[509253]: starting server
Apr 05 01:16:57 montreal pveproxy[509253]: starting 3 worker(s)
Apr 05 01:16:57 montreal pveproxy[509253]: worker 509254 started
Apr 05 01:16:57 montreal pveproxy[509253]: worker 509255 started
Apr 05 01:16:57 montreal pveproxy[509253]: worker 509256 started
Apr 05 01:16:57 montreal systemd[1]: Started pveproxy.service - PVE API Proxy Server.

The failed process is the certificate updater.
Running it manually, it looks like LE's ACME server is not being responsive to the request:

Code: 
~# /usr/bin/pveupdate
Loading ACME account details
Placing ACME order
Order URL: https://acme-v02.api.letsencrypt.org/acme/order/2989786306/497458308601

Getting authorization details from 'https://acme-v02.api.letsencrypt.org/acme/authz/2989786306/683376243031'
The validation for montreal.highlandarrow.com is pending!
Setting up webserver
Triggering validation
Sleeping for 5 seconds
Status is still 'pending', trying again in 10 seconds
Status is still 'pending', trying again in 10 seconds
Status is still 'pending', trying again in 10 seconds
validating challenge 'https://acme-v02.api.letsencrypt.org/acme/authz/2989786306/683376243031' failed - status: invalid

If we think it's that, I could install a simple comodo certificate, and see, they're not expensive. Though I'd need documentation on setting it up with a paid cert.

This is a single bare metal machine, there is no clustering.

Disk has over ten TB remaining so I'd find its doubtful to be that. Again, worth mentioning: the client containers are working just fine:

Code: 
~# qm list
      VMID NAME                 STATUS     MEM(MB)    BOOTDISK(GB) PID
       100 azalin.toud.pw       running    5722            1862.65 2139
       101 ezra.toud.pw         running    11444            931.32 2360
       102 nwysnc.toud.pw       running    2048              32.00 2554
       103 www.toud.pw          running    4096            3725.29 2643
       104 wiki.toud.pw         running    4096            3725.00 22109
       105 git.toud.pw          running    11444           1862.65 2730
       106 forums.toud.pw       running    3814            3725.29 2848
       107 www.highlandarrow.com running    9537             931.32 2978
       108 mail.highlandarrow.com running    7629              74.51 3077
       109 enshrouded.server    running    17166            465.66 3187
       110 chat.toud.pw         running    9537             931.32 3297
       112 git.highlandarrow.com running    11444            465.66 3399
       113 chat.highlandarrow.com running    9536              32.00 3502
       200 montreal.supernerdland.com running    61035           9024.52 3620
       301 chat.housestuart.com running    11444             50.00 3754
       400 www.undermountain.ca running    5587            1862.65 3860
       401 icarus.server        running    9537             512.00 3906

Its the UI that isn't. All of this stuff I can do through the CLI, but I prefer the GUI (and I'd like to figure out why it's not working as that may be indicative of larger problems - though hopefully not)
 
You must log in or register to reply here.
Top Bottom
Back