Proxmox web gui crashes after couple of minutes

[OfficerCat]

Hey,

I am currently running into an issue where the Proxmox Web GUI stops working after a couple of minutes.
The built in console is not usuable at all during the time tho.

Sometimes reloading the page fixes the issue with the non functioning gui (console still doesnt work)

Proxmox itself tells me "Connection error - server offline?" on the GUI

I attached to log of the pveproxy service and the pvedaemon service.

Since i am using BTRFS on this server, i did a scrub in advance and it showed no errors.

If someone has some ideas, i would be greatful

Spoiler: pvedaemon

-- Boot a3018c4c7c65408e9f7a63ee2bec3bea --
Aug 05 10:55:51 kfswarm systemd[1]: Starting PVE API Daemon...
Aug 05 10:55:52 kfswarm pvedaemon[1435]: starting server
Aug 05 10:55:52 kfswarm pvedaemon[1435]: starting 3 worker(s)
Aug 05 10:55:52 kfswarm pvedaemon[1435]: worker 1436 started
Aug 05 10:55:52 kfswarm pvedaemon[1435]: worker 1437 started
Aug 05 10:55:52 kfswarm pvedaemon[1435]: worker 1438 started
Aug 05 10:55:52 kfswarm systemd[1]: Started PVE API Daemon.
Aug 05 11:09:14 kfswarm pvedaemon[1438]: <root@pam> successful auth for user 'root@pam'
Aug 05 11:09:24 kfswarm pvedaemon[1437]: <root@pam> successful auth for user 'root@pam'
Aug 05 11:09:24 kfswarm pvedaemon[45429]: starting vnc proxy UPID:kfswarm:0000B175:000140DE:62ECDE44:vncproxy:103:root@pam:
Aug 05 11:09:24 kfswarm pvedaemon[1437]: <root@pam> starting task UPID:kfswarm:0000B175:000140DE:62ECDE44:vncproxy:103:root@pam:
Aug 05 11:09:34 kfswarm pvedaemon[45429]: connection timed out
Aug 05 11:09:34 kfswarm pvedaemon[1437]: <root@pam> end task UPID:kfswarm:0000B175:000140DE:62ECDE44:vncproxy:103:root@pam: connection timed out
Aug 05 11:10:23 kfswarm pvedaemon[47992]: starting lxc termproxy UPID:kfswarm:0000BB78:000157B8:62ECDE7F:vncproxy:101:root@pam:
Aug 05 11:10:23 kfswarm pvedaemon[1437]: <root@pam> starting task UPID:kfswarm:0000BB78:000157B8:62ECDE7F:vncproxy:101:root@pam:
Aug 05 11:10:25 kfswarm pvedaemon[48004]: starting lxc termproxy UPID:kfswarm:0000BB84:0001587D:62ECDE81:vncproxy:110:root@pam:
Aug 05 11:10:25 kfswarm pvedaemon[1438]: <root@pam> starting task UPID:kfswarm:0000BB84:0001587D:62ECDE81:vncproxy:110:root@pam:
Aug 05 11:10:33 kfswarm pvedaemon[47992]: command '/usr/bin/termproxy 5900 --path /vms/101 --perm VM.Console -- /usr/bin/dtach -A /var/run/dtach/vzctlconsole101 ->
Aug 05 11:10:33 kfswarm pvedaemon[1437]: <root@pam> end task UPID:kfswarm:0000BB78:000157B8:62ECDE7F:vncproxy:101:root@pam: command '/usr/bin/termproxy 5900 --pat>
Aug 05 11:10:35 kfswarm pvedaemon[48004]: command '/usr/bin/termproxy 5901 --path /vms/110 --perm VM.Console -- /usr/bin/dtach -A /var/run/dtach/vzctlconsole110 ->
Aug 05 11:10:35 kfswarm pvedaemon[1438]: <root@pam> end task UPID:kfswarm:0000BB84:0001587D:62ECDE81:vncproxy:110:root@pam: command '/usr/bin/termproxy 5901 --pat>
Aug 05 11:10:36 kfswarm pvedaemon[48545]: start VM 107: UPID:kfswarm:0000BDA1:00015CE5:62ECDE8C:qmstart:107:root@pam:
Aug 05 11:10:36 kfswarm pvedaemon[1438]: <root@pam> starting task UPID:kfswarm:0000BDA1:00015CE5:62ECDE8C:qmstart:107:root@pam:
Aug 05 11:10:37 kfswarm pvedaemon[1438]: <root@pam> end task UPID:kfswarm:0000BDA1:00015CE5:62ECDE8C:qmstart:107:root@pam: OK
Aug 05 11:29:26 kfswarm pvedaemon[1437]: <root@pam> successful auth for user 'root@pam'
Aug 05 11:36:02 kfswarm pvedaemon[1436]: <root@pam> successful auth for user 'root@pam'
Aug 05 11:36:11 kfswarm pvedaemon[118965]: starting lxc termproxy UPID:kfswarm:0001D0B5:0003B496:62ECE48B:vncproxy:110:root@pam:
Aug 05 11:36:11 kfswarm pvedaemon[1437]: <root@pam> starting task UPID:kfswarm:0001D0B5:0003B496:62ECE48B:vncproxy:110:root@pam:
Aug 05 11:36:21 kfswarm pvedaemon[118965]: command '/usr/bin/termproxy 5900 --path /vms/110 --perm VM.Console -- /usr/bin/dtach -A /var/run/dtach/vzctlconsole110 >
Aug 05 11:36:21 kfswarm pvedaemon[1437]: <root@pam> end task UPID:kfswarm:0001D0B5:0003B496:62ECE48B:vncproxy:110:root@pam: command '/usr/bin/termproxy 5900 --pat>
Aug 05 11:39:02 kfswarm pvedaemon[1436]: <root@pam> starting task UPID:kfswarm:0001F3EB:0003F75B:62ECE536:vncproxy:110:root@pam:
Aug 05 11:39:02 kfswarm pvedaemon[127979]: starting lxc termproxy UPID:kfswarm:0001F3EB:0003F75B:62ECE536:vncproxy:110:root@pam:
Aug 05 11:39:12 kfswarm pvedaemon[127979]: command '/usr/bin/termproxy 5900 --path /vms/110 --perm VM.Console -- /usr/bin/dtach -A /var/run/dtach/vzctlconsole110 >
Aug 05 11:39:12 kfswarm pvedaemon[1436]: <root@pam> end task UPID:kfswarm:0001F3EB:0003F75B:62ECE536:vncproxy:110:root@pam: command '/usr/bin/termproxy 5900 --pat>
Aug 05 11:40:49 kfswarm pvedaemon[1436]: <root@pam> successful auth for user 'root@pam'
lines 948-1003/1003 (END)

Spoiler: pveproxy

-- Boot a3018c4c7c65408e9f7a63ee2bec3bea --
Aug 05 10:55:52 kfswarm systemd[1]: Starting PVE API Proxy Server...
Aug 05 10:55:53 kfswarm pveproxy[1444]: Using '/etc/pve/local/pveproxy-ssl.pem' as certificate for the web interface.
Aug 05 10:55:53 kfswarm pveproxy[1448]: starting server
Aug 05 10:55:53 kfswarm pveproxy[1448]: starting 3 worker(s)
Aug 05 10:55:53 kfswarm pveproxy[1448]: worker 1449 started
Aug 05 10:55:53 kfswarm pveproxy[1448]: worker 1450 started
Aug 05 10:55:53 kfswarm pveproxy[1448]: worker 1451 started
Aug 05 10:55:53 kfswarm systemd[1]: Started PVE API Proxy Server.
Aug 05 11:29:13 kfswarm systemd[1]: Stopping PVE API Proxy Server...
Aug 05 11:29:13 kfswarm pveproxy[1448]: received signal TERM
Aug 05 11:29:13 kfswarm pveproxy[1448]: server closing
Aug 05 11:29:13 kfswarm pveproxy[1449]: worker exit
Aug 05 11:29:13 kfswarm pveproxy[1450]: worker exit
Aug 05 11:29:13 kfswarm pveproxy[1451]: worker exit
Aug 05 11:29:13 kfswarm pveproxy[1448]: worker 1450 finished
Aug 05 11:29:13 kfswarm pveproxy[1448]: worker 1449 finished
Aug 05 11:29:13 kfswarm pveproxy[1448]: worker 1451 finished
Aug 05 11:29:13 kfswarm pveproxy[1448]: server stopped
Aug 05 11:29:14 kfswarm systemd[1]: pveproxy.service: Succeeded.
Aug 05 11:29:14 kfswarm systemd[1]: Stopped PVE API Proxy Server.
Aug 05 11:29:14 kfswarm systemd[1]: pveproxy.service: Consumed 4.351s CPU time.
Aug 05 11:29:14 kfswarm systemd[1]: Starting PVE API Proxy Server...
Aug 05 11:29:15 kfswarm pveproxy[98491]: Using '/etc/pve/local/pveproxy-ssl.pem' as certificate for the web interface.
Aug 05 11:29:15 kfswarm pveproxy[98492]: starting server
Aug 05 11:29:15 kfswarm pveproxy[98492]: starting 3 worker(s)
Aug 05 11:29:15 kfswarm pveproxy[98492]: worker 98493 started
Aug 05 11:29:15 kfswarm pveproxy[98492]: worker 98494 started
Aug 05 11:29:15 kfswarm pveproxy[98492]: worker 98495 started
Aug 05 11:29:15 kfswarm systemd[1]: Started PVE API Proxy Server.
Aug 05 11:29:37 kfswarm systemd[1]: Stopping PVE API Proxy Server...
Aug 05 11:29:38 kfswarm pveproxy[98492]: received signal TERM
Aug 05 11:29:38 kfswarm pveproxy[98492]: server closing
Aug 05 11:29:38 kfswarm pveproxy[98494]: worker exit
Aug 05 11:29:38 kfswarm pveproxy[98493]: worker exit
Aug 05 11:29:38 kfswarm pveproxy[98495]: worker exit
Aug 05 11:29:38 kfswarm pveproxy[98492]: worker 98494 finished
Aug 05 11:29:38 kfswarm pveproxy[98492]: worker 98493 finished
Aug 05 11:29:38 kfswarm pveproxy[98492]: worker 98495 finished
Aug 05 11:29:38 kfswarm pveproxy[98492]: server stopped
Aug 05 11:29:39 kfswarm systemd[1]: pveproxy.service: Succeeded.
Aug 05 11:29:39 kfswarm systemd[1]: Stopped PVE API Proxy Server.
Aug 05 11:29:39 kfswarm systemd[1]: pveproxy.service: Consumed 2.281s CPU time.
Aug 05 11:29:39 kfswarm systemd[1]: Starting PVE API Proxy Server...
Aug 05 11:29:40 kfswarm pveproxy[99503]: Using '/etc/pve/local/pveproxy-ssl.pem' as certificate for the web interface.
Aug 05 11:29:40 kfswarm pveproxy[99510]: starting server
Aug 05 11:29:40 kfswarm pveproxy[99510]: starting 3 worker(s)
Aug 05 11:29:40 kfswarm pveproxy[99510]: worker 99512 started
Aug 05 11:29:40 kfswarm pveproxy[99510]: worker 99513 started
Aug 05 11:29:40 kfswarm pveproxy[99510]: worker 99514 started
Aug 05 11:29:40 kfswarm systemd[1]: Started PVE API Proxy Server.

Spoiler: pveversion

proxmox-ve: 7.2-1 (running kernel: 5.15.39-3-pve)
pve-manager: 7.2-7 (running version: 7.2-7/d0dd0e85)
pve-kernel-5.15: 7.2-8
pve-kernel-helper: 7.2-8
pve-kernel-5.13: 7.1-9
pve-kernel-5.11: 7.0-10
pve-kernel-5.15.39-3-pve: 5.15.39-3
pve-kernel-5.15.39-1-pve: 5.15.39-1
pve-kernel-5.15.35-3-pve: 5.15.35-6
pve-kernel-5.13.19-6-pve: 5.13.19-15
pve-kernel-5.13.19-2-pve: 5.13.19-4
pve-kernel-5.11.22-7-pve: 5.11.22-12
pve-kernel-5.11.22-1-pve: 5.11.22-2
ceph-fuse: 15.2.13-pve1
corosync: 3.1.5-pve2
criu: 3.15-1+pve-1
glusterfs-client: 9.2-1
ifupdown2: 3.1.0-1+pmx3
ksm-control-daemon: 1.4-1
libjs-extjs: 7.0.0-1
libknet1: 1.24-pve1
libproxmox-acme-perl: 1.4.2
libproxmox-backup-qemu0: 1.3.1-1
libpve-access-control: 7.2-4
libpve-apiclient-perl: 3.2-1
libpve-common-perl: 7.2-2
libpve-guest-common-perl: 4.1-2
libpve-http-server-perl: 4.1-3
libpve-storage-perl: 7.2-7
libspice-server1: 0.14.3-2.1
lvm2: 2.03.11-2.1
lxc-pve: 5.0.0-3
lxcfs: 4.0.12-pve1
novnc-pve: 1.3.0-3
proxmox-backup-client: 2.2.5-1
proxmox-backup-file-restore: 2.2.5-1
proxmox-mini-journalreader: 1.3-1
proxmox-widget-toolkit: 3.5.1
pve-cluster: 7.2-2
pve-container: 4.2-2
pve-docs: 7.2-2
pve-edk2-firmware: 3.20210831-2
pve-firewall: 4.2-5
pve-firmware: 3.5-1
pve-ha-manager: 3.4.0
pve-i18n: 2.7-2
pve-qemu-kvm: 6.2.0-11
pve-xtermjs: 4.16.0-1
qemu-server: 7.2-3
smartmontools: 7.2-pve3
spiceterm: 3.2-2
swtpm: 0.7.1~bpo11+1
vncterm: 1.7-1
zfsutils-linux: 2.1.5-pve1

 

[Matthias.]

Are there any relevant messages in the syslog?
Can you connect to the server with SSH?
If you open the dev tools in the browser (press F12), do you see any error messages in the console?
You say the console is not working at all - is there a proxy between you and the Proxmox VE server?

 

[OfficerCat]

Well ... i fixed the issue after looking at the dev tools
Would have never thought to look there.

It was indeed a missconfiguration of my proxy, guess it just decided to not work anymore

Thanks for your help Matthias

 

[Matthias.]

Great! Please mark the thread as solved (edit it, there's a dropdown near the title) so others with the same problem can find it more easily.