Hi All,
I have a 3 node cluster running v7.3-3 and access the GUI behind a traefik instance using a wildcard certificate (traefik load balances between the nodes when i visit proxmox.local.example.com). Everything works except for the VM consoles, however if I access the nodes individually, the consoles work fine. On the proxmox side, I have already tried adding all 3 hosts into the /etc/hosts file, and I verified that I can SSH to each node, from each other as well (passwordless). On the traefik side, I have tried creating a TCP router/service for port 5900 and adding in:
under my default-headers middleware. Here's some logs from my nodes:
Node 1:
Node 2:
Any help is appreciated, and I can post my traefik config files if needed. Thanks!
I have a 3 node cluster running v7.3-3 and access the GUI behind a traefik instance using a wildcard certificate (traefik load balances between the nodes when i visit proxmox.local.example.com). Everything works except for the VM consoles, however if I access the nodes individually, the consoles work fine. On the proxmox side, I have already tried adding all 3 hosts into the /etc/hosts file, and I verified that I can SSH to each node, from each other as well (passwordless). On the traefik side, I have tried creating a TCP router/service for port 5900 and adding in:
Code:
customRequestHeaders:
X-Forwarded-Proto: httpunder my default-headers middleware. Here's some logs from my nodes:
Node 1:
Code:
Dec 21 13:14:24 hyrule pvedaemon[2265404]: <root@pam> starting task UPID:hyrule:0022D54B:0418BD10:63A34D00:vncproxy:103:root@pam:
Dec 21 13:14:24 hyrule pvedaemon[2282827]: starting vnc proxy UPID:hyrule:0022D54B:0418BD10:63A34D00:vncproxy:103:root@pam:
Dec 21 13:14:30 hyrule pmxcfs[914]: [status] notice: received log
Dec 21 13:14:34 hyrule pvedaemon[2282827]: connection timed out
Dec 21 13:14:34 hyrule pvedaemon[2265404]: <root@pam> end task UPID:hyrule:0022D54B:0418BD10:63A34D00:vncproxy:103:root@pam: connection timed out
Code:
root@hyrule:~# pveversion -v
proxmox-ve: 7.3-1 (running kernel: 5.15.74-1-pve)
pve-manager: 7.3-3 (running version: 7.3-3/c3928077)
pve-kernel-helper: 7.3-1
pve-kernel-5.15: 7.2-14
pve-kernel-5.15.74-1-pve: 5.15.74-1
ceph-fuse: 15.2.17-pve1
corosync: 3.1.7-pve1
criu: 3.15-1+pve-1
glusterfs-client: 9.2-1
ifupdown2: 3.1.0-1+pmx3
ksm-control-daemon: 1.4-1
libjs-extjs: 7.0.0-1
libknet1: 1.24-pve2
libproxmox-acme-perl: 1.4.3
libproxmox-backup-qemu0: 1.3.1-1
libpve-access-control: 7.2-5
libpve-apiclient-perl: 3.2-1
libpve-common-perl: 7.3-1
libpve-guest-common-perl: 4.2-3
libpve-http-server-perl: 4.1-5
libpve-storage-perl: 7.3-1
libspice-server1: 0.14.3-2.1
lvm2: 2.03.11-2.1
lxc-pve: 5.0.0-3
lxcfs: 4.0.12-pve1
novnc-pve: 1.3.0-3
proxmox-backup-client: 2.3.1-1
proxmox-backup-file-restore: 2.3.1-1
proxmox-mini-journalreader: 1.3-1
proxmox-widget-toolkit: 3.5.3
pve-cluster: 7.3-1
pve-container: 4.4-2
pve-docs: 7.3-1
pve-edk2-firmware: 3.20220526-1
pve-firewall: 4.2-7
pve-firmware: 3.6-1
pve-ha-manager: 3.5.1
pve-i18n: 2.8-1
pve-qemu-kvm: 7.1.0-4
pve-xtermjs: 4.16.0-1
qemu-server: 7.3-1
smartmontools: 7.2-pve3
spiceterm: 3.2-2
swtpm: 0.8.0~bpo11+2
vncterm: 1.7-1
zfsutils-linux: 2.1.6-pve1Node 2:
Code:
Dec 21 13:13:22 lorule pvedaemon[2380748]: <root@pam> starting task UPID:lorule:002E44DD:03251B08:63A34CC2:vncproxy:110:root@pam:
Dec 21 13:13:32 lorule pvedaemon[3032285]: command '/usr/bin/termproxy 5900 --path /vms/110 --perm VM.Console -- /usr/bin/dtach -A /var/run/dtach/vzctlconsole110 -r winch -z lxc-console -n 110 -e -1' failed: exit code 1
Dec 21 13:13:32 lorule pvedaemon[2380748]: <root@pam> end task UPID:lorule:002E44DD:03251B08:63A34CC2:vncproxy:110:root@pam: command '/usr/bin/termproxy 5900 --path /vms/110 --perm VM.Console -- /usr/bin/dtach -A /var/run/dtach/vzctlconsole110 -r winch -z lxc-console -n 110 -e -1' failed: exit code 1
Dec 21 13:13:32 lorule pvedaemon[3032365]: starting lxc termproxy UPID:lorule:002E452D:03251F15:63A34CCC:vncproxy:110:root@pam:
Dec 21 13:13:32 lorule pvedaemon[3019971]: <root@pam> starting task UPID:lorule:002E452D:03251F15:63A34CCC:vncproxy:110:root@pam:
Dec 21 13:13:42 lorule pvedaemon[3032365]: command '/usr/bin/termproxy 5900 --path /vms/110 --perm VM.Console -- /usr/bin/dtach -A /var/run/dtach/vzctlconsole110 -r winch -z lxc-console -n 110 -e -1' failed: exit code 1
Dec 21 13:13:42 lorule pvedaemon[3019971]: <root@pam> end task UPID:lorule:002E452D:03251F15:63A34CCC:vncproxy:110:root@pam: command '/usr/bin/termproxy 5900 --path /vms/110 --perm VM.Console -- /usr/bin/dtach -A /var/run/dtach/vzctlconsole110 -r winch -z lxc-console -n 110 -e -1' failed: exit code 1
Dec 21 13:13:43 lorule pmxcfs[1091]: [status] notice: received log
Dec 21 13:13:53 lorule pmxcfs[1091]: [status] notice: received log
Dec 21 13:13:54 lorule pvedaemon[3032565]: starting lxc termproxy UPID:lorule:002E45F5:03252774:63A34CE2:vncproxy:110:root@pam:
Dec 21 13:13:54 lorule pvedaemon[2380748]: <root@pam> starting task UPID:lorule:002E45F5:03252774:63A34CE2:vncproxy:110:root@pam:
Dec 21 13:13:59 lorule pvedaemon[3019971]: <root@pam> starting task UPID:lorule:002E460C:03252963:63A34CE7:vncproxy:103:root@pam:
Dec 21 13:13:59 lorule pvedaemon[3032588]: starting vnc proxy UPID:lorule:002E460C:03252963:63A34CE7:vncproxy:103:root@pam:
Dec 21 13:14:02 lorule pveproxy[3029787]: Clearing outdated entries from certificate cache
Dec 21 13:14:04 lorule pvedaemon[3032565]: command '/usr/bin/termproxy 5900 --path /vms/110 --perm VM.Console -- /usr/bin/dtach -A /var/run/dtach/vzctlconsole110 -r winch -z lxc-console -n 110 -e -1' failed: exit code 1
Dec 21 13:14:04 lorule pvedaemon[2380748]: <root@pam> end task UPID:lorule:002E45F5:03252774:63A34CE2:vncproxy:110:root@pam: command '/usr/bin/termproxy 5900 --path /vms/110 --perm VM.Console -- /usr/bin/dtach -A /var/run/dtach/vzctlconsole110 -r winch -z lxc-console -n 110 -e -1' failed: exit code 1
Dec 21 13:14:09 lorule pvedaemon[3032588]: connection timed out
Dec 21 13:14:09 lorule pvedaemon[3019971]: <root@pam> end task UPID:lorule:002E460C:03252963:63A34CE7:vncproxy:103:root@pam: connection timed out
Dec 21 13:14:24 lorule pmxcfs[1091]: [status] notice: received log
Dec 21 13:14:30 lorule pmxcfs[1091]: [status] notice: received log
Dec 21 13:14:34 lorule pmxcfs[1091]: [status] notice: received log
Dec 21 13:17:01 lorule CRON[3033645]: pam_unix(cron:session): session opened for user root(uid=0) by (uid=0)
Dec 21 13:17:01 lorule CRON[3033646]: (root) CMD ( cd / && run-parts --report /etc/cron.hourly)
Dec 21 13:17:01 lorule CRON[3033645]: pam_unix(cron:session): session closed for user root
Dec 21 13:18:03 lorule pvedaemon[3012447]: <root@pam> end task UPID:lorule:002E3BAC:03248245:63A34B3B:vncproxy:110:root@pam: OK
Dec 21 13:18:08 lorule pmxcfs[1091]: [status] notice: received log
Dec 21 13:18:08 lorule pveproxy[3032125]: connect to 'localhost:5900' failed: Connection refused at /usr/share/perl5/PVE/APIServer/AnyEvent.pm line 552.
Dec 21 13:18:13 lorule pveproxy[3032125]: worker exit
Dec 21 13:18:13 lorule pveproxy[1254]: worker 3032125 finishedAny help is appreciated, and I can post my traefik config files if needed. Thanks!