Stuggling with noVNC/xterm.js behind pfSense/HAProxy

[Krambjas]

Looking for some help with an issue I've been struggling with:

I have a new installation of Proxmox 8.0.4 on hardware located behind a pfSense firewall. The pfSense firewall has HAProxy installed on it in order to provide SSL certificates to all LAN servers and to redirect certain DNS names to certain ports (removing the need to remember different ports and IPs for every service). The HAProxy setup is working great, and I can access the Proxmox website and GUI just fine (and other servers). What doesn't work well is trying to bring up a console shell on a Proxmox node, VM, or Container.

If I connect to Proxmox using the DNS name with no port, essentially redirecting port 443 to 8006 through HAProxy, when I click on the _Shell button, and start up noVNC, the console comes up just fine. But it spontaneously disconnects anywhere from 25-60 seconds later. Same with xterm.js, where I get an disconnect with an error 1006.

If I connect to Proxmox using the DNS name and explictly use port 8006, I can bring up noVNC or xterm.js and it stays connected indefinitely. So the issue defintiely appears to be my HAProxy config.

When the disconnection happens, I get a log entry for HAProxy that looks like:
viceroy haproxy[33532]: 192.168.10.100:54911 [11/Nov/2023:02:54:18.340] https_shared-merged~ Dagobah_ipvANY/dagobah 0/0/3/3/30194 101 4329 - - cD-- 3/3/0/0/0 0/0 "GET /api2/json/nodes/dagobah/vncwebsocket?port=5900&vncticket=PVEVNCxxxxxx%3D%3D HTTP/1.1"

where viceroy is the pfsense/haproxy server and dagobah is the proxmox host, and the vncticket is a long keystring. So it appears that a websocket is trying to open using the vncwebsocket api after a period of time and doesn't transfer correctly.

This seems like a conflict between HAproxy and Proxmox with websockets, but all the responses say that HAProxy should handle websockets just fine. I've tried a number of different configs I've found in the these forums to get that websocket to work with no luck. I've attached my haproxy.cfg file to see if there's something I'm missing. Any guidance or ideas would be helpful.

Regards,
Jason

 

[Joe Baker]

Very interesting. I'm looking to put an email server behind HAProxy and thought I'd have to hand configure it. Didn't know pfSense had this feature.

Here's an AI response to your problem...

Based on your description, it seems like your Proxmox setup behind a pfSense firewall using HAProxy is experiencing issues with maintaining a stable connection when using noVNC and xterm.js, leading to spontaneous disconnections. The error code 1006 and the log entry you provided give some insights into potential issues.

Here are some aspects to consider and troubleshoot:

1. WebSocket Connection: Error 1006 indicates that the WebSocket connection is being closed abnormally. This can occur if the WebSocket protocol isn't being handled correctly through the proxy. Ensure that HAProxy is configured to properly forward WebSocket connections. You might need to add specific configuration settings in HAProxy for WebSocket traffic, especially for ports used by noVNC and xterm.js.
2. HAProxy Configuration: Verify your HAProxy configuration, especially the timeout settings. Insufficient timeout durations might be causing the disconnections. Adjust the timeout client, timeout server, and timeout http-keep-alive settings to allow longer persistent connections.
3. Proxmox Ports: Proxmox typically uses port 8006 for accessing its web interface over HTTPS. However, for noVNC, it might use different ports for WebSocket connections (like 5900 as seen in your log). Ensure that these additional ports are properly forwarded and allowed through your firewall and HAProxy setup. It's important to allow both the standard port (8006) and any additional WebSocket ports used by Proxmox.
4. SSL/TLS Termination: If HAProxy is handling SSL certificates and performing SSL termination, ensure that it is configured to correctly pass through WebSocket traffic. Improper SSL termination might disrupt WebSocket connections.
5. Firewall Rules: Double-check your pfSense firewall rules to ensure that all necessary ports for Proxmox and noVNC/xterm.js are open and properly routed.
6. Proxmox Logs: Besides HAProxy logs, check Proxmox logs for any additional error messages or warnings that might give clues about the disconnections.
7. Network Stability: Ensure that your network is stable and that there are no underlying issues causing packet loss or connection instability, which could affect WebSocket connections.
8. Client-Side Issues: Sometimes, the issue might be on the client side. Check if different browsers or clients exhibit the same behavior. Client-side firewall or network settings might also interfere with WebSocket connections.
9. HAProxy Versions and Features: Ensure that the version of HAProxy you're using supports all necessary features, especially those related to WebSocket handling.

By systematically going through these areas, you should be able to pinpoint the cause of the disconnections. If you're still facing issues, consider reaching out to communities or forums dedicated to Proxmox, pfSense, or HAProxy for more specialized assistance.

 

[Aaton35]

@Krambjas - Not sure if you ever figured this out but I've been fighting with it for quite some time. I hit ChatGPT hard, and while it gave me several valuable recommendations, what seems to have fixed it was some good old-fashioned Googling.

The Haproxy website has a KB article that says you should add the config entry: timeout tunnel 1h

I added this in my pfsense backend HTTP server (that I am using for my HTTP connections to port 8006) under the advanced settings - Backend pass thru.

As soon as I added this, my VNC connections stopped refreshing every 20 seconds.

Hope this works for you!