Can't access web gui from Linux, but can from macOS/Android

[dalavitang]

Hi, I seem to have come across a very strange problem.

I cannot access the web-ui from my main Fedora workstation even though I can ping and ssh into the pve, but I can access the web-ui from my MacBook and my Android phone without any issue.

I searched around and found that I can troubleshoot a bit by curling from the pve, so I tried on all four of my linux machines (2 Fedora, 1 AlmaLinux, 1 TrueNAS) as well as my OPNsense firewall.

curl -sk https://${pve-ip}:8006

won't succeed on the four linux machines or the OPNsense firewall, but works fine on my MacBook.

I've checked network and firewall settings on my OPNsense and there shouldn't be any issue, as the MacBook share the same firewall rules. I've also tried disabling firewall on my Fedora workstation and the AlmaLinux install, but that didn't help either.

It's not the end of the world because I can still access the web-ui with my MacBook but this is really strange behaviour and it's a bit inconvenient. Anyone has any idea on what's going on or what I should check next?

Thanks

 

[dj423]

Just curious, have you tried netcat from those effected nodes, just to rule out firewall issues?

example:

nc -zv <pve_ip> 8006

 

[dalavitang]

Just curious, have you tried netcat from those effected nodes, just to rule out firewall issues?

example:

nc -zv <pve_ip> 8006

Thanks for your advice, I tried netcat following you recommendation and it did connect to port 8006 of my pve server, but I have some more info on the situation down below:

$nc -zv pve 8006
Ncat: Version 7.92 ( https://nmap.org/ncat )
Ncat: Connected to 192.168.31.253:8006.
Ncat: 0 bytes sent, 0 bytes received in 0.02 seconds.

you’ve already done some solid troubleshooting, so let’s see if we can pinpoint the issue further. One thing to check is SELinux, as it could be blocking access to the port even if the firewall is disabled. You can temporarily set SELinux to permissive mode (setenforce 0) to see if that’s causing the problem. Additionally, it’s worth verifying if there's any proxy configured on your Fedora workstation or if there are DNS resolution discrepancies between your devices. Since your MacBook works fine, this might point to a configuration difference. Another possible culprit could be SSL/TLS versions, as https connections sometimes fail due to mismatched versions between client and server. You might want to check if there’s any SSL-related issue, particularly if Fedora has different SSL libraries or settings. Lastly, while you can ping and SSH into the PVE, it’s still worth checking for any subtle routing or local network issues that could block specific traffic, like to port 8006. It might also help to look through logs like /var/log/firewalld or /var/log/messages for any clues of blocked traffic. Let me know what you find out!

I've disabled SELinux altogether on my Fedora worksation and I still couldn't connect. However I did try curl again with verbose on and I can narrow down the problem a bit. When I curl from Fedora or OPNsense I get the following:

$curl -skv https://pve:8006
* Host pve:8006 was resolved.
* IPv6: (none)
* IPv4: 192.168.31.253
*   Trying 192.168.31.253:8006...
* Connected to pve (192.168.31.253) port 8006
* ALPN: curl offers h2,http/1.1
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* Recv failure: Connection reset by peer
* OpenSSL SSL_connect: Connection reset by peer in connection to pve:8006
* closing connection #0

While from the MacBook:

* Host pve:8006 was resolved.
* IPv6: (none)
* IPv4: 192.168.31.253
*   Trying 192.168.31.253:8006...
* Connected to pve (192.168.31.253) port 8006
* ALPN: curl offers h2,http/1.1
* (304) (OUT), TLS handshake, Client hello (1):
* (304) (IN), TLS handshake, Server hello (2):
* (304) (IN), TLS handshake, Unknown (8):
* (304) (IN), TLS handshake, Certificate (11):
* (304) (IN), TLS handshake, CERT verify (15):
* (304) (IN), TLS handshake, Finished (20):
* (304) (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / AEAD-AES256-GCM-SHA384 / [blank] / UNDEF
* ALPN: server did not agree on a protocol. Uses default.
(Continues...)

So it seems that the connection to pve:8006 was successful, but the server never replied to client hello from Fedora or OPNsense. I tried to stop the firewall on pve:

#systemctl stop proxmox-firewall
#systemctl stop pve-firewall

But it didn't help. What should I do next?

 

[dj423]

So looks like its using TLS 1.3 (* TLSv1.3 (OUT), TLS handshake, Client hello) - seems like the pve host does not like the ssl client perhaps?


Just curious, what version of openssl is on the fedora client? "rpm -qa |grep openssl" - just want to make sure the version support TLS1.3 - just to rule that out. But it's not happy with the fedora client it would seem from that output.

Might not hurt to inspect the pve cert from the fedora client just for grins and validate it looks correct:
"openssl s_client -showcerts -connect pve:8006 -servername pve"

Should see something like:

Post-Handshake New Session Ticket arrived:
SSL-Session:
    Protocol  : TLSv1.3
    Cipher    : TLS_AES_256_GCM_SHA384

 

[dalavitang]

Oh I got it. The problem was MTU and FRAGMENTATION.

When I googled my problem this thread came up:

https://stackoverflow.com/questions/21767003/ssl-handshake-fails-after-clienthello

I read the first answer and immediately it made sense.

I have Jumbo frame enabled on all my Linux workstation and the OPNsense firewall. But my MacBook (due to driver limitations) did not allow me to use Jumbo frame, so it was at 1500 MTU the whole time.

And when I changed one of my Fedora boxes to use 1500 MTU it immediately worked.

I checked my network settings in the pve web-ui and found the network bridge to have 9000 MTU, but when I sshed into the pve and

ip link sh

I noticed the physical interface was still set to use 1500 MTU.

So after I

ip link set <physical-int> mtu 9000

the problem gets resolved.

Thanks for your replies! It turned out to be a dumb oversight from my side.

 

[dj423]

Glad you got it sorted out. Good ol jumbo frames, jumbo the fun! lol