[SOLVED] Web Interface not accessible by some LAN clients.

[Fflam]

Hello,

I Have had a Proxmox VE 8.1.4 server running for a few months now with no issues. I was able to access the Web Interface from all the computers on the LAN that I have needed to. Over the last week I got another server (more hardware) and set up Proxmox VE 8.2.2 on that and it is also working well.

I recently moved and the servers are staying in their current location, so I will no longer have 24/7 local access to the servers. with this in mind I set up a Wireguard VPN on a OpnSense VM on the 8.1.4 server (OpnSense has been set up from day 1 and has been acting as firewall/router with no issues from day 1). The VPN connects, and I am able to access all of the individual VM's, I am able to access the WAN thugh the VPN (whatsmyip shows server IP not client offsite compter IP). The issue I am having is the VPN client is able to access the 8.2.2 Web interface just fine, however it cannot access the web interface of the 8.1.4 server. The vpn client comuter also cannot ping the 8.1.4 server, but it can ping the 8.2.2 as well.

I tested spining up a Mint VM on the 8.2.2 server and giving it a local ip. that computer can access the Web interface (and ping) both the 8.1.4 and 8.2.2 servers. Also if teh VPN client (laptop) is harwired into the LAN on site it can access both servers as well.

Fail2Ban is installed on both servers. I checked the logs (/var/log/fail2ban.log ) and there are no banned IPs.

any ideas where to look next?

 

[sw-omit]

Is the network-adaptor that the opnsense-lan is on the same that the proxmox 8.1.4 management-IP is configured on? Thinking about if there's maybe some kind of "loop-to-self" being strange.
Also, can the OpnSense server itself ping the 8.1.4? And the 8.1.4, can it ping/trace the VPN-client or the gateway-IP of the VPN-segment?

(Sidenote, maybe the "Networking and Firewall" forum might have been slightly better for this: https://forum.proxmox.com/forums/proxmox-ve-networking-and-firewall.17/ )

 

[Fflam]

Hi and thank you for the response,

OPNsense server can ping all servers, and VMs on the local network. all server and VMs can ping all other servers and VMs. I also have a 3rd physical port on the 8.1.4 server (the one with the OPNsence VMs) that i can plug a laptop into to get local emergency access to the net, and that port can ping and connect to all servers and VM's.

however if I VPN in that computer can ping and access all of the VM's and it can ping and access server 8.2.2. however it can not ping or access the proxmox server with 8.1.4 on it. It seems strange (HA! all computer issues are a little strange.) that the wireguard VPN that is a VM on the 8.1.4 server can ping all the VMs on that server, but not the host server it self.

the networking and firewall form could be a better spot to ask for help, but to me it allmost seems like there is something blacklisted on the 8.1.4 server, and not the 8.2.2 server. I would be happy to move my post if you think that is a better spot.

 

[sw-omit]

Not even sure if you can move it or if a mod would need to, and there are plenty of network-people visiting this forum as well I would guess (and it's not like a newbie like me makes the rules )

My thoughts were that network-traffic from the VPN to the VM's on 8.1.4 will probably never leave the software-network, whereas to other servers it goes to the hardware switch. To the proxmox-server is a "strange" one in this regard, although I also would guess it should just work.

Three other questions:
I know you said the OpnSense worked since day one, but have you double-checked that the default-gateway of your proxmox-server is also set to this OpnSense VM?
That third port, does it have it's own bridge/IP, and is it in the same range as the LAN? Would you be able to connect that port to lan and are you then able to ping this new IP over the vpn?
If you make a portforward to 8006 on the OpnSense (of course restricted to your secondary location's current IP, or for testing your mobile network's IP), does that work? Both would be coming from a non-LAN range, but one's VPN and other is NAT, just to see if it's more a routing or more a protocol issue.

Also, never answered if the 8.1.4 could ping any of the VPN-IPs

 

[Fflam]

My thoughts were that network-traffic from the VPN to the VM's on 8.1.4 will probably never leave the software-network, whereas to other servers it goes to the hardware switch. To the proxmox-server is a "strange" one in this regard, although I also would guess it should just work.

I do not have a separate hardware switch. I made a virtual bridge to one of the ports in Proxmox and passed that through to all of the VMs. all the ports are passed to OPNsence, then that manages the traffic between ports (1 WAN, 1 to another physical server, 1 as an emergency local connection)

I know you said the OpnSense worked since day one, but have you double-checked that the default-gateway of your proxmox-server is also set to this OpnSense VM?

honestly this might be the issue. I had 1 extra port set up as a 2nd out side ip to connect from not on the LAN as a temp measure. gateway is still set to that not the local gateway. though it is a different physical port on the server. ill test this latter. But if other VMs on the LAN (both on the 8.1.4 server and the 8.2.2 server) can connect to the Local IP of 8.1.4 should it mater? this is there my Networking knowledge gets weak.

That third port, does it have it's own bridge/IP, and is it in the same range as the LAN? Would you be able to connect that port to lan and are you then able to ping this new IP over the vpn?

as states above its pased though to OPNsence and whats connected to it is assigned an IP though DHCP

If you make a portforward to 8006 on the OpnSense (of course restricted to your secondary location's current IP, or for testing your mobile network's IP), does that work? Both would be coming from a non-LAN range, but one's VPN and other is NAT, just to see if it's more a routing or more a protocol issue.

I can try this as well.

Also, never answered if the 8.1.4 could ping any of the VPN-IPs

no noting can ping the VPN-IPs while the laptops VPN link is active.

again thanks for the help. I will do some more testing.

 

[sw-omit]

honestly this might be the issue. I had 1 extra port set up as a 2nd out side ip to connect from not on the LAN as a temp measure. gateway is still set to that not the local gateway. though it is a different physical port on the server. ill test this latter. But if other VMs on the LAN (both on the 8.1.4 server and the 8.2.2 server) can connect to the Local IP of 8.1.4 should it mater? this is there my Networking knowledge gets weak.

From the UI you should only be able to configure 1 Gateway for a reason, as if there are multiple, how would the server know where to go for any IP's it doesn't recognise?
Traffic coming from the VPN is coming from another IP-range then the local lan, so the server will always return the traffic to it's default gateway, and because that is the same server as the VPN-Server, OpnSense knows how to route the traffic back to you. If it however routes to a different / external gateway, that gateway doesn't know where to find that (internal-only) network, so it drops the traffic. Traffic that IS inside of the local network though doesn't need to go to any gateway, and it will just broadcast on the LAN, find the connection and send the data, that's why LAN works and VPN (and if added NAT/Port-forward) would not. It does not matter that it is a different port, it will just try to find a way outside through any port with a default gateway (partly simplified)
The only way for this setup to work with two or a different default gateway, would be also to set up a static route on the PVE, saying that, to reach the IP-Range of the VPN, it should contact the Opnsense router next/first, if it's not for that network or lan, THEN go to the external default gateway.
If you right now would do "traceroute ip.of.the.VPN" I would expect the first "hop" to be the wrong default gateway and also "traceroute 1.1.1.1" to go to that external service instead of the OpnSense.


Also, didn't know/expect the two servers to be just directly connected without Switching equipment, although it should work still though, just a less frequent setup in my field so didn't think about it right away.

edit: typo's and additions

 

[Fflam]

That Solved the issue.

it was exactly what you said, the Gateway on the 8.1.4 PVE. once I fixed the Gateway everything is working great.

Again thank you for your assistance!

 

[sw-omit]

Good to hear, enjoy your (remotely accessible) Proxmox-servers