Not able to access Proxmox VMs in PfSense (LAN-side) via Laptops on "WAN"-side

[John Driessen]

Hi,

I'm not so experienced in the networking part.
I have a PC on which I installed Proxmox, which has only one entry point for a network cable (so only 1-NIC).
The vmbr0 interface is used to be "port/slave" of the actual network interface.
I have defined several VMs / LXCs in Proxmox, that have their IP's defined in Proxmox and are connected to vmbr0.
My IP-range (local network) is 192.168.10.0/24 (with 192.168.10.1 being the Gateway).
The router I have from the ISP provider is defined as the DHCP server.
So also my laptops, mobile devices have a local IP address in that range.
I am able to access the VMs / LXCs (via SSH, TCP/UDP etc) from these laptops, etc, and also vice-versa.
I have set several firewall rules in the Proxmox GUI for these VMs/LXCs. Everything works fine: so far so good.

To transfer in the future some functionality from the router to pfSense and obtain more functionality I have installed a PfSense VM (2.5.1 CE) in Proxmox 6.3.6.
The PfSense WAN interface is connected vmbr0 (and has an IP in the above mentioned local network IP-range)
I created an vmbr1 interface in Proxmox without connection to a real NIC.
The PfSense LAN interface (Gateway defined as 192.168.100.1) connects to this vmbr1 interface.
I have moved some of the LXCs and VMs to use vmbr1 iso vmbr0 and changed their IP-addresses to be in the new range (192.168.100.0/24).
I also created a VM (Kubuntu 20.04) in the new range to be able to maintain the Webinterface of PfSense. The VM's are able to reach the internet via PfSense.
With this new Ubuntu VM is am able to access the transferred VMs/LXCs.
Several WAN and LAN Firewall rules were created in PfSense to accommodate access from LAN to WAN (the local network IP-range) and from WAN (local network IP range) to LAN (PfSense LAN IP range).
After this I was able to ping from the new Ubuntu VM in the PfSense LAN to the laptops in the local network.
I also tried to ping the new Ubuntu VM from the laptops, but I did not succeed.
I followed the "tutorial" Lawrence Systems 2.4.5 PfSense Installation to install and configure PfSense related to the Firewall rules.

I have attached a part of the topology of my network.
I'm not sure if the problem is related to Proxmox / PfSense or that is only a PfSense issue.
Does anyone know why this access to PfSense-LAN VMs/LXCs from the local network is not working?
And provide a solution?

Thanks!

 

[ph0x]

You installed a production grade firewall and now you're complaining that it does its job ...
Connections from WAN to LAN are forbidden by default. If you already set up a rule to allow icmp pings from WAN to LAN you probably forgot to uncheck the box "block traffic from private/RFC1918 subnets". Since your WAN address is an RFC1918 address you must allow such traffic in order for your rules to apply. That does of course not mean that all that traffic is permitted, only that it is not blocked completely.

 

[John Driessen]

Thanks ph0x for your answer. Indeed PfSense is doing great on protection!
Unfortunately the box was already unchecked (so was the bogon one), so I don't know why this then still did not work.
Do you suggest to put everything behind PfSense? (Would probably be the safest option)
I guess this would mean also
- Turn off DHCP server on the router
- Map MAC-addresses of the devices (also real devices like laptops etc) to the "100" subnet, on the pfSense DHCP-service.
This triggers several questions to me in that case:
- I have my Powerline adapters and Wifi Booster also mapped to current "10" subnet (which is "WAN" for pfSense)
Should I map these as well to be on the "100" subnet? What implications would that bring?
And if I would not map them to the "100" subnet but leave them on "10", what would the implications be?
- Should the Proxmox host (192.168.10.21) itself also be on the "100" subnet, and if so how do I accomplish that.
What happens if PfSense "breaks", how can I reach Proxmox then?
- Similar question when installing a fresh instance of Proxmox, how to put that one on the "100" subnet?
- I have also a guest-wifi subnet defined on the router. If I would turn off DHCP on the router, would that also mean the guest-network will not be able to provide IP-addresses on the "guest" subnet?
Should I add another vmbr configured as a 2nd LAN, (similarly configured as the PfSense LAN) and have it as a "200" subnet including DHCP server and make the firewall rules in PfSense so that it can reach the other LAN? WIll this work also as a Wifi guest network?

I know these questions are more related to PfSense (on Proxmox) than on Proxmox itself, but I hope that you are willing to answer the questions.
Thanks in advance!

 

[ph0x]

I think I can put this all in a nutshell. Yes, it's possible to virtualize your router and route everything through that particular VM.
The drawback is, if your host is down for whatever reason, your whole network doesn't have internet.

To reach the host if the firewall is down you basically have three options:
1. Out of band management (IPMI, iLO, ...)
2. Separate NIC for direct access by a dedicated computer
3. Physical access

That is a decision that only you can make, I would only recommend it, if the hardware is reliable. But that's completely up to you.

Now back to the actual problem, why you can't ping a VM behind the firewall. Again, I see three options:
1. Traffic from private subnets is blocked (you ruled that out)
2. No rule in place that allows ICMP from WAN to LAN (you did not rule that out credibly )
3. Proxmox firewall blocks ICMP traffic

 

[John Driessen]

Hi ph0x,

I also set the firewall option in Proxmox on DaraCenter and Hostlevel on "no". And tried to ping a local LAN address from one of my laptops (on the "WAN" local network.) Still no luck.
Unless I don know how to set an ICMP allowance from WAN to LAN also option 2 in your reply above is not working.
So all three options seem not to be the case here.
Unless you have a brilliant idea how to solve my issue, I'm going to stop using PfSense and move the "100"-subnet items back to the "10" subnet.
I'm out of ideas

 

[ph0x]

Of course I have, namely to setup the stuff correctly.
It's a bit of a hassle sometimes to begin with pfSense, but I think it's worth the effort. Although in your setup the Proxmox firewall could also serve most purposes, it's still a project to learn something.

So you
1. setup a pfSense VM with no Proxmox firewall rules in place
2. define vmbr0 as net0 and vmbr1 as net1 in pfSense and install them as WAN and LAN
3. uncheck the private subnet box on the WAN interface
4. create a rule to allow ICMP traffic from WAN to LAN for any client
5. (and that's most probably the issue here) define a static route for '100' network through your Proxmox host for all clients in the '10' network.

Then a client connected to vmbr0 (the Proxmox host for example or another VM) should be able to ping a VM with vmbr1 as their only NIC.
Without the static route your machines try the default gateway which of course doesn't know of subnet '100' behind Proxmox ...

 

[John Driessen]

Hi ph0x,

Thanks again for your answer. Luckily point 1-4 were already in place , so I guess indeed the static route is not in place.
Would this be the command to be entered on the Proxmox Host that would do it?
route add -net 192.168.100.0 netmask 255.255.255.0 dev vmbr1

Or should it be a less "global" one?

 

[ph0x]

You have to define the route for every machine that wants to communicate with this subnet. This can be achieved by a DHCP option usually, or you set it manually.
The command would be ip route add 192.168.100.0/24 via 192.168.10.70

 

[John Driessen]

Ah OK. So on every laptop I have to add that. And how does that DHCP option work?

 

[ph0x]

That is dhcp option 121 (classless static route). How you set that up depends on your router model. You should consult your router's manual for that.

 

[John Driessen]

OK, well now at least I've got the pinging working now from one of my laptops, and from that laptop I now also can use the webconfigurator of PfSense. Thanks for your help!!

 

[John Driessen]

That is the pinging of the PfSense VM (LAN-side gateway). The other VMs do not work yet. I'll check on that later.

 

[ph0x]

But the LAN gateway is already a step forward!
Now it's probably the rules that do not match.

 

[indraraj001]

Hi ph0x.. Can you share any link or tutorial for static route. My proxmox is not able to ping vm behind the pfsense and not even pfsense LAN ip 192.168.1.1 .. on pfsense icmp rule is there. Gateway is showing only one 192.168.50.1 (vmbr1 related ip).

 

[ph0x]

That should do it:
ip route add 192.168.1.0/24 via 192.168.50.1