[SOLVED] pfsense - how to route VMs

[Boyd911]

I am newbie w pfsense.

I've installed pfsense - it works well. Have 2 NICs - one for LAN (Public IP: x.x. 198.121) and one for WAN (Public IP: x.x.198.122). Both gateways are online. Traffic graph shows traffic on both interfaces and with right host IPs.

My VMs get via DHCP a LAN IP (something like192.168.2.x) but if I query their public Ip via host myip.opendns.com resolver1.opendns.com I get the LAN public IP. So my assumption is that it routes via LAN outbound.

How do I route outbound VM traffic via WAN and incoming WAN traffic to a specific LAN IP 192.168.2.x?

 

[bobmc]

This doesn't sound quite right...

you mention that pfsense has LAN x.x.198.121 and WAN x.x.198.122, you then mention that DHCP is using 192.168.2.x ? in this scenario, I would expect that the LAN interface on pfSense would also be in the same subnet that DHCP is using...??

How does your internet connection function, is this a home setup (with an isp supplied router) or a business setup?

 

[Boyd911]

Ok let me clarify.

My ISP gave me 6 public IPs (x.x.198.121 - x.x.198.126) but their public gateway is the same x.x.198.121

So the LAN router is connected with a public IP called WAN1 x.x.198.121. I tried to clarify also in a diagram. PVE is running on LAN for mgt accessibility, but the VMs get their IP via LAN DHCP.


So actually, in this scenario, there are two routes to the internet via WAN1 and WAN2. But I'd like the VM's only to use WAN2.

How to do that?

Something weird is happening. I set up pfsense with static IP, no DHCP, and I use Ubuntu VMs. During installation of the VM, it asks to confirm its networking setup and I manually define an IP outside the DHCP range. I can ssh that VM from LAN.

I found out the VM doesn't have internet access and ping won't work. So I assume that in this case, pfsense firewall blocks it?

Attached is the pfsense dashboard.

 

[bobmc]

Do you actually have two physical ISP modems or is that just how the diagram is drawn?

If your LAN DHCP server, pfsense and VM's are all in the same broadcast domain then the VM's will use the 1st dhcp offer it receives so it will probably flip between router and pfsense. To make that work predictably you would need to implement some form of dhcp policy based on source mac-addresses.

 

[Boyd911]

Do you actually have two physical ISP modems or is that just how the diagram is drawn?

If your LAN DHCP server, pfsense and VM's are all in the same broadcast domain then the VM's will use the 1st dhcp offer it receives so it will probably flip between router and pfsense. To make that work predictably you would need to implement some form of dhcp policy based on source mac-addresses.

It is one physical modem, but I drew two boxes for clarity.

I actually have the DHC assign IPs on the basis of MAC-address indeed.

But how do I get the VMs using WAN2?

 

[bobmc]

I may be wrong but I don't think you have a 'WAN2', at least in the way you seem to be thinking. It looks to me that you have a range of public IP's but a single gateway - hence all external traffic will go via that gateway.

Having multiple Public IP's will allow you to NAT inbound connections onto multiple local systems based upon their associated public IP address or DNS record.

 

[Boyd911]

Yeah, you might be right.

If that's the case, that doesn't have to be a problem as long as my PVE have unique inbound public IPs.

I removed the second DHCP server --> only the ISP router DHCP is active, pfsense uses static IP, and I set static IPs in the VMs. (updated above diagram to reflect this)

But the challenge is that the VMs don't have internet access, nor are they able to ping. Is this being blocked by pfsense?

 

[bobmc]

Do you really need pfSense in this setup? Are you able to configure your ISP router to do NAT and port forwarding?

Alternatively, perhaps the 'routing' function of your ISP modem can be disabled meaning that pfSense can then become the internet gateway/firewall/router for all of your home LAN. This simplifies setup but does then mean that a) this probably will disable the Wifi function on the router so you would have to provide an alternate Wifi network and b) if the proxmox/pfsense system is offline for any reason, then internet access is off for everyone in the house.

You could still use pfSense, just for the VM's if you wished, it will just take a bit more planning and configuration

 

[Spoonman2002]

Yeah, you might be right.

If that's the case, that doesn't have to be a problem as long as my PVE have unique inbound public IPs.

I removed the second DHCP server --> only the ISP router DHCP is active, pfsense uses static IP, and I set static IPs in the VMs. (updated above diagram to reflect this)

But the challenge is that the VMs don't have internet access, nor are they able to ping. Is this being blocked by pfsense?

you have mail....

 

[Boyd911]

Do you really need pfSense in this setup? Are you able to configure your ISP router to do NAT and port forwarding?

Alternatively, perhaps the 'routing' function of your ISP modem can be disabled meaning that pfSense can then become the internet gateway/firewall/router for all of your home LAN. This simplifies setup but does then mean that a) this probably will disable the Wifi function on the router so you would have to provide an alternate Wifi network and b) if the proxmox/pfsense system is offline for any reason, then internet access is off for everyone in the house.

You could still use pfSense, just for the VM's if you wished, it will just take a bit more planning and configuration

Thx for your reply.

Theoretically, I don't need pfsense - you’re right.

But, I want pfsense; otherwise, WAN2 has no firewall, which is a security risk. Also I’d like to keep using the LAN firewall and set up.

Another member suggested creating a VLAN and separating the VMs from LAN. I will do that.

Am back in a week #holiday

 

[Boyd911]

Ok folks. I'm back in my lab after a short holiday. Used that time also to study more pfsense and I thought I understood it....

I've added a VLAN to my pfsense configuration, it is connected to the WAN interface. I haven't set any firewall rule yet on the VLAN.

But am still running in a problem - my VM's don't get an IP from the DHCP from the VLAN. At VM creation stage I assign the VM to the WAN interface with VLAN label 20 - see below as vtnet0.20

What am I doing wrong?

 

[remark]

As far as I see, question is simple as banana.
First, you must understand, do you need pfSense in this scheme or you do not, as mentioned above by bobmc.
Outbound NAT and port forward for your VMs can be performed by ISP modem. So, if you have modem and pfSense, then you have 2 routers. Is it exactly you needed?
Try to setup simple scheme, without VLANs.
If you really want pfSense, please tell, how ISP is connected? UTP cable? Optic fiber?
Can you connect ISP directly, without modem?
If so, you may have alternative scheme. Using managed switch, you can setup VLANs and use pfSense as single router for all your setup.

 

[Boyd911]

OK, problem solved. I installed OPNsense and used it to define WAN, LAN, VLAN and it works well.

I’ve VLAN working as described above. The second WAN2 (Public IP) is used and Internet access works well.

Thanks everyone for helping me here, especially @Spoonman2002 who helped me a lot!

I learned a ton and am now “playing with the firewall rules”.