How to configure Proxmox and PfSense VM so that all network requests go through PfSense

[panos.john]

Currently, I have a PfSense VM running on Proxmox. I can access the internet and Proxmox through the LAN port, however, I can't access the internet from the Proxmox Host, or any VMs or containers. I am unsure if this is a problem with PfSense or Proxmox.

I have two virtual switches that correspond to two real ports. I have the WAN port (vmbr0) and the LAN port (vmbr1). The WAN port goes out to my school's network, and the LAN port goes out to my Nighthawk router running in AP mode.

That works perfectly and I can access the internet from all devices through the Nighthawk router (LAN).

But, I can't access the internet through Proxmox itself. I can access Proxmox through the LAN using the IP specified on vmbr1 which is 10.0.0.100. On the Proxmox host, I can ping 10.0.0.1 (PfSense VM) but I can't ping google.com. This holds true with VMs and containers under Proxmox too.

It is not a DNS issue because hosts resolve to an IP. It just can't get data once the DNS resolves.

Proxmox Host Shell:


The network setup is outlined below (/etc/network/interfaces):

auto lo
iface lo inet loopback

iface eno1 inet manual

iface eno2 inet manual

iface eno3 inet manual

iface eno4 inet manual

auto vmbr0
iface vmbr0 inet manual
        bridge-ports eno1
        bridge-stp off
        bridge-fd 0

auto vmbr1
iface vmbr1 inet static
        address  10.0.0.100
        netmask  255.255.255.0
        gateway  10.0.0.1
        bridge-ports eno2
        bridge-stp off
        bridge-fd 0
        bridge-vlan-aware yes
        bridge-vids 2-4094
#pfSense

 

[chotaire]

You are attempting to do something pretty wild, which is have Proxmox itself use the pfSense VM as its default gateway. If this were configured correctly, this would mean that Proxmox is basically down the moment your pfSense is not running, e.g. during and after boots, during maintenance, etc. I can only ask to rethink this because pfSense is highly dependant on Proxmox but Proxmox should not be dependant on one of its VMs to function. If pfSense is ever screwed after an update, you cannot do anything unless you reconfigure the network on the Proxmox host. In this were a cluster, this could be fatal. Just imagine there is an issue with your pfSense VM and you need to fix it remotely...

Consider planning your setup like this:

a) You have an internet gateway, probably your Nighthawk AP, which is going to be connected to vmbr0 (eno1). Proxmox will be configured to have an IP address on the same network as your Nighthawk AP and use it as its default gateway. This way Proxmox will always have an internet connection and be able to provide this internet connection via routing to pfSense.
b) You configure vmbr0:fw (which means adding an additional private IP network to this bridge) to communicate between Proxmox and pfSense.
c) pfSense will be configured to use vmbr0 as WAN interface
d) Configure vmbr1 and bridge it to eno2. Do not configure a default gateway on eno2/vmbr1.
e) pfSense will be configured to use vmbr1 as LAN interface. Give it an IP on the school network. This becomes the default gateway for them.
f) Add an additional vmbr2 interface with bridge-ports set to none and no ip configuration. This will be the device for your VMs/containers.
g) pfSense will be configured to use vmbr2 as OPT interface, e.g. "VM". Configure the VM interface on pfsense with a valid private IP network. The VMs and containers will reside on the same network and use the pfSense IP on this network as default gateway. Could run a DHCP server on pfSense for this interface.
h) Make sure that NAT is configured on pfSense for IPs coming from LAN and VM to be translated to the WAN IP address of pfSense (which is on vmbr0:fw).
i) On the host, NAT the vmbr0:fw network to be translated to the IP address of vmbr0/eno1. Yes, that's a double NAT but that's fine, you will likely be carrier-grade NATed more than once on top anyway, welcome to the crowded world of IPv4. This double NAT configuration keeps you from changing routing on both pfSense and hypervisor each single time you make a change or add new networks.
j) Double-check your routing is intact. Just make sure Proxmox can route to the Internet and to pfSense on vmbr0:fw.
k) If you dislike Proxmox not using a pfSense firewall for its internet connection then you have multiple options, among them to configure the integrated firewall in the Proxmox UI or use iptables-persistent package or to consider using a hardware pfSense firewall instead, which will be connected between Proxmox host and internet uplink.

Have fun!

 

[AllanM]

Few tips for what you're trying to accomplish:

1. I see nothing wrong with virtualization your router, especially if you're already inside another private network anyway. if you bork up your pfsense VM and need internet for proxmox, you can always give proxmox an IP on the "WAN" (private school network) and move on, then switch back to using your "internally" hosted gateway when ready. I virtualize my home router on proxmox and it worked fine.






2.You'll note, that in my deployment, my "WAN" IP the actual public IP. Pfsense is negotiating the PPPoE with my DSL provider over a DSL modem in bridge mode (all 4 nodes connected to the 4 port switch on the modem). In your environment, you're likely already in a private network space... By default, pfsense won't "route" to private network addresses on ports it considers to be the WAN port. You'll have to "allow" this (uncheck this on interface settings):



3. Of course, you'll also need to make sure you are hosting a network that doesn't share broadcast range with your school network. If they are say, a 10.0.0.0/16 or something like that, then you might want to host a 192.168.X.0/24 for yourself.

4. Also... you will likely need to define the upstream gateway for the WAN interface in pfsense:


5. Don't forget to disable hardware offload in pfsense (system>advanced). When virtualized, these don't work.


6.If you want to use Suricata in Inline mode, you'll probably have to set this system tunable: (force netmap emulation).

 

[chotaire]

Even PPPoE handled by the VM, now that's one single point of fail.

if you bork up your pfsense VM and need internet for proxmox, you can always give proxmox an IP on the "WAN"

Try that remotely In the end I don't care, people should do whatever pleases them best. I can only ask people to rethink this if someone expects this to be somewhat professional.

 

[AllanM]

You are attempting to do something pretty wild, which is have Proxmox itself use the pfSense VM as its default gateway.

I suspect more people do this than you think. Some of us like to live on the wild side

If this were configured correctly, this would mean that Proxmox is basically down the moment your pfSense is not running, e.g. during and after boots, during maintenance, etc.

proxmox doesn't just stop working without a gateway, it just looses access to updates and time servers. It can go awhile without either in most cases.

I can only ask to rethink this because pfSense is highly dependant on Proxmox but Proxmox should not be dependant on one of its VMs to function. If pfSense is ever screwed after an update, you cannot do anything unless you reconfigure the network on the Proxmox host.

You could just boot the last backup of pfense? I always make a backup of my pfsense VM as soon as I have it in a nice working configuration and after major configuration changes. Isn't that sort of the point of virtualization complex server/firewall functions?

In this were a cluster, this could be fatal. Just imagine there is an issue with your pfSense VM and you need to fix it remotely...

For a datacenter bunker, yes, that's a problem. For a homelab? I don't see the issue.

Biggest risk is loss of access to timeserver. This could kill a ceph cluster if gone untreated for too long.

 

[AllanM]

Even PPPoE handled by the VM, now that's one single point of fail.

Try that remotely In the end I don't care, people should do whatever pleases them best.

I don't understand your point. If the internet connection goes down, there's not going to be any way to remote in to fix it regardless. Being virtualized doesn't change this. If you're making changes to your firewall configuration "remotely" to a site with a single internet connection that's just bad planning period.

Almost all home/dorm/apartment's have a single internet connection and a single firewall. It's a single point of failure in nearly every home in the world. How is this any different? It's a homelab, it's not mission critical. If someone chooses to remote in to their homelab and break their firewall configuration from afar, then the problem here is really not that the firewall is virtualized.

Furthermore, OP is setting this up from behind a private network that he likely doesn't have any control. A direct remote access connection likely isn't even an option here anyway.

I can only ask people to rethink this if someone expects this to be somewhat professional.

Lets save the professional approach for work.

I'm building a proxmox cluster at work. It will be much more "professional" than my homelab environment. Newer hardware, bare-hardware edge appliance to host a gateway and time server for the cluster. All enterprise grade SSD's and drives with full data path protection, etc etc etc...

 

[chotaire]

"IF the internet goes down" is a difference to "that single point of failure WILL take down the internet" during maintenance, when used in a cluster, during firewall updates, whenever something goes wrong with that pfSense that can so easily break in its entirety when only one of its low quality modules goes crazy. Not everyone is sitting behind their proxmox box, people invented remote access over the Internet.

Virtualizing a firewall is not professional, but hey I do it too on non-critical deployments. But using it as the default gateway for the entire host, and even worse, as PPPoE connection for the entire network makes it a playground. If you recommend this to anyone out there, you should ask them first if they are planning on a playground and intend to sit behind their Proxmox box during all times they dare to click a button. If they would like it this way, fine. But if you're keen in building a professional environment, try a professional advice?

Anyway, this discussion just costs time. We both made our points. Thanks mate.

 

[AllanM]

Not everyone is sitting behind their proxmox box, people invented remote access over the Internet.

This thread is not talking about everyone. OP is already behind another private network and lives with the hardware. Would not likely be performing administrative changes remotely in such a deployment.

Virtualizing a firewall is not professional,

Virtualized firewalls are commonplace in professional cloud based environment deployments and as non-edge firewall/routers in on-premise deployments. Virtualizing a complex pfsense deployment has huge benefits for recovery from mistakes/misconfiguration/failed-updates/etc.

Netgate, Fortinet, Cisco, HPE, Barracuda, IBM, Sophos. Just a few examples of major players in virtualized firewall offerings, available for immediate spin-up on some of the worlds largest datacenters like Azure, AWS, IBM, etc.

But using it as the default gateway for the entire host, and even worse, as PPPoE connection for the entire network makes it a playground. If you recommend this to anyone out there, you should ask them first if they are planning on a playground and intend to sit behind their Proxmox box during all times they dare to click a button. If they would like it this way, fine. But if you're keen in building a professional environment, try a professional advice?

Again, he's already behind a private network and living in the same place with the firewall.

 

[velocity08]

Hi Team

interestingly enough I’m seeing exactly the same issue on a test deployment.

on ProxMox I’ve been troubleshooting this for about a week.

vm’s on private lan can ping external resources but when trying to get any data from those resources it fails.

example.

ping works
wget fails
web browsing fails.

thought I would set it up on our VMware test bed with VXLAN and it started working as expected almost straight out of the box with minimal config.

VM’s on private lan can:

ping
wget
surf web

the only difference is the hosting platform, im theorising that ProxMox is blocking something somewhere or maybe something simple that I haven’t noticed yet.

might blow away pfSense and give it another run from fresh.

will report back.

””Cheers
G

 

[GalacticSpines]

Did anyone come up with a solution? I have the same issue. pfSense VM running everything 100% on lan and wan side, but Proxmox has no internet connectivity though gateway and ip setup correctly.

 

[zb26]

I had a similar problem. I got it to work, although I don't understand why it works and in theory it seems wrong. I would just share my experience and hope it will help someone or someone will explain to me what's going on.

Initially I used a wireless router connected to a mini PC installed with Proxmox. I gave the router a static ip 192.168.1.5 and the Proxmox a static ip 192.168.1.20.

Then I installed pfsense through web interface. Pfsense's IP address was 192.168.1.1. I created two bridges on proxmox, one to LAN (IP address 192.168.1.20, gateway 192.168.1.5) and the other to WAN.

After installing pfsense, I removed wireless router, and the proxmox/pfsense box becomes the router. I connected its WAN NIC directly to the Fios modem, and a computer to the LAN NIC. Miraculously, the network works. I can open the pfsense web page at 192.168.1.1, and the proxmox page at 192.168.1.20. I can also browse the Internet.

There was one thing that didn't work. I couldn't ping any site from the Proxmox shell. This seems not surprising, because its expected gateway 192.168.1.5 (the initial router I used to set proxmox box) is gone. Then I change the gateway of the Proxmox PFsense LAN bridge to 192.168.1.1. Now everything works including that I can ping internet sites from the proxmox, and do apt update, upgrade etc. I can also reboot the box and all functionalities of proxmox and pfsense would resume automatically.

Now let's see what seems wrong. The Proxmox hosts a virtualized pfsense through a bridge. The gateway of the bridge is 192.168.1.1 which is the ip address of pfsense. It is just logically confusing that Proxmox connects pfsense to the WAN, yet pfsense with its DNS servers helps proxmox to connect to WAN too.

Am I making any sense?

 

[GalacticSpines]

I've found the problem, I didn't change my initial DNS from the installation to pfSense (10.10.0.1). Don't know why I missed such a simple thing...

 

[HandreasH]

I had the same problem. Proxmox and Opnsense on the same bridge.
Could not get the proxmox updates to work.

My solution is to passthrough two nic's to Opnsense. One for LAN and one for WAN.
Passing through pci devises to Opnsense you have to uncheck PCIe in the configuration. There is a bug in the freebsd kernel used by Opnsense!

 

[Andrey.Lysikov]

HI! My work proxmox setup with router as VM (opnSense).
internet -> swith (level2, and internet port is vlan tag 2) -> proxmox (standart bridge, and fixed local ip):

vm config is:

inside in vm:


and internet work wery well, inside of console proxmox, and inside all vm, and local network.

 

[tjaybs06]

Hi! I wanna share also my work. NIC Teaming works great in Proxmox and OPNsense/pfSensense. First, I create Linux bond from my NIC and use it as a second virtual bridge. You don't need to passthrough the device, just add it only as normal device network. Then, I use the default bridge (vmbr0) as WAN and the second bridge (vmbr1) I created as LAN in OPNSense/pfSensense. I attached a picture of sample VM (TrueNAS) which already connected to OPNsense/pfSensense and use vmbr1 s a normal network adapter.

 

[barrynza]

HI! My work proxmox setup with router as VM (opnSense).
internet -> swith (level2, and internet port is vlan tag 2) -> proxmox (standart bridge, and fixed local ip):
View attachment 21784
vm config is:
View attachment 21785
inside in vm:
View attachment 21786

and internet work wery well, inside of console proxmox, and inside all vm, and local network.

Nice setup but i want to have no switch between - more direct to isp but no exposure of proxmox at all.

 

[MagicCamera]

Glad I came across this. I was toying with the idea of virtualization of pfSense, but one thing kept bugging me. Egg or the Hen situation. I have a Protectli FW6 Vault and stuffed it up with lots of RAM and SDDs, sure it will work, but putting it at the front of my network seemed daft because now there are 2 elements that if either one fails the whole internet is down and my wife hits me over the head with a stale loaf of bread. I guess you could call this a workaround as it prevents the whole house from losing internet connection: ISP--->Linksys Router with WiFi--->DMZ (all ports open) to Protectli Vault FW6A WAN port issued to pfSense VM--->LAN port issued to pfSense VM--->Managed Switch (only workstations connected). The workstations however have WiFi but will only be used if either pfSense or Proxmox fail. pfSense will handle the firewall for the workstations on the DMZ from the consumer router. Make sense? Not my ideal solution but it prevents total failure and a bash on the head.

 

[turtlehurdle]

Few tips for what you're trying to accomplish:

1. I see nothing wrong with virtualization your router, especially if you're already inside another private network anyway. if you bork up your pfsense VM and need internet for proxmox, you can always give proxmox an IP on the "WAN" (private school network) and move on, then switch back to using your "internally" hosted gateway when ready. I virtualize my home router on proxmox and it worked fine.

View attachment 14397

View attachment 14398


2.You'll note, that in my deployment, my "WAN" IP the actual public IP. Pfsense is negotiating the PPPoE with my DSL provider over a DSL modem in bridge mode (all 4 nodes connected to the 4 port switch on the modem). In your environment, you're likely already in a private network space... By default, pfsense won't "route" to private network addresses on ports it considers to be the WAN port. You'll have to "allow" this (uncheck this on interface settings):

View attachment 14399

3. Of course, you'll also need to make sure you are hosting a network that doesn't share broadcast range with your school network. If they are say, a 10.0.0.0/16 or something like that, then you might want to host a 192.168.X.0/24 for yourself.

4. Also... you will likely need to define the upstream gateway for the WAN interface in pfsense:
View attachment 14401

5. Don't forget to disable hardware offload in pfsense (system>advanced). When virtualized, these don't work.
View attachment 14402

6.If you want to use Suricata in Inline mode, you'll probably have to set this system tunable: (force netmap emulation). View attachment 14404

I can't find the option for IPV4 upstream gateway. Where is it located? I have DHCP enabled, if that would change things?

 

[AllanM]

I can't find the option for IPV4 upstream gateway. Where is it located? I have DHCP enabled, if that would change things?

The option to define an upstream gateway for an interface only appears with the interface configured for static IP assignment. With DHCP, the upstream gateway should be "provided," however, if you are the administrator of both the pfsense you're configuring and the upstream gateway, you might want to configure these as static addresses instead.

 

[jerry001]

i have attached the virtual ips on Proxmox server so for lan I am using the virtual IP for LAN and static IP for WAN. and when I am configuring the nat port fordwarding in proxmox server this is not working in my case, So anyone have any suggestion How can I overcome this.