How to install pfsense as a VM inside proxmox and handle all traffic before all VM

[quangnhut123]

Hello everybody,

Now i have just bought a new server a plan to setup it with promox. But beside i need pfsense install as a VM and act like a firewall for all my VM and LXC .Could you please teach me how to do that. I am very confuse i will sent to you some images here is my Infra :

Notice : i setup all infra with VMWare so 2 NIC of Proxmox is virtual NIC
VMWare settings :

PVE Network : create 2 bridge vmbr0 and vmbr1

Pfsense NICs assigned 2 bridge vmbr0 as WAN and vmbr1 as LAN

Test Linux Workstation assign vmbr1 as bridge to connect with LAN interface of Pfsense:

Here is some default config on Pfsense :

Routing

Firewall Rules

NAT

Pfsense Ping Test :

Ping to Pfsense LAN ip 172.16.2.6 from Linux Workstation:

Ping to Pfsense WAN ip 192.168.93.138 from Linux Workstation:

But Finally the workstation cannot connect to internet :

Could i do correct ? Please help me to solve this problem . Thanks

 

[quangnhut123]

Any one can help me figure out ?

 

[jim.bond.9862]

this seams like a bad setup, you may need an extra NIC here.
I am building out a dedicated machine for my router right now, and based on what I have research so far you really need 3 NICs.
it is possible to run with 2 but ideally 3 is needed.
that said, can you provide a bit more info about your setup.

specifically :

your wiring . i.e. WAN(provider modem) --> router --> <myserver NIC0 IN > --><my server nic1 out > --> switch .....
id pfSense VM will only provide firewall for local vms only not on network?
how did you configure pfSense VM? FYI if you only did basic setup in pfSense it will not allow any traffic to go through,
you need an incoming and outgoing rules to be setup first before any traffic will pass through

 

[quangnhut123]

this seams like a bad setup, you may need an extra NIC here.
I am building out a dedicated machine for my router right now, and based on what I have research so far you really need 3 NICs.
it is possible to run with 2 but ideally 3 is needed.
that said, can you provide a bit more info about your setup.

specifically :

your wiring . i.e. WAN(provider modem) --> router --> <myserver NIC0 IN > --><my server nic1 out > --> switch .....
id pfSense VM will only provide firewall for local vms only not on network?
how did you configure pfSense VM? FYI if you only did basic setup in pfSense it will not allow any traffic to go through,
you need an incoming and outgoing rules to be setup first before any traffic will pass through

Hi there, thanks for your help. First i need pfsense is the firewall of proxmox and all inside vm. For futher i want to have more proxmox cluster node so i want to pfsense will be firewall for these nodes too.

Pleasr help me setup this. And tell me how to setup rule on pfsense. First i see in the rule section we have automatic rule created allow all connections from wan to lan. Is it not ok ?

For wiring : actually i do a lab before do on real server so i start with vmware workstation 12.
I create a proxmox vm on vmware with 2 intel virtual nic as NAT mode assign for the first NIC. Its like you plug in wan from router. The second Nic I just choose host only if i choose Nat so the vm insidr proxmox can connect internet via pfsense but seem to be every rules i create in pfsense not affect with new vm.
For proxmox configuration. I create 2 bridge. Vmbr0 and vmbr1 . I create pfsense vm inside and assign 2 e1000 nic with vmbr0 as wan and vmbr1 as lan.
For testinf i create a linux vm inside proxmox and mapping to vmbr1 as a network card. As i understand that will connect with lan of pfsense. When i start vm its auto get ip from pfsense ok. I see it in side dhcp lease of pfsense. Linux vm can ping pfsense and pfsense can ping back. Proxmox and pfsense have internet but linux vm cannot connect to internet. I try to ping ip like 8.8.8.8 but no reply from the internet.

If you can guide me could you help me setup in details. Im new in this one
Thanks

 

[daemaz]

So are you assigning vmbr1 to a CT or VM and it can't connect to pfsense? In the CT/VM is it assigned to the same 192.168.93.0/24 network? You said there's no internet access but can you ping 192.168.93.131 or 192.168.93.2?

 

[quangnhut123]

So are you assigning vmbr1 to a CT or VM and it can't connect to pfsense? In the CT/VM is it assigned to the same 192.168.93.0/24 network? You said there's no internet access but can you ping 192.168.93.131 or 192.168.93.2?

I can ping pfsense 172.16.2.6 from my VM or CT and still can receive ip from DHCP from pfsense but just cannot connect to internet. My VM ip 172.16.2.11 and pfsense have ip 172.16.2.6.

 

[daemaz]

If you can ping pfsense and not the internet, it sounds like you have to configure pfsense to allow traffic to the internet from the internal IP network to the WAN network by assigning a static route in pfSense and also allowing it through the firewall. In my case I have a /30 network inbetween my pfsense and the rest of my internal network because I have another L3 device doing routing, so in the pfsense dropdown you have to go to:

1) system > routing and create a gateway for the internal network through whatever network the interface on pfsense is connected to
2) then click static routes and add a route to the distant internal network using that gateway
3) add firewall rules on the LAN interface to allow traffic sourced from the internal network out to the internet
4) ensure either automatic outbound NAT is selected or map the NAT manually on the WAN interface from the internet network to the WAN address

 

[mir]

If you can ping pfsense and not the internet, it sounds like you have to configure pfsense to allow traffic to the internet from the internal IP network to the WAN network by assigning a static route in pfSense and also allowing it through the firewall.

It could also simply be a matter of wrong default route inside VM's. For pfsense to act as firewall/router for the internal LAN every configured VM behind pfsense needs to have pfsense configured as their default route.

 

[quangnhut123]

If you can ping pfsense and not the internet, it sounds like you have to configure pfsense to allow traffic to the internet from the internal IP network to the WAN network by assigning a static route in pfSense and also allowing it through the firewall. In my case I have a /30 network inbetween my pfsense and the rest of my internal network because I have another L3 device doing routing, so in the pfsense dropdown you have to go to:

1) system > routing and create a gateway for the internal network through whatever network the interface on pfsense is connected to
2) then click static routes and add a route to the distant internal network using that gateway
3) add firewall rules on the LAN interface to allow traffic sourced from the internal network out to the internet
4) ensure either automatic outbound NAT is selected or map the NAT manually on the WAN interface from the internet network to the WAN address

Hello can you teach me how to do that. Iam very confuse i will sent to you some images here is my Infra :

Notice : i setup all infra with VMWare so 2 NIC of Proxmox is virtual NIC
VMWare settings :

PVE Network : create 2 bridge vmbr0 and vmbr1

Pfsense NICs assigned 2 bridge vmbr0 as WAN and vmbr1 as LAN

Test Linux Workstation assign vmbr1 as bridge to connect with LAN interface of Pfsense:

Here is some default config on Pfsense :

Routing

Firewall Rules

NAT

Pfsense Ping Test :

Ping to Pfsense LAN ip 172.16.2.6 from Linux Workstation:

Ping to Pfsense WAN ip 192.168.93.138 from Linux Workstation:

But Finally the workstation cannot connect to internet :

 

[daemaz]

The default gateway, 172.16.2.6, is that another VM?

So your workstations are on a 172.16.2.0/24 network, but what is the network of the LAN interface of the pfsense?

 

[quangnhut123]

The default gateway, 172.16.2.6, is that another VM?

So your workstations are on a 172.16.2.0/24 network, but what is the network of the LAN interface of the pfsense?

172.16.2.6 is LAN ip of pfsense and same network with my Linux workstation running as a CT inside proxmox too.

 

[daemaz]

What's after the pfsense? The IP on the WAN interface is not a public one. Do you have proxmox nested ontop of a ESXi VM?

edit: ah you just want the pfsense to be a firewall between your VMs and not out to the internet. In that case you may have to uncheck "Block private networks and loopback addresses" on the WAN interface since it's not truly a WAN

 

[quangnhut123]

What's after the pfsense? The IP on the WAN interface is not a public one. Do you have proxmox nested ontop of a ESXi VM?

edit: ah you just want the pfsense to be a firewall between your VMs and not out to the internet. In that case you may have to uncheck "Block private networks and loopback addresses" on the WAN interface since it's not truly a WAN

As i said i install Proxmox on VMware Workstation and assign 2 NICs. eth0 is NAT so can connect to internet by share with host address. the eth1 is host-only NIC. I actually want inside VM can connect to internet. While pfsense can connect to internet but the VM go through cannot connect to internet !

Now i want my VM inside proxmox can connect to internet but after pfsense firewall. pfsense vm install inside in proxmox too. Proxmox wil install on top of Vmware as you see in my pictures !

 

[mir]

As i said i install Proxmox on VMware Workstation and assign 2 NICs.

I don't think you can achieve NAT on top double virtual nic. NAT requires the nic to be in promiscuous mode which IMHO opinion is not possible with a virtual nic over a virtual nic.

 

[quangnhut123]

I don't think you can achieve NAT on top double virtual nic. NAT requires the nic to be in promiscuous mode which IMHO opinion is not possible with a virtual nic over a virtual nic.

So you mean i cannot do this on VMWare ?

 

[FastLaneJB]

Just for reference but I've got a virtual Sophos UTM running on my Proxmox and that also is using VLAN's so I can attach VM's / LXC's to different VLAN interfaces on Sophos UTM. I do have a VLAN capable switch but if it's all on the same box you don't have to add in all the VLAN's, I only add ones that flow off the box but the rest can happily just be self contained inside Proxmox.

The Sophos UTM has 2 NIC's but I've actually VLANed the WAN traffic into the box so physically the traffic into the Proxmox server could all work over a single NIC. It's bonded but just saying it would work with a single NIC in this case but for that you'd have to have a VLAN capable switch to split off your WAN traffic.

I'd cut out the VMware and just install Proxmox native to do this. There's quite a few ways to do what you want but its definitely possible and you seem like your close to what you want to do anyway.

 

[gosha]

Hi Quangnhut123!

I see on the picture that your vmbr1 does not have IP-address on node. It get address from DHCP?
If yes then make static please and try.

 

[Jospeh Huber]

Perhaps offtopic - and only my opinion:
We have a similar infrastructure, but we separate ProxMox from pfSense (redundant HA-Setup with two nodes). Because of security reasons, I would always separate the main firewall from the VM infrastructure, if it is possible.

Setup dedicated boxes for pfsense or buy pfSense hardware https://www.pfsense.org/hardware/
If you need a firewall inside your VM Infrastructure use the Proxmox built in firewall.

 

[zima]

I have pfsense on one physical nic (vmbr0). pfsense have two virtio devices. Second virtio is set with vlan to communicate with wm's on that vlan. Working without problem. Looks like you have routing misconfiguration.
For freebsd remember to set System->Advanced->networking->Disable hardware checksum offload and reboot.

 

[quangnhut123]

Hi Quangnhut123!

I see on the picture that your vmbr1 does not have IP-address on node. It get address from DHCP?
If yes then make static please and try.

cannot assign ip for this one since it local (Lan) of pfsense just use internal vm only not pubic to internet. Only vmbr- bridge to eth0 can connect to outside and have wan ip

I have pfsense on one physical nic (vmbr0). pfsense have two virtio devices. Second virtio is set with vlan to communicate with wm's on that vlan. Working without problem. Looks like you have routing misconfiguration.
For freebsd remember to set System->Advanced->networking->Disable hardware checksum offload and reboot.

I do as you said Disable hardware checksum offload and reboot. but still cannot get traffic. Seem to be you have same infra with me but not on proxmox vm @@