[SOLVED] PFsense best way for network configuration

[xillion]

Hello every body,

I would like to ask you the best way to configure PFsense regarding network setup.

I have 1 NIC (vmbr200) for interco with my box provider.

I have 4 NIC to my switch with bond (lacp) and vlan setup :
- vmbr0 to bond0
- vmbr0.10 to bond0.10
- ...

This part work like a charm.

My question is wich interface I must assign to my VM to use my bond and vlan?
I need to use vmbr0 with tag or directly the sub interface like vmbr0.10 ?

What the difference ?

Thanks for tout support.

Xillion

 

[Dunuin]

vmbr0 is a bridge.
vmbr0.10 is a vlan interface ontop of that bridge.

You can only attach VMs to bridges, so you need to use the vmbr0 and set the "VLAN Tag" for the VMs virtio NIC. And of cause your vmbr0 needs to be vlan aware.

 

[patch]

The best set up probably depends on other details of your application.
I don't have a network of Proxmox servers so live migration is not an option in my use case.
I pass through all the NIC pfsense uses. The advantages of doing so are:

• Minimize the attack surface of my router
• Allow use of all hardware features of my NIC, optimising performance and power consumption
• Simplify network configuration as VLAN's are completely handled by pfsense
• Nic stats are accurately shown in pfsense not just the the Proxmox software bridge interface stats

The disadvantages are

• Live migration is not readily supported
• The NIC used by pfsense can not be used by Proxmox or other VM while the pfsense VM is running
• Pass through is not as extensively tested by Proxmox compared to visualized devices.

 

[Dunuin]

• Allow use of all hardware features of my NIC, optimising performance and power consumption

Jep, but with OPNsense/pfsense you should disable all hardware offloading features anyway (and should be the default) so that isn't that useful.

• Simplify network configuration as VLAN's are completely handled by pfsense

Thats what vlan aware bridges are for. If you don't set the VLAN tag to a virtio NIC the pfsense will still be able to handle all the different VLANs using just one virtual NIC. You are not limited to a single VLAN per virtual NIC, so you could also completly handle VLANs by your pfsense without PCI passthrough your NICs.

And you forgot a disadvantage:
With virtual NICs and bridges your bandwidth will be 10Gbit and more as long as the packets won't leave your host (for example VM to VM communication) and only the CPU performance is the bottleneck. So VMs in different VLANs connected by your pfsense could communicate with several Gbits if your CPU can handle all the packets. Only if the packets need to leave the host they will get crippled down to the bandwidth of your physical NIC. With PCI passthrough you always only get 1 Gbit when using Gbit NICs.

 

[patch]

with OPNsense/pfsense you should disable all hardware offloading

Some NIC hardware offloading is best disabled but not all.

Thats what vlan aware bridges are for. If you don't set the VLAN tag to a virtio NIC the pfsense will still be able to handle all the different VLANs using just one virtual NIC. You are not limited to a single VLAN per virtual NIC, so you could also completly handle VLANs by your pfsense

I did not say it could not be done with a virtual NIC in Proxmox, I said it was simpler if Proxmox is not involved at all. With a virtual NIC the configuration must be correct in both pfsense and Proxmox. Indeed there are multiple ways to achieve similar functionality if traffic goes through both pfsense and Proxmox, but that complicated not simplifies setup. And comparing network configuration in pfsense vs Proxmox, pfsense is easier to configure individual interfaces (as expect, it's a router firewall product).

And you forgot a disadvantage:
With virtual NICs and bridges your bandwidth will be 10Gbit and more as long as the packets won't leave your host

Horses for courses.

When running pfsense on hardware there is a choice of running it bare metal or under a hypervisor. The most common recommendation is just run it bare metal, principally because of the difficulty in maintaining the hypervisor (Proxmox hypervisor does not have the facility to readily backup the hypervisor configuration, nor readily take a snap shot of the hypervisor and revert to a prior version if required). Pass through minimises this disadvantage as the hypervisor is not exposed to the WAN, limiting site exposure to hypervisor vulnerabilities. However having a hypervisor on router hardware allows building a DIY all in one "router" such as adding 3cx for VoIP, a Ubiquiti controller for wifi, or Pihole DNS. The bandwidth requirements of which are well within the sites network switch in all sensibly designed networks.

In contrast if I was building a server running many applications, one of which happened to be a firewall then, sure I would fit the firewall software within the hypervisor's preferred configuration not visa versa.

Which is why

The best set up probably depends on other details of your application.

 

[xillion]

µvmbr0 is a bridge.
vmbr0.10 is a vlan interface ontop of that bridge.

You can only attach VMs to bridges, so you need to use the vmbr0 and set the "VLAN Tag" for the VMs virtio NIC. And of cause your vmbr0 needs to be vlan aware.

Hello Dunin,

Thanks for your reply.

I try your recommendation but I have an issue.
Like i said previously, my bond configurationt is as the follow (with vlan aware) :

• bond 0 --> vmbr 0
• bond 0.10 --> vmbr 0.10
• bond 0.20 --> vmbr 0.20

...
and I have assigned the vmbr with tagged vlan to Pfsense:

• vlan 10 --> vmbr 0.10
• vlan 20 --> vmbr 0.20

In this case, I can ping my PVE in all vlan from my switch, but I can't ping other device in the same vlan (PFsense or Windows, but I can ping between the two device). So for exemple in vlan 10:
Ping between :

• Switch and PVE --> ok
• Switch and PFsense or Windows -- not ok
• PFsense and windows --> ok

I change my netowrk configuration like the follow and It's work:

• bond 0 --> vmbr 0
• vmbr 010 (name) --> vmbr 0.10 (port bridge)
• vmbr 020 (name)--> vmbr 0.20 (port bridge)

But i would like to know why It doen't work with my first setup.
Here, my configuration with 2 configuration

auto lo
iface lo inet loopback

iface enp40s0 inet manual

auto enp37s0f0
iface enp37s0f0 inet manual
        mtu 9000

auto enp37s0f1
iface enp37s0f1 inet manual
        mtu 9000

auto enp38s0f0
iface enp38s0f0 inet manual
        mtu 9000

auto enp38s0f1
iface enp38s0f1 inet manual
        mtu 9000

auto bond0
iface bond0 inet manual
        bond-slaves enp37s0f0 enp37s0f1 enp38s0f0 enp38s0f1
        bond-miimon 100
        bond-mode 802.3ad
        bond-xmit-hash-policy layer2+3
        mtu 9000
#LACP vers le switch

auto bond0.10
iface bond0.10 inet manual
        mtu 9000


auto bond0.20
iface bond0.20 inet manual
        mtu 9000


auto bond0.30
iface bond0.30 inet manual
        mtu 9000


auto bond0.40
iface bond0.40 inet manual
        mtu 9000


auto bond0.50
iface bond0.50 inet manual
        mtu 9000


auto bond0.100
iface bond0.100 inet manual
        mtu 9000


auto vmbr200
iface vmbr200 inet static
        address X.X.X.X/24
        gateway X.X.X.X
        bridge-ports enp40s0
        bridge-stp off
        bridge-fd 0


auto vmbr0
iface vmbr0 inet manual
        bridge-ports bond0
        bridge-stp off
        bridge-fd 0
        bridge-vlan-aware yes
        mtu 9000
#lacp bond0

auto vmbr020
iface vmbr020 inet static
        bridge-ports bond0.20
        bridge-stp off
        bridge-fd 0
        bridge-vlan-aware yes
        bridge-vids 2-4094
        mtu 9000


auto vmbr030
iface vmbr030 inet static
        bridge-ports bond0.30
        bridge-stp off
        bridge-fd 0
        bridge-vlan-aware yes
        bridge-vids 2-4094
        mtu 9000


auto vmbr040
iface vmbr040 inet static
        bridge-ports bond0.40
        bridge-stp off
        bridge-fd 0
        bridge-vlan-aware yes
        bridge-vids 2-4094
        mtu 9000


auto vmbr050
iface vmbr050 inet static
        bridge-ports bond0.50
        bridge-stp off
        bridge-fd 0
        bridge-vlan-aware yes
        bridge-vids 2-4094
        mtu 9000


auto vmbr100
iface vmbr100 inet static
        bridge-ports vmbr0.100
        bridge-stp off
        bridge-fd 0
        bridge-vlan-aware yes
        bridge-vids 2-4094
        mtu 9000

auto vmbr010
iface vmbr010 inet static
        bridge-ports bond0.10
        bridge-stp off
        bridge-fd 0
        bridge-vlan-aware yes
        bridge-vids 2-4094
        mtu 9000

So what is the right setup for bond and vlan configuration ?

 

[xillion]

@patch
Thanks to share your experience with me.

My installation is very basic and for home use.
I only have one node and I need to share my network on many VM.
So I don't want to pass through my network card directly to PFsense.

Regarding the security point, I use a dedicated vlan (Point to Point) for wan connectetion and the separe network card, I think It's good approach (but may be not the best).

 

[Dunuin]

Looks like you mixed up stuff.
There are 2 ways you can handle multiple VLANs:

1.) you use a single vlan aware bridge and set the VLAN Tag for each NIC in each VM config. Like this:

auto bond0
iface bond0 inet manual
        bond-slaves enp37s0f0 enp37s0f1 enp38s0f0 enp38s0f1
        bond-miimon 100
        bond-mode 802.3ad
        bond-xmit-hash-policy layer2+3
        mtu 9000
#LACP vers le switch

auto vmbr0
iface vmbr0 inet manual
        bridge-ports bond0
        bridge-stp off
        bridge-fd 0
        bridge-vlan-aware yes
        bridge-vids 2-4094
        mtu 9000
#lacp bond0

And then attach all guests to vmbr0.

2.) you don't use vlan aware bridges and create a new VLAN interface and a new bridge for each VLAN. Like this:

auto bond0
iface bond0 inet manual
        bond-slaves enp37s0f0 enp37s0f1 enp38s0f0 enp38s0f1
        bond-miimon 100
        bond-mode 802.3ad
        bond-xmit-hash-policy layer2+3
        mtu 9000
#LACP vers le switch

auto bond0.10
iface bond0.10 inet manual
        mtu 9000

auto bond0.20
iface bond0.20 inet manual
        mtu 9000

...

auto vmbr10
iface vmbr10 inet static
        bridge-ports bond0.10
        bridge-stp off
        bridge-fd 0
        mtu 9000

auto vmbr20
iface vmbr20 inet static
        bridge-ports bond0.20
        bridge-stp off
        bridge-fd 0
        mtu 9000

...

And then attach guests to vmbr10, vmbr20, ... or whatever that guest should be attached to.

 

[xillion]

Hi Dunin,

Manu thanks for your explanation, it' s clear now.

All work fine with your setup