Proxmox + Pfsense + VLANs (including VMs) with two NICs?

[kdjordjev]

Hello,
I am trying to setup Pfsense on Proxmos in a VM as router.
The server have two NICs (eno1 and enp4s0).
In Pfsense I have assigned eno1 to be WAN and enp4s0 to be the LAN(192.168.1.1/24) + few VLANs on the same interface.
Proxmox runs on the enp4s0/vmbr0.
Configuration below (/etc/network/interfaces)

My HW is: Modem(WAN) --> eno1/vmbr1 (Pfsesne) --> enp4s0/vmbr0 (LAN + VLANS on PFsense) + Proxmox + other VMs --> Unifi managed switch --> WiFi APs (Unifi) + clients

What I want to achieve is:

1. Sort the different clients by VLAN which i can achieve by the APs and the managed switch
   1. users on LAN
   2. Iot (20) , VOIP(1010), Guest(1020) , other Servers/NAS behind the unifi switch (100)
2. Have Proxmox and all VMs running on VLAN 100
   1. I am ok of the PFsense needs to be on the LAN

With my current configuration the Proxmox and Pfsense are running in the LAN, and i can achieve (1) above. Within Proxmox/VMs/ and clients sitting on LAN i can see all VLANs (as per the firewall rules). I can assign other clients to different VLANs. Seems to work as expected.
However i can not get (2) working.

I tried:

(a) assigned vmbr0 to be VLAN aware --> As result all clients behind the switch lost connection to the VLANS . Only LAN was functional​

(b) assigned vmbr0 to be VLAN aware + assigned some VMs to VLAN100 tag --> the VMs did not get any connectivity to the VLAN​

(c) vmbr0 not aware of VLANs + assigned some VMs to VLAN100 tag --> the VMs did not get any connectivity to the VLAN​

(d) manually changed the vmbr0 address to belong to VLAN100 (192.168.100.100/24) range and gateway (192.168.100.1) and modified "hosts" file to 192.168.100.100 --> in this case i lost connection to the proxmox and had to recover it through a terminal/CLI​

​

Any suggestions on how to achieve (2) above? Basically want to have Proxmox and all VMs (potentially excluding the pfsense) running on a VLAN , while Pfsense serves LAN and all other VLANs

Thank you

​

auto lo
iface lo inet loopback

iface enp4s0 inet manual

iface eno1 inet manual

auto vmbr0
iface vmbr0 inet static
address 192.168.1.100/24
gateway 192.168.1.1
bridge-ports enp4s0
bridge-stp off
bridge-fd 0

auto vmbr1
iface vmbr1 inet manual
bridge-ports eno1
bridge-stp off
bridge-fd 0

 

[MarvinE]

I think the simplest way is to create a vmbr for every VLAN. And take some more NICs in pfsense.
Simply set bridge-ports enp4s0.100 (für VLAN 100).

Now you can add an address in Proxmox to have it connected to "SERVERS".

 

[kdjordjev]

MarvinE
Thank you!!
I do not have a choice to add more NICs. I have only two integrated in a small "mini-PC type of computer".

Would you mind to help me with an example for my case?
I think i tried to setup enp4s0.100 but failed and had to revert the interfaces file through the terminal. I must have done something wrong
Thanks

 

[Dunuin]

You can give pfsense 6 virtual virtio NICs (one for each VLAN) even if you only got 2 physical NICs.
Something like:

auto lo
iface lo inet loopback

//physical NICs
iface enp4s0 inet manual
iface eno1 inet manual


//LAN
iface enp4s0.10 inet manual

auto vmbr0
iface vmbr0 inet static
address 192.168.1.100/24
gateway 192.168.1.1
bridge-ports enp4s0.10
bridge-stp off
bridge-fd 0

// WAN
auto vmbr1
iface vmbr1 inet manual
bridge-ports eno1
bridge-stp off
bridge-fd 0

//Server
iface enp4s0.100 inet manual

auto vmbr100
iface vmbr0 inet manual
bridge-ports enp4s0.100
bridge-stp off
bridge-fd 0

//IoT
iface enp4s0.20 inet manual

auto vmbr20
iface vmbr0 inet manual
bridge-ports enp4s0.20
bridge-stp off
bridge-fd 0

//VoiP
iface enp4s0.1010 inet manual

auto vmbr1010
iface vmbr0 inet manual
bridge-ports enp4s0.1010
bridge-stp off
bridge-fd 0

//Guest
iface enp4s0.1020 inet manual

auto vmbr1020
iface vmbr0 inet manual
bridge-ports enp4s0.1020
bridge-stp off
bridge-fd 0

And then use 6 virtio NICs for your pfsense VM, each attached to another bridge.
Also don't forget to setup the port of your managed switch that is connected to the enp4s0 to handle all of the vlans (so a trunk that it will allow tagged vlan 10/20/100/1000/1010/1020 over that single port). If you forget that no other client will be able connected to the switch would be able to access your pfsense/guests/PVE host.

 

[MarvinE]

You don't need more physical NICs to do that.

Simply create a new Bridge vmbr2 with (bridge-ports) enp4s0.100 for VLAN 100, it's simply possible over the Proxmox GUI or by adding in /etc/network/interfaces

auto vmbr2
iface vmbr2 inet manual
bridge-ports enp4s0.100
bridge-stp off
bridge-fd 0

Do this for all your VLANs:

• vmbr3 with enp4s0.20 for VLAN 20
• vmbr4 with enp4s0.1010 for VLAN 1010
• vmbr5 with enp4s0.1020 for VLAN 1020
• vmbr6 with enp4s0.10 for VLAN 10

When you need Proxmox in VLAN 100, simply add an IP/Netmask to the vmbr2 like:

auto vmbr2
iface vmbr2 inet manual
address 172.16.0.1/24
bridge-ports enp4s0.100
bridge-stp off
bridge-fd 0

You can give your pfsense some more (virtual) NICs by simply adding new Network Devices over the VM Hardware tab. Take the right vmbr for the wanted VLAN (Now you can add, this as real NIC in pfsense, without to set any VLAN IDs in pfsense or VM config.)

 

[kdjordjev]

Thank you all !!!
Finally I was able to do it after some fiddling with the setup today following your suggestions.
It seems Proxmox already assigns the VLAN tags after creating the bridges.
One of the mistakes i made was that I aslo used VLANS in Pfsesne (kind of double taging) and did not work ofcourse.
What i needed to do was simply assign the new virtual NICS in Pfsesne directly to the DHCP servers without any VLAN in Pfsesnes .
My main LAN has no VLAN tag (unmanaged) and is attached directly to the vmbr0 / emp4s0.
(is this a good practice? In any case the intention for the users of the LAN is that thay have access to every subnet)

Now it works !!
The final setup is below

VLAN setup in Proxmox - Network


Passing all vmbrs to pfsense


In proxmox assign the vtnets/vmbrs directly as interface without VLANS in pfsense + add DHCP on each one

 

[MarvinE]

Nice! Looks good!

Yes you are right, all clients on your LAN can simply connect to any VLAN.
A Linux client simply by creating the interface eth0.100 or eth0.10 ...

At your first setup it would be the same. When your subnets schuld not be available from the LAN simply remove the network port (example: enp4s0.100) from your vmbr100.

VLAN Tagging, is only used for outgoing packetes. Internal Proxmox routes this over the vmbr100 without any tag.