Pfsene, proxmox and VLANs

[ryttge]

Hi,

I'm having some trouble with 802.1Q VLANs on a pfSense VM. I have quit a decent experience with vlans in general so it's not my first rodeo but i'm coming up short here.

It's a homelab used mostly for learning.

The Idea

To have 3 different VLANs routed through the pfSense firewall located on host 1.

10 Servers/networking
20 LAN
30 Guest.

The quirk is that i want have the 2 proxmox hosts i have on VLAN 10 which is then routed through the pfsense VM, VLAN 10 should be tagged. Also the Vlan 10 is going to address range wise replace default VL1

The setup looks like follows both hosts run on pve-manager/4.4-1/eb2d6f1e

Host 1

HP DL380 G6 with 4 Nics

eth0 and eth1 is bonded with 802.3ad. vmbr0
eth2 is where WAN comes in. vmbr1
eth3 is where LAN goes into the switch. vmbr2

Host 2

Old Supermicro 2 Nics
eth0 and eth1 is bonded with 802.3ad
vmbr0

Switch
HP 1920G

pfSense VM Version 2.3.2
WAN is on vmbr1 bridged port eth2 (pfsense vtnet0)
LAN is currently on vmbr2 bridged port eth3 (pfsense vtnet1)
VLAN20 configured in pfSense runs on vtnet1
VLAN30 configured in pfSense runs on vtnet1

I also have a VLAN10 on vtnet1, the plan is to switch the network port used on LAN interface from "vtnet1" to "VLAN10 on vtnet1" in pfSense

However when i do that i lose connection to everything on that interface, even with the switch correctly configured.

Vlan 20 and 30 continues to work normally and i can reach the firewall and switch on those VLANs and configure everything back.

I currently use Linux Bridges but the first step would probably be to switch the current configuration to OVS.

I tried to just switch all the ports related to untagged VLAN 10 in the switch, at which point i could reach everything but the proxmox hosts and the virtual machines on vmbr0 on both servers.

Current working config, everything here works except switching over from "vtnet1" to "VLAN10 on vtnet1" for LAN interface in pfSense

Host 1

# network interface settings; autogenerated
# Please do NOT modify this file directly, unless you know what
# you're doing.
#
# If you want to manage part of the network configuration manually,
# please utilize the 'source' or 'source-directory' directives to do
# so.
# PVE will preserve these directives, but will NOT its network
# configuration from sourced files, so do not attempt to move any of
# the PVE managed interfaces into external files!

auto lo
iface lo inet loopback

iface eth0 inet manual

iface eth1 inet manual

iface eth2 inet manual

iface eth3 inet manual

auto bond0
iface bond0 inet manual
        slaves eth0 eth1
        bond_miimon 100
        bond_mode 802.3ad

#MGMT

auto vmbr0
iface vmbr0 inet static
        address  10.100.100.10
        netmask  255.255.255.0
        gateway  10.100.100.1
        bridge_ports bond0
        bridge_stp off
        bridge_fd 0
#LAN IN

auto vmbr1
iface vmbr1 inet manual
        bridge_ports eth2
        bridge_stp off
        bridge_fd 0
#WAN

auto vmbr2
iface vmbr2 inet manual
        bridge_ports eth3
        bridge_stp off
        bridge_fd 0
        bridge_vlan_aware yes
#LAN OUT

Host 2

# network interface settings; autogenerated
# Please do NOT modify this file directly, unless you know what
# you're doing.
#
# If you want to manage part of the network configuration manually,
# please utilize the 'source' or 'source-directory' directives to do
# so.
# PVE will preserve these directives, but will NOT its network
# configuration from sourced files, so do not attempt to move any of
# the PVE managed interfaces into external files!

auto lo
iface lo inet loopback

iface eth0 inet manual

iface eth1 inet manual

#bond0
auto bond0
iface bond0 inet manual
        slaves eth0 eth1
        bond_miimon 100
        bond_mode 802.3ad
#MGMT
auto vmbr0
iface vmbr0 inet static
        address  10.100.100.20
        netmask  255.255.255.0
        gateway  10.100.100.1
        bridge_ports bond0
        bridge_stp off
        bridge_fd 0

pfSense

Seems to work if you click on it.

I've tried an assortment of different configurations but since the hosts are real servers (with real boot times) and they also run my internet uplink i now seek higher powers for aid. (Before my roommate runs me through for dropping our internet connection at my leisure)


As a side note i'm thinking about running eth1 and eth3 as a LAGG in pfsense and instead use eth0 only as mgmt interface so that the machines if need be could be put on vmbr2 and utilize the full power of VirtIO for transfers between them, There are however no use for that at the moment since they don't move any data between eachother.

Any faults in spelling is to be promptly blamed on my swedish nationality.

Kind Regards David

 

[dragon2611]

Proxmox's own traffic is usually untagged, unless you specifically put the IP interface on a Vlan with a 802.1Q kernel module loaded.

You probably want the Switch port (Or LACP/Bonding/Trunk/Portchannel/Whatever HP want to call it this week) in hybrid mode so that any untagged frames are put into VLAN10 (The management)

So assuming it behaves like my 1810-24G

Vlan 10 U (I think, since there's no mixed option in the GUI and it doesn't have a CLI.)
Vlan 20 T
Vlan 30 T

PVID 10

I'm not sure how the switch will behave with an incoming packet tagged with Vlan10

Annoying as the cisco syntax would be (If I remember rightly)

Switchport mode trunk
switchport trunk allowed vlan 10,20,30
switchport trunk native vlan 10

 

[dragon2611]

Proxmox's own traffic is usually untagged, unless you specifically put the IP interface on a Vlan with a 802.1Q kernel module loaded.

You probably want the Switch port (Or LACP/Bonding/Trunk/Portchannel/Whatever HP want to call it this week) in hybrid mode so that any untagged frames are put into VLAN10 (The management)

So assuming it behaves like my 1810-24G

Vlan 10 U (I think, since there's no mixed option in the GUI and it doesn't have a CLI.)
Vlan 20 T
Vlan 30 T

PVID 10

I'm not sure how the switch will behave with an incoming packet tagged with Vlan10

Annoying as the cisco syntax would be (If I remember rightly)

Switchport mode trunk
switchport trunk allowed vlan 10,20,30
switchport trunk native vlan 10

Or if you DO want everything tagged load the 802.1Q module and create a vlan subinterface and put the management IP for proxmox on that.

 

[ryttge]

Sorry for late reply, much to do during christmas.

The switch configuration isn't a problem

Would however like it to be like this. Not required though.

Vlan 10 T
Vlan 20 U
Vlan 30 T

The problem is that i lose connection to all machines and the mgmt gui if i switch to untagged vlan 10. And i would like the pfsense to have VLAN 10 on the LAN interface, but when i switch that around i lose all communication on that interface even though the switch is correctly configured.

I've done this quite a few times with physical firewalls but never with a virtual one.