How to pass VLAN 0 Priority Tags to pfSense for DHCP

C

chocolateturtle

New Member
Jul 17, 2022
3
0
1
Some background:
https://redmine.pfsense.org/issues/12070
https://forum.netgate.com/topic/160592/how-to-get-pfsense-wan-to-accept-vlan-0

"Internet fiber providers in the USA and abroad tag their packets with vlan0 id and pfsense is unable to negotiate an IP address from their DHCP servers."

The issue is that FreeBSD 12.X, which is what pfSense runs on, removes vlan 0 tagged packets and so pfSense never can get the answers from the DHCP server to configure the WAN port.

So some work-arounds have been suggested to untag the VLAN 0 traffic using either an intermediate switch between the modem and the firewall, or running pfSense inside a Virtual Machine and then having the bridge remove the VLAN 0 tags before passing it to the Virtual Machine.

How would I configure this in Proxmox?

From what I can see, I'd have to setup a Linux bridge for my WAN port, let's call it vmbr1 and make it "VLAN Aware".
But the next step, which is to how to remove the VLAN 0 tag passing to the virtual machine is where I'm stuck. The Virtual Machine->Hardware->Network Device box has field for VLAN Tag, but it only allows tags higher than 1. I can't put a 0 in there.

Is there some other way to accomplish this?

Realistically, I only need this to strip the VLAN 0 tag from the traffic going *to* the VM so it can get the DHCP address. Traffic going the other way does not need to be tagged. It's just this initial DHCPOFFER that I need the VM to see.
 
Are you using with att? I use this as part of a proxmox-centered ATT bypass with an OPNsense VM. Regardless, using vlan0 ends up being way easier than it initially seems with proxmox. Adding a .0 on the end of your bridge port interface directly in the proxmox webgui is all that is needed to tag that traffic vlan0. To make a linux bridge (vmbr*) use eth1 over vlan0 type in eth1.0 in Bridge ports: text entry field for example. It is not necessary to select Vlan Aware for this bridge. Resultant example from my proxmox /etc/network/interfaces file.

Code: 
auto vmbr1
iface vmbr1 inet manual
    bridge-ports enp2s0.0
    bridge-stp off
    bridge-fd 0
#Wan1 v.0 AT&T bypass

EDIT: @bobmc gave an easy option to untag this traffic, but I think you will find that setting the linux bridge up as above at the proxmox node level then using that vmbr1 as your wan in pfsense VM will allow pfsense to negotiate a wan DHCP address over your ISP's vlan0 requirement. Pfsense will have no clue vlan0 is being used. Your ISP will no clue vlan0 is not being used. Traffic from pfsense to the vmbr1 appears to pfsense as regular untagged communication whereas traffic between ISP and vmbr1 will have the appropriate vlan0 tag.
 
Last edited:
  • Like
Reactions: bobmc
It would seem to me that assigning the WAN port to vmbrx and then configuring that as not-VLAN-aware would cause the VLAN information to be dropped from the packets - don't know for sure but would cost nothing to try :)
 
  • Like
Reactions: vesalius
I'm not using AT&T - I'm using Frontier Fiber. I tried vesalius' solution of adding a ".0" to the end of the bridge-ports but unfortunately that just made *all* traffic to my WAN stop.

Isn't the default configuration (unchecked VLAN Aware) the same as "not-VLAN-aware"? Or is this a specific setting that I need to put in /etc/network/interfaces manually?

EDIT: So I've seen some references to other settings that may work -

bridge-vlan-aware no
bridge-allow-untagged yes

Will try again shortly. I have to do this at good stopping points because people are not happy about my frequent network interruptions. :)
 
Last edited:
What isp modem do you have. Isn't frontier just using ATT or am I mistaken there?
 
Frontier Fiber is basically what Verizion Fios used to be. I can't check the modem right now; it's located in an inaccessable space and I only have the ethernet drop in my wiring closet at the moment, where the rest of the switch gear is located.
 
Once you get to a point you can interrupt network traffic again, I would try my suggestion one more time but first restart the ONT and the proxmox node with the pfsense VM connected to the modem. ISP can be finicky about negotiating with a new mac address without a restart in between. the other option is to spoof/clone the ISP modem's mac address in the linux bridge and/or the pfsense Wan settings.

I assume you are trying to remove the ISP modem completely? ATT requires an EAP authentication step first with the modem (not over vlan0) and only after that will dhcp/traffic work (this is over vlan0).

see the reddit thread below in regards to powering down the Frontier ONT for some time before it will negotiate with a new device.

https://www.reddit.com/r/firewalla/.../?utm_source=share&utm_medium=web2x&context=3
 
Last edited:
@vesalius, referencing post #2.

Can you clarify how you defined everything in proxmox and *sense to get the vlan0 to be removed?

I tried defining the wan nic (enp5s0) as enp5s0.0 in the bridge, but wpa_supplicant wasn't seeing the traffic. Att is the provider, with known good certs (working with sophos utm/linux and netgraph).

Thanks!
 
Where are you keeping the certs ... on proxmox, a router VM or elsewhere? I bridge my ONT connection to my ATT modem for that part with the below:
Code: 
#Linux Bridge to complete EAP/802.1X auth between ATT Modem and ONT.
#this files keeps br0 invisible to proxmox
#group_fwd_mask makes sure 802.1X traffic is bridged
#enp2s0 connects to ONT, also used as enp2s0.0 for router VM wan
#enp3s0 connects to RG, ATT gateway/modem on ONT port

auto br0
iface br0 inet manual
        bridge-ports enp2s0 enp3s0
        bridge-stp off
        bridge-fd 0
        post-up echo 8 > /sys/class/net/br0/bridge/group_fwd_mask
#RG-ONT EAP Auth

That initial authorization connection does not use vlan0. The line post-up echo 8 > /sys/class/net/br0/bridge/group_fwd_mask is what makes it possible. I define this bridge at /etc/network/interfaces.d/eap_auth and my /etc/network/interfaces file has the below line: source /etc/network/interfaces.d/*

I think this line is part of the default proxmox install, but it has been a while and I honestly can not remember. Regardless, if I remember correctly, br0 above will work fine if added to /etc/network/interfaces, but I didn't want to see it in the webgui.
 
Last edited:
Thanks for the reply. Why haven't you pulled certs to get rid of the gateway entirely?

The certs are in the vm.

I'm not sure what I'm doing wrong, I'm just not getting the eapol traffic passed through to the vm.

Nic - intel i211
Proxmox 7.2
pfsense + 23.01
opnsense 23.1
vm nic - virtio

I can see the eapol requests coming in to proxmox (tcpdump -i enp5s0 -e -U ether proto 0x888e )

But using the same commandline (adjusting for interface), I don't see them getting passed through inside the vm. After a bunch of failures I decided to test that way as there's little sense setting up wpa_supplicant if the traffic is not coming in.

The interfaces file has the group_fwd_mask included for the respective bridge name. It's just not getting passed into the vm.

What ver of opnsense are you using where this is working?

My next step is to try the tcpdump cmdline in a linux firewall to see if the eapol traffic is getting passed to the vm.
 
Just never found the time to root my gateway and pull the certs, I should have as it is likely easier to migrate the Opnsense Vm among proxmox nodes that way. Used My setup for opnsense 19 or so through the latest as of yesterday's update. Also works with vyos, pfsense and openwrt VM.

if the certs are in the VM, your setup should be a little different. You won't need that separate bridge between proxmox and the gateway. This Reddit post from a while back outlines your likely way forward. https://www.reddit.com/r/PFSENSE/comments/guwm2u/comment/fsnh9fx/?context=0

So if vmbr1 is your wan connection over enp5s0.

Code: 
auto vmbr1
iface vmbr1 inet manual
    bridge-ports enp5s0.0
    bridge-stp off
    bridge-fd 0
    post-up echo 8 > /sys/class/net/vmbr1/bridge/group_fwd_mask

You change the mac of the WAN network device in the *sense proxmox GUI to the att gateway mac, virtio=00:00:00:00:00:00,bridge=vmbr1
then you need a shell command script in *sense to run early as described in the reddit post above.
 
^^Yes, that's exactly how "interfaces" was configured, except no eapol traffic getting into the VM. *sense vm had the rg mac as its mac (where you have 00:00:00.... above).

wpa_sup was failing, so tried tcpdump to monitor for eapol trafic... Nothing. Yet as mentioned before, it is being received on the host side.

What version((s) of proxmox are you using/tested with? 7.2?

kernel 5.15.53-1-pve #1 SMP PVE 5.15.53-1 (Fri, 26 Aug 2022 16:53:52 +0200) x86_64 GNU/Linux
 
Last edited:
Last edited:
  • Like
Reactions: das1996
^^Thanks. Worth a shot.

It is curious why you didn't have to set the flag on the fwbr interface while others did. Wonder what's different.
 
@vesalius Partial success.

Adding
Code: 
 post-up echo 8 > /sys/class/net/fwbr105i1/bridge/group_fwd_mask

Sort of works. Specifically, it gets applied if the network is reloaded *AFTER* the vm is already started.

On a cold boot, the bridge fwbr105i1 DOES NOT exist. Proxmox only creates that *after* starting the vm (or immediately before starting it). The other flag for vmbr1 does get passed as vmbr1 is set to autostart.

I tried googling for a way to run a script or command just before the guest starts but came up short. On suggest was to execute something via ssh back to the host. This seems clunky as this is something that needs to be done early on in the *sense boot process.
 
Spit-balling, but Could you put a delay inside the *sense vm itself? Boot script or something that would slow down the startup?

or can you move the wpa-sup into Proxmox and let Debian do the EAP-Auth, then, just pass vlan0 to *sense as wan.

EDIT: one reason mine may work is that I am not doing the EAP-auth in the router vm. It happens over a Debian bridge with the group_fwd_mask between the Ont and gateway.
 
Last edited:
  • Like
Reactions: das1996
Speeding up *sense bootup doesn't really matter. The eapol auth requests come in about every 30 seconds none stop until authorized. The bigger issue is just sending something back to the host early on before the network is up. If wan can't dhcp then it stalls for about a minute.

What other purpose would the delay serve?

I've been thinking of doing the wpa auth on proxmox itself too. I'm not even sure vlan0 is needed then. Will test both ways once I get my testbox going so I can fully experiment.

The other concern is what may be the security implications of having the host handle the auth. Best I can tell the "wan" interface won't have any ip or gateway info assigned to it at the host level.
 
Vlan0 not needed for the initial eap-auth.

Vlan0 still required for ATT DHCP/gateway access and normal internet traffic, by memory when I set this up.

by my understanding no host, gateway,dhcp, or dns traffic other than eAP-auth to/from att without vlan0, so not sure what security issues that presents other than theoretical. You are basically replicating the att fiber workaround others have done for years with switches Running Linux code.
 
Last edited:
Running this command in current firewall (sophos utm, suse based) where wan nic is in passthrough to the vm.

tcpdump -i eth2 port 67 or port 68 -n -e -A -vv

Doing dhcp renew, shows outbound traffic NOT vlan0 tagged, inbound is vlan0 tagged. However, no vlan0 interface is actually defined.

As for regular net traffic, im not seeing any vlan0 tags either. Eapol frames only showing vlan0 tags on inbound traffic, outbound does not appear to be tagged. Maybe it is tagged but because it is vlan0, it's not showing up¿?
 
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom
Back