[SOLVED] Proxmox and VLANs: Trying again, going simpler

[ADFHogan]

Hey folks,

I've asked some particularly specific questions in the past, and thought perhaps as they were so specific, they perhaps didn't have any general benefit.

I'd like to know how to both pass VLAN trunks through Proxmox, AND how to tap selected VLANs out of those trunks for some containers and VMs.

Eg. Pass a VLAN trunk through Proxmox between a firewall VM and physical networking hardware (so both the firewall, and the networking hardware can see the VLANs natively), but also being able to tap the trunk to pass it to specific VMs.

I understand VLAN theory, and how you can have tagged and untagged VLANs on a given virtual port, but haven't done it often on Linux itself so without examples, I'm a little stuck. I have seen some suggestions that say "just create a NIC per VLAN" ... but this seems like a bit of over abstraction in the case of the firewall VM and the physical VM hardware?

Does someone have some examples of how they've established a trunk through proxmox between VM and physical world, and then tapped select VLANs off the trunk for other specific VMs and containers?

I assume there's gotta be a way to do this, or a good reason not to?

 

[gurubert]

Create a VLAN-aware bridge interface in Proxmox with the physical interface being a port of it. A bonding interface (link aggregation) could also be the physical port of that bridge.

You can then create virtual interfaces for the VMs that either have a VLAN tag configured or not.

WIth a configured VLAN tag the interface in the VM will only see the traffic of this VLAN.
Without the tag in the configuration the VM will see all VLANs tagged.

 

[ADFHogan]

Ahh.. thanks, @gurubert .. that goes to explaining some more of it.. I'll try again!

How does it work when a VLAN aware bridge has untagged traffic going over it as well?

 

[Bingo600]

My guess (newbie too) , would be that you set the "untagged vlan" with this command
bridge-pvid xx
Then you set the "untagged" ip addr etc. , directly on the vmbr0

iface vmbr0 inet static
        address 192.168.1.18/24
        gateway 192.168.1.1
        bridge-pvid xx
        ..
        ..
        ..

If your untagged/native is vlan1 ... Then i don't have any idea , other than bridge-pvid 01, or maybe that's already the default

/Bingo

 

[ADFHogan]

So far I have:
* Direct mapped my WAN interface as a PCI device directly into the OPNsense firewall VM (as I don't want it going anywhere but the firewall VM)
* Added my default physical LAN interface enp4s0 to a VLAN aware bridge
* Assigned that bridge, vmbr0, IP 192.168.1.254/24
* Added virtio NIC on to firewall VM interface for LAN (192.168.1.1/24), and put that on the vmbr0 bridge
* Added tagged VLAN in OPNsense for IoT devices as VLAN 3 on LAN NIC
* Configured the IoT VLAN 3 to have range 192.168.3.1/24 and enabled DHCP between .20-.200 in OPNsense
* Left my admin LAN untagged
* Set my physical switch uplink port connected to enp4s0 to accept tagged and untagged lans, and to treat untagged as VLAN 1 within the network hardware
* Set all other physical switch ports for now to untagged VLAN 1, except for the WiFi AP ports, where I have set one SSID to VLAN 3 for IoT devices, and other SSID for VLAN 1 admin LAN

So far I can:
* Successfully acquire an IP and connect to the internet from untagged network (designated VLAN 1 inside physical network hardware)
* Successfully acquire an IP, but not connect to the internet, or ping OPNsense on the IoT LAN ... I CAN arping the gateway though - 192.168.3.1

Wondering where I've fluffed up

When I arping my 192.168.1.1 gateway on the untagged admin LAN, or the 192.168.3.1 gateway on the tagged IoT LAN, I get the MAC address of the virtual NIC in OPNsense which I expect is correct.

 

[Bingo600]

I have no experience with OPNsense, and especially not hosted on Proxmox.
On pfSense ... All "lans/Vlans" beside the "first lan/mgmt" starts with "no rules" , meaning everything is (default) deny'ed.
They "do cheat a bit", and will allow dhcp to pass ...

You might be figthing OPN' rules here

Compare rules on "Lan" vs IOT interface

Edit:
Ohh ... And arp (L2) would work even if the packets aren't allowed on the FW interface (L3).

 

[ADFHogan]

OPNsense was at some point forked from pfSense, so the logic seems reasonable.. I'll check it out...

EDIT: I think you're on to something..

Firewall -> Rules -> IoT has no rules
Firewall -> Rules -> LAN *does* have rules

EDIT: Yeap.. that's done it.. thanks, @Bingo600 !

 

[ADFHogan]

I've now been able to move my IoT SSID across to VLAN 3, with its own IP subnet, distinct from the default untagged LAN with its subnet.
Right now, it's freely able to route to other networks via the firewall, but ultimately I'll restrict it.

 

[ADFHogan]

I added a third untagged LAN connection from another device (security DVR), then updated the interface I connected it to, to tag all untagged traffic as specific VLAN, then added that to VLAN aware bridge, and it seems to be working as expected as well