Issue Creating Linux VLANs For VMs

[Arszilla]

Hi there,

I am trying to create 2 VLANs or several of my VMs. The 2 VLANs in question are VLAN10 and VLAN20. They're a part of my network topology as seen in the image below:



The issue here is that I have a VM or two that I need in VLAN10 (such as my AdGuardHome DNS server), while I plan to place every other VM in VLAN20 (Such as my Gitlab Runner).

As seen in the graph, Port 3 of my TL-SG108E switch is PVID 10, while Port 4-5-6 are PVID 20. The tags/untags on my switch are as follows:



Thus, I created 2 Linux Bridges in Proxmox, vmbr0 and vmbr1. I added 2 Linux VLANs, one under each Bridge and as a result, my /etc/network/interfaces looks like the following:

auto lo
iface lo inet loopback

iface eno1 inet manual

iface eno2 inet manual

iface eno3 inet manual

iface eno4 inet manual

iface ens2f0 inet manual

iface ens2f1 inet manual

auto vmbr0
iface vmbr0 inet static
        bridge-ports eno1
        bridge-stp off
        bridge-fd 0
        bridge-vlan-aware yes
        bridge-vids 2-4094

auto vmbr0.10
iface vmbr0.10 inet static
        address 10.10.10.5/24
        gateway 10.10.10.1

auto vmbr1
iface vmbr1 inet static
        bridge-ports eno2
        bridge-stp off
        bridge-fd 0
        bridge-vlan-aware yes
        bridge-vids 2-4094

auto vmbr1.20
iface vmbr1.20 inet static
        address 10.10.20.1/24

However, upon saving and either doing systemctl restart networking.service or rebooting the server, I am unable to access my Proxmox GUI.

Additionally, I tried just keeping vmbr0, and keeping it VLAN Aware, and adding vmbr0 to my VMs with the VLAN ID, but when I do that, my VMs (specifically my DNS server) are unable to talk to my OPNsense or the internet.

Can I request some assistance with this?

Thanks!

 

[Dunuin]

auto vmbr0
iface vmbr0 inet static

and

auto vmbr1
iface vmbr1 inet static

should be

iface vmbr0 inet manual

iface vmbr1 inet manual

The "auto vmbr0.10" and "auto vmbr1.20" is enough to autostart the bridges.
And an interace shouldn't be set to "static" hen not assigning a static IP.

I would just use a single VLAN aware bridge (vmbr0) and then set the VLAN tags for each guests virtual NIC. Except for the OPNsense VMs LAN NIC. There the "VLAN Tag" field should be kept empty, so the OPNsense can communbicate with all tagged VLANs. That way you can manage VLANs inside the OPNsense VM. Would also be useful to create a LACP bond and attach that to the vmbr0, so all your VLANs can share the same bond giving you better throughput and failover.

 

[Arszilla]

So from what I understand from your response, this is what I should end up with for my /etc/network/interfaces:

auto lo
iface lo inet loopback

iface eno1 inet manual

iface eno2 inet manual

iface eno3 inet manual

iface eno4 inet manual

iface ens2f0 inet manual

iface ens2f1 inet manual

iface vmbr0 inet manual
        bridge-ports eno1
        bridge-stp off
        bridge-fd 0
        bridge-vlan-aware yes
        bridge-vids 2-4094

auto vmbr0.10
iface vmbr0.10 inet static
        address 10.10.10.5/24
        gateway 10.10.10.1

auto vmbr0.20
iface vmbr0.20 inet static
        address 10.10.20.1/24

This should allow me to retain access to my Proxmox GUI from any device in VLAN30 (as I have a firewall rule allowing VLAN30 to access all of VLAN10 with no restrictions, compared to other VLANs which have a restricted access). Then, when I add my vmbr0 to a VM (like my DNS server), I should be good to go by just adding the PVID/VLAN ID. correct?

Would also be useful to create a LACP bond and attach that to the vmbr0, so all your VLANs can share the same bond giving you better throughput and failover.

I am unsure what you mean with this and how I can I achieve this. Mind elaborating a bit further?

 

[Dunuin]

This should allow me to retain access to my Proxmox GUI from any device in VLAN30 (as I have a firewall rule allowing VLAN30 to access all of VLAN10 with no restrictions, compared to other VLANs which have a restricted access). Then, when I add my vmbr0 to a VM (like my DNS server), I should be good to go by just adding the PVID/VLAN ID. correct?

Jup.

I am unsure what you mean with this and how I can I achieve this. Mind elaborating a bit further?

You got 6 NICs in your server. You use 2 for OPnsense (which might be fine in case you for example want to use PCI passthrough) and 4 NICs for your host or other guests. I personally would create a linux bond with those 4 NICs using LACP (best case "layer3+4" hash policy in caseyour switch supports that) and trunk all VLANs over that bond. That way the traffic is split between those 4 links and you get a 4 Gbit bandwidth (in case those are 1Gbit NICs) when doing multiple connections in parallel. And in case a NIC fails you still have a working connection.

For exmaple useful in case your windows-desktop wants to access a VM with 1Gbit and your iphone wants to access another VM with 1 Gbit at the same time. Then both devices can access the server with full Gbit instead of just 500Mbit, as the server got a 4Gbit bandwidth so its not bottlenecking.

Same with the OPNsense VM. I gave my OPNsense VM a quadport NIC and created a bond. That way my OPNsense can route with full Gbit (or een up to 2Gbit) between VLANs. Like you got it now, routing between VLAN20 and 30 would be limited to 500Mbit, as both need to share the same single Gbit NIC.

https://pve.proxmox.com/wiki/Network_Configuration#sysadmin_network_bond

 

[Arszilla]

So, upon applying the new /etc/network/interfaces config and rebooting my server, I am once again unable to access my Proxmox's UI. I can't seem to ping it from 10.10.30.2 nor 10.10.10.1 (OPNsense's Web UI). The server is plugged on from NIC1-1 to Port 3 and Port 3 is untagged on VLAN10 and VLAN20 on my switch (and the PVID is 10).

When I tried to ping the OPNsense FW or my laptops, all I got was this:

$ ping 10.10.10.1
PING 10.10.10.1 (10.10.10.1) 56(84) bytes of data.
From 10.10.10.5 icmp_seq=1 Destination Host Unreachable
From 10.10.10.5 icmp_seq=2 Destination Host Unreachable
From 10.10.10.5 icmp_seq=3 Destination Host Unreachable

Additionally, my ip r is below:

$ ip r
default via 10.10.10.1 dev vmbr0.10 proto kerne; onlink
10.10.10.0/24 dev vmbr0.10 proto kernel scope link src 10.10.10.5
10.10.20.0/24 dev vmbr0.20 proto kernel scope link src 10.10.20.1

Any idea how can I fix this? It does not seem to be a firewall issue, as I do not see any "deny" etc. happening in OPNsense.

 

[mjtbrady]

There are multiples ways of putting some thing like this together.

Personally I strongly prefer not to use PCIe passthrough as it is too easy to break and it makes it harder to move the VM to another host. Personally I would have one interface with a bridge on it for WAN and bond all the other interfaces and place a vlan aware bridge on the bond for everything else.

Having said that, these are personal preferences on my part and are not your issue. The issue is that vlan30 isn't connected to anything. To get from vlan30 (10.10.30.0/24) to vlan10 (10.10.10.0/24) you need to go through a router (I assume OPNsense in this case). But based on you interfaces file and your diagram, OPNsense and the underlying Proxmox server networking aren't connected to vlan30.

 

[Arszilla]

That's the thing, by the looks of it, my firewall rules are solid:

For VLAN10:


For VLAN30:


I am not using my AdGuard DNS as my DNS server for now as I am trying to move it around, thus all my queries are being processed by OPNsense by default.

Again, I also tried accessing the server from VLAN10 itself. Currently, since my TL-EAP225 is not properly configured, it's at VLAN10, thus all my WiFi devices are in VLAN10 (i.e. kali-thinkpad, which has an IP of 10.10.10.16), and even that can't access it.

Lastly, on OPNsense, I cannot see any blocks etc. to 10.10.10.5 happening. The requests are clearly arriving to VLAN10, but since Proxmox is refusing to join the VLAN (by my assumption), I can't connect to it.

It should be noted that I am using nginx to serve the panel over on Port 443, instead of Port 8006.

 

[mjtbrady]

That's the thing, by the looks of it, my firewall rules are solid:

For VLAN10:
View attachment 44786

For VLAN30:
View attachment 44787

I am not using my AdGuard DNS as my DNS server for now as I am trying to move it around, thus all my queries are being processed by OPNsense by default.

Again, I also tried accessing the server from VLAN10 itself. Currently, since my TL-EAP225 is not properly configured, it's at VLAN10, thus all my WiFi devices are in VLAN10 (i.e. kali-thinkpad, which has an IP of 10.10.10.16), and even that can't access it.

Lastly, on OPNsense, I cannot see any blocks etc. to 10.10.10.5 happening. The requests are clearly arriving to VLAN10, but since Proxmox is refusing to join the VLAN (by my assumption), I can't connect to it.
View attachment 44788
It should be noted that I am using nginx to serve the panel over on Port 443, instead of Port 8006.

Ok so the diagram is all wrong. I should have paid more attention to the switch config.

What does your interfaces file look like now?

What ports on the switch are eno1 and eno2 connected to?

 

[Arszilla]

My current /etc/network/interfaces config:

auto lo
iface lo inet loopback

iface eno1 inet manual

iface eno2 inet manual

iface eno3 inet manual

iface eno4 inet manual

iface ens2f0 inet manual

iface ens2f1 inet manual

iface vmbr0 inet manual
        bridge-ports eno1
        bridge-stp off
        bridge-fd 0
        bridge-vlan-aware yes
        bridge-vids 2-4094

auto vmbr0.10
iface vmbr0.10 inet static
        address 10.10.10.5/24
        gateway 10.10.10.1

auto vmbr0.20
iface vmbr0.20 inet static
        address 10.10.20.1/24

Notice how there's no auto vmbr0 above iface vmbr0 inet manual.

My switch configuration is below:




eno1 is plugged into Port 3 on my switch. Currently, since eno2, eno3 and eno4 are not active, they're not plugged into the switch (for now). But if my eno2 was active, it'd be plugged into Port 4.

 

[mjtbrady]

My current /etc/network/interfaces config:

auto lo
iface lo inet loopback

iface eno1 inet manual

iface eno2 inet manual

iface eno3 inet manual

iface eno4 inet manual

iface ens2f0 inet manual

iface ens2f1 inet manual

iface vmbr0 inet manual
        bridge-ports eno1
        bridge-stp off
        bridge-fd 0
        bridge-vlan-aware yes
        bridge-vids 2-4094

auto vmbr0.10
iface vmbr0.10 inet static
        address 10.10.10.5/24
        gateway 10.10.10.1

auto vmbr0.20
iface vmbr0.20 inet static
        address 10.10.20.1/24

Notice how there's no auto vmbr0 above iface vmbr0 inet manual.

My switch configuration is below:
View attachment 44793

View attachment 44794

eno1 is plugged into Port 3 on my switch. Currently, since eno2, eno3 and eno4 are not active, they're not plugged into the switch (for now). But if my eno2 was active, it'd be plugged into Port 4.

Ok.

I think that Port 3 on the switch should only have tagged traffic on it. Remove all the untagged vlans and add vlans 10 and 20 as tagged.

On Proxmox Remove vmbr0.20 for now.

Test access to Proxmox.

 

[Arszilla]

So that kind of fixed it! I just tagged Port 3 on VLAN10 and VLAN20 and now I have access to the server! Currently vmbr0.10 is working, and both Proxmox and AdGuard are working all perfectly!

I spun up a Ubuntu Desktop VM to test VLAN20 functionality with vmbr0.20, but it's broken by the looks of it. The VM gets an IP etc. but has no proper internet, similar to the Proxmox server earlier today. Can't ping OPNsense, 8.8.8.8, etc.

 

[mjtbrady]

With a vlan aware bridge you shouldn't have a vmbr0.20. Remove it.

Instead in the VM's network configuration specify Bridge=vmbr0 and VLAN Tag=20.

For instance one of my VMs on vlan 42.


If you want to have a bridge per VLAN instead of one VLAN aware bridge somethings need to be changed.

 

[vesalius]

This may not be the only issue but could be causing other problems at some time, on the switch, unless I am mistaken you should not have ant most 1 untagged vlan per switch port. You can have multiple tagged vlans per switchport though.

1. when a network Packet arrives at the switch port without a vlan tag (you do not tag it at Proxmox), then the switch will add a vlan tag to that packet with the vlan listed as untagged on the switch port Or if none programmed in by you pass the packet with default vlan (PVID), so if multiple vlans listed as untagged this process is nonsensical. Important to remember the switch port will also drop external packets already tagged with any vlan listed as untagged on the switch port.
2. when a network packet arrives at that port already tagged (you tagged it at proxmox) the switch port will either allow that packet through (if the vlan is listed as tagged on the switch port) or otherwise the packet is dropped.