VLAN-Configuration in Cluster PVE 8.0.3

[sysadmin_97]

Hello everyone,

I just recently set up two Proxmox-Servers (Intel NUC) and joined them into a cluster. Now I want to take the next step and separate my network into VLANs.
I plan to set up a pfSense or OPNsense box (in Proxmox of course) which will service the VLANs.

Both systems run the latest pve-manager/8.0.3/bbf3993334bfa916 (running kernel: 6.2.16-3-pve).

The Plan is:
VLAN10 = Management
VLAN20 = (Web-)Servers
VLAN30 = Computers
VLAN40 = IoT

My Proxmox-Servers have only one NIC so the Proxmox-Management Website has to be reachable either over VLAN1 or VLAN10.

Since I'm pretty new to networking I'm not sure how to configure this in Proxmox. I've seen a few YouTube Tutorials but to be completely honest they confused me more than further my goal.

I somehow ended up with this config. Will this do for my use-case or will this even work in a Cluster set-up?

auto lo
iface lo inet loopback

auto eno1
iface eno1 inet manual

auto vmbr0
iface vmbr0 inet static
    address 10.10.10.3
    gateway 10.10.10.1
    netmask 255.255.255.0
    bridge_ports eno1
    bridge_stp off
    bridge_fd 0

auto eno1.1
iface eno1.1 inet manual
    vlan-raw-device eno1

auto eno1.2
iface eno1.2 inet manual
    vlan-raw-device eno1

auto eno1.3
iface eno1.3 inet manual
    vlan-raw-device eno1
    
auto eno1.4
iface eno1.4 inet manual
    vlan-raw-device eno1

auto vmbr0.1
iface vmbr0.1 inet static
    address 192.168.10.2
    gateway 192.168.10.1
    netmask 255.255.255.0
    bridge_ports eno1.1
    bridge_stp off
    bridge_fd 0

auto vmbr0.2
iface vmbr0.2 inet static
    address 192.168.20.2
    gateway 192.168.20.1
    netmask 255.255.255.0
    bridge_ports eno1.2
    bridge_stp off
    bridge_fd 0

auto vmbr0.3
iface vmbr0.3 inet static
    address 192.168.30.2
    gateway 192.168.30.1
    netmask 255.255.255.0
    bridge_ports eno1.3
    bridge_stp off
    bridge_fd 0
    
auto vmbr0.4
iface vmbr0.4 inet static
    address 192.168.40.2
    gateway 192.168.40.1
    netmask 255.255.255.0
    bridge_ports eno1.4
    bridge_stp off
    bridge_fd 0

Any feedback is greatly appreciated!

 

[spirit]

you don't need to mix "vmbr0.X" + "eno1.Y" , or you are going to do double tag vlan ^_^.

vmbr0.X is if you want to use vlan aware.

you don't need to define vm vlan && (vms) ip . you just need to choose the tag at the vm nic level, and configure vm ip inside the vm os.

for /etc/network/interfaces, you just need something like

auto vmbr0
iface vmbr0 inet static
    address 10.10.10.3
    gateway 10.10.10.1
    netmask 255.255.255.0
    bridge_ports eno1
    bridge_stp off
    bridge_fd 0

then configure tag on the vm nic + vmbr0 as bridge.

(This is the management ip on default vlan1).

now if you want management on vlan10


without vlan-aware bridge (tag on physical interface)

auto vmbr0
iface vmbr0 inet manual
    bridge_ports eno1
    bridge_stp off
    bridge_fd 0

auto eno1.10
iface eno1.10 inet static
    address 10.10.10.3
    gateway 10.10.10.1
    netmask 255.255.255.0

or with vlan-aware bridge (tag on bridge)

auto vmbr0
iface vmbr0 inet manual
    bridge_ports eno1
    bridge_stp off
    bridge_fd 0
    bridge-vlan-aware yes
    bridge-vids 10,20,30,40

auto vmbr0.10
iface vmbr0.10 inet static
    address 10.10.10.3
    gateway 10.10.10.1
    netmask 255.255.255.0

 

[Spoonman2002]

Or install and use OpenvSwitch (OVS).
Makes life a bit easier.

 

[spirit]

Or install and use OpenvSwitch (OVS).
Makes life a bit easier.

with more bugs

 

[Spoonman2002]

with more bugs

I would very much like to hear them.....(seriously).

 

[spirit]

I would very much like to hear them.....(seriously).

https://forum.proxmox.com/threads/u...reaks-network-connectivity.121237/post-530553
https://forum.proxmox.com/threads/openvswitch-switch-upgrade-bricht-netzwerk.123032/

(is has been fixed, but this bug was present for 1 or 2 year in upstream ovs, where the control plane was breaking network on update)

They are no advantage to use ovs until you want to use dpdk, but it's not implemented in proxmox anyway.
no userland daemon = no problem.

 

[Spoonman2002]

https://forum.proxmox.com/threads/u...reaks-network-connectivity.121237/post-530553
https://forum.proxmox.com/threads/openvswitch-switch-upgrade-bricht-netzwerk.123032/

(is has been fixed, but this bug was present for 1 or 2 year in upstream ovs, where the control plane was breaking network on update)

They are no advantage to use ovs until you want to use dpdk, but it's not implemented in proxmox anyway.
no userland daemon = no problem.

Thanks for the reply.
I know and have experienced for myself that an openvswitch update can indeed break connectivity.
Then someone had the bright idea of adding "ifup -a" at the end of the pveupgrade command.
(or something similar, that the last command is to bring up the interface. Not sure if that is ifup -a).

 

[spirit]

Thanks for the reply.
I know and have experienced for myself that an openvswitch update can indeed break connectivity.
Then someone had the bright idea of adding "ifup -a" at the end of the pveupgrade command.
(or something similar, that the last command is to bring up the interface. Not sure if that is ifup -a).

this has been fixed by proxmox team, backporting a fix not present in debian ovs package

https://git.proxmox.com/?p=ovs.git;a=commit;h=5322884328bfe7ce3945680dcffb2860f6e2d0fa

 

[sysadmin_97]

with vlan-aware bridge (tag on bridge)

auto vmbr0
iface vmbr0 inet manual
    bridge_ports eno1
    bridge_stp off
    bridge_fd 0
    bridge-vlan-aware yes
    bridge-vids 10,20,30,40

auto vmbr0.10
iface vmbr0.10 inet static
    address 10.10.10.3
    gateway 10.10.10.1
    netmask 255.255.255.0

Thank you for the response. You helped my quite a bit to understand it better.

I do have a question though. If my config is like this, will the Management-Interface automatically be assigned to VLAN10 with 10.10.10.3 since it's the first VLAN or do I have to explicitly configure it in the vmbr0 block to be that way?

auto vmbr0
iface vmbr0 inet manual
    bridge_ports eno1
    bridge_stp off
    bridge_fd 0
    bridge-vlan-aware yes
    bridge-vids 10,20,30,40

auto vmbr0.10
iface vmbr0.10 inet static
    address 10.10.10.3
    gateway 10.10.10.1
    netmask 255.255.255.0

auto vmbr0.20
iface vmbr0.20 inet static
    address 192.168.20.2
    gateway 192.168.20.1
    netmask 255.255.255.0

auto vmbr0.30
    address 192.168.30.2
    gateway 192.168.30.1
    netmask 255.255.255.0

auto vmbr0.40
    address 192.168.40.2
    gateway 192.168.40.1
    netmask 255.255.255.0

I think I'll go with the tagging on the bridge since it sounds like it's easier to configure with VM to VM communication.
I guess if I want to configure my other servers network the same way so that the systems can communicate with each other I have to use the next free IP-Address, right?
Just out of interest: Is there a performance penalty between both options?

 

[spirit]

I do have a question though. If my config is like this, will the Management-Interface automatically be assigned to VLAN10 with 10.10.10.3 since it's the first VLAN or do I have to explicitly configure it in the vmbr0 block to be that way?

you need to defined explicitly with vmbr0.10 . (vmbr0 is default vlan=1)

I think I'll go with the tagging on the bridge since it sounds like it's easier to configure with VM to VM communication.
I guess if I want to configure my other servers network the same way so that the systems can communicate with each other I have to use the next free IP-Address, right?

yep.

note: you can't use multiple default gw. and you don't need to defined other vmbr0.X with ip address.
for the vms, you just need to configure the tag on the vm nic and configure ip inside your vm os.
(vmbr0 is just a switch, ips are only to manage your proxmox node)

Just out of interest: Is there a performance penalty between both options?

no, performance is same.