Proxmox 7.0-11 management on VLAN with Linux Bond

[liam135db]

Hi,
long time listener, first time caller

I am trying to set up the VE 7.0-11 management interface on a VLAN through a Linux Bond.
System is 3x Dell R620s and R730s which I want to cluster into a 6 node Ceph storage and compute cluster

I am trying to bond 4x 10GB links on dell servers using balance-alb.
I had the bond, then setup a Linux VLAN (bond0.4) followed by a bridge pointing at the VLAN

I tried following all three of the examples in the documentation chapter "Network Configuration > Linux Bond" and none of them worked.
[edit] I have already tried commenting out the "Auto" IF in /etc/network/interfaces with no joy in this case (it did solve the issue on my older smaller cluster).

Can anyone confirm if they are having any success with a similar setup?
Thanks

 

[spirit]

do you use ifupdown or ifupdown2 ? (ifupdown2 should be the default for fresh 7.0 install).

It seem that they are bug with bonding with ifupdown1 currently (the auto if...)
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=968368
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=990428

can you post your /etc/network/interfaces ?

 

[liam135db]

Thanks for your reply.
Must've been a typo in the file as I tried it again today and got it working.
Guess that's working too late on it!


although I did mistakenly delete a line taken from the example without affecting it. I'm not sure if the "iface bond0.4" definition is needed as the VLAN is already connected to vmbr0.4?

I also notice that when changing the VLAN aware switch from the gui it adds the line "Bridge-vids 2-4094" but they don't appear in the examples so not sure if that is required or not in this version?

auto lo
iface lo inet loopback

iface eno1 inet manual

iface eno2 inet manual

iface eno3 inet manual

iface eno4 inet manual

auto bond0
iface bond0 inet manual
bond-slaves eno1 eno2 eno3 eno4
bond-miimon 100
bond-mode balance-alb

iface bond0.4 inet manual

auto vmbr0v4
iface vmbr0v4 inet static
address 192.168.255.182/24
gateway 192.168.255.1
bridge-ports bond0.4
bridge-stp off
bridge-fd 0

auto vmbr0
iface vmbr0 inet manual
bridge-ports bond0
bridge-stp off
bridge-fd 0
bridge-vlan-aware yes

 

[spirit]

ok, be carefull, if you use vlan-aware bridge, you should use bond0.4 linux tagging. (mixing tagging inside the vlan-awara bridge and linux interface don't work well).


So, if you use vlan-aware bridge :

auto vmbr0.4
iface vmbr0.4 inet static
address 192.168.255.182/24
gateway 192.168.255.1


auto vmbr0
iface vmbr0 inet manual
bridge-ports bond0
bridge-stp off
bridge-fd 0
bridge-vlan-aware yes

or without vlan aware bridge
(don't need iface bond0.4 inet manual)

auto vmbr0v4
iface vmbr0v4 inet static
address 192.168.255.182/24
gateway 192.168.255.1
bridge-ports bond0.4
bridge-stp off
bridge-fd 0

auto vmbr0
iface vmbr0 inet manual
bridge-ports bond0
bridge-stp off
bridge-fd 0

 

[liam135db]

ok, be carefull, if you use vlan-aware bridge, you should use bond0.4 linux tagging. (mixing tagging inside the vlan-awara bridge and linux interface don't work well).

Sorry, did you mean I shouldn't use Bond0.4 tagging for the management interface?
does the "vmbr" in vmbr0.4 description direct that traffic through vmbr0 even though it has bridge ports set to bond0.4 then?


I tried to fire up an ubuntu VM tagged to VLAN2 in the hardware settings for the VM and only seemed able to ping the gateway after adding the line
so it does seem that the vlan aware bridge is needed with the tags to get the traffic out to the network

auto vmbr0
iface vmbr0 inet manual
bridge-ports bond0
bridge-stp off
bridge-fd 0
bridge-vlan-aware yes
bridge-vids 2-4094

 

[liam135db]

when I tried this I lost all connection to the node and i am struggling to edit /etc/network/interfaces to bring it back online as the terminal just keeps filling with libceph errors that overwrite what I am trying to type!

[edit] I got it back but it was a bit awkward as I couldn't suppress the error messages

So, if you use vlan-aware bridge :

Code:
auto vmbr0.4
iface vmbr0.4 inet static
address 192.168.255.182/24
gateway 192.168.255.1


auto vmbr0
iface vmbr0 inet manual
bridge-ports bond0
bridge-stp off
bridge-fd 0
bridge-vlan-aware yes

or without vlan aware bridge

 

[liam135db]

OK so I think I have found the solution.

I followed your second suggestion of not using a VLAN aware bridge.
wierdly, changing /etc/network/interfaces and running "ifreload -a" wouldn't allow any VMs to access the VLANs which was why I made the bridge VLAN aware in the first place, BUT rebooting the server results in full functionality.

checking "ip a" shows that with the VMs tagged at VM hardware level it creates the bon0.x interfaces as required.

Thanks for your help, it drove me to keep working on trying to find out whhat was misbehaving and it seems to have been the difference between a reload and a reboot.