Proxmox Network configuration for bonded nics using LACP and also including VLANs for

[techguys]

the VMs.

While I've comfortable with networking, virtualization and linux, I'm a relative newbie to both proxmox and Debian. I'm clearly missing something on configuring the network and am hoping to get some help.

What I'm trying to do is the following:

I have 4 nic's in the proxmox server, and I'm trying to create 2 bonds of 2 nic's each. 1 for Storage only (vlan6) and then 1 for everything else. Server management is in vlan1 and then I'm hoping to use vlan1 11 for VM's. The subnets used are as follows:
vlan1: 10.9.0.0/24
vlan11: 10.9.11.0/24
Both on bond0

vlan6: 10.6.6.0/24
Bond1:
LACP is working fine on both sides (I've got a Cisco 3750 switch and have used etherchannel for years).

What doesn't work is vlan11 for the vm's. I'm sure that I've just misunderstand the 'container' concept here... and basically vmbr11 doesn't seem to work and I can't see any traffic on vlan11.

Here's my interfaces file:
# network interface settings
auto lo
iface lo inet loopback
iface eth0 inet manual
iface eth1 inet manual
iface eth2 inet manual
iface eth3 inet manual
auto bond0
iface bond0 inet manual
slaves eth0 eth3
bond_miimon 100
# Bond Mode 802.3ad
bond_mode 4
pre-up ifup eth0 eth3
post-down ifdown eth0 eth3
auto bond0.11
iface bond0.11 inet manual
vlan-raw-device bond0
auto bond1
iface bond1 inet static
address 10.9.6.10
netmask 255.255.255.0
slaves eth1 eth2
bomd_miimon 100
bond_mode 802.3ad
pre-up ifup eth1 eth2
post-down ifdown eth1 eth2
auto vmbr0
iface vmbr0 inet static
address 10.9.0.10
netmask 255.255.255.0
gateway 10.9.0.254
bridge_ports bond0
bridge_stp off
bridge_fd 0
auto vmbr11
iface vmbr11 inet static
address 10.9.11.253
netmask2 255.255.255.0
bridge_ports bond0.11
bridge_stp off
bridge_fd 0

I've tried vmbr11 both with and without an address...

thanks in advance.

 

[udo]

Re: Proxmox Network configuration for bonded nics using LACP and also including VLANs

Hi,
you used mixed vlan-tagging and default network which not work reliable.
If you use bond0.1 for vmbr0 and enable vlan-tagging for vlan1 on this ports, it should work.

Udo

 

[techguys]

Re: Proxmox Network configuration for bonded nics using LACP and also including VLANs

Hi Udo,

Thanks very much for the quick response! I've switched it to use bond0.1 now. eth0 and eth3 didn't come up automatically so I brought them up manually which brought up the port-channel on the switch side and also vmbr0. I can't get to anything in vlan1 though so I'm still missing something. I marked the trunk to the proxmox server as native vlan to 1.

I'm not clear on where/how to enable vlan-tagging on the proxmox/debian side?

Thanks.

 

[techguys]

Re: Proxmox Network configuration for bonded nics using LACP and also including VLANs

So interestingly enough... vmbr0 and vmbr11 both come up now and the local IP's are pingable on itself but not actually available on the network. Here's where I'm at now:
# network interface settings
auto lo
iface lo inet loopback
iface eth0 inet manual
iface eth1 inet manual
iface eth2 inet manual
iface eth3 inet manual

auto bond0
iface bond0 inet manual
slaves eth0 eth3
bond_miimon 100
# Bond Mode 802.3ad
bond_mode 4
pre-up ifup eth0 eth3
post-down ifdown eth0 eth3

auto bond0.1
iface bond0.1 inet manual
vlan-raw-device bond0

auto bond0.11
iface bond0.11 inet manual
vlan-raw-device bond0

auto bond1
iface bond1 inet static
address 10.9.6.10
netmask 255.255.255.0
slaves eth1 eth2
bomd_miimon 100
bond_mode 802.3ad
pre-up ifup eth1 eth2
post-down ifdown eth1 eth2

auto vmbr0
iface vmbr0 inet static
address 10.9.0.10
netmask 255.255.255.0
gateway 10.9.0.254
bridge_ports bond0.1
bridge_stp off
bridge_fd 0

auto vmbr11
iface vmbr11 inet static
address 10.9.11.253
netmask 255.255.255.0
bridge_ports bond0.11
bridge_stp off
bridge_fd 0

 

[techguys]

Re: Proxmox Network configuration for bonded nics using LACP and also including VLANs

So where I'm at now... is that I can either have vmbr0 or vmbr11 work but not at the same time. If I have an IP address on vmbr11 that's what takes over and I can't use the ip on vmbr0.

Maybe I'm trying to do something that I can't do, or? Am I just doing this incorrectly?

I'd really like to have the VM traffic all be in vlan11 (10.9.11 subnet) and the 'management' of the proxmox server in vlan1 on the 10.9.0 subnet.

Should I just not use a 'container' for vmbr0 in the first place is that usually only for VM networks? And put the 10.9.0.10 address that I wanted for this server directly on bond0.1 ?

 

[udo]

Re: Proxmox Network configuration for bonded nics using LACP and also including VLANs

So where I'm at now... is that I can either have vmbr0 or vmbr11 work but not at the same time. If I have an IP address on vmbr11 that's what takes over and I can't use the ip on vmbr0.

Maybe I'm trying to do something that I can't do, or? Am I just doing this incorrectly?

I'd really like to have the VM traffic all be in vlan11 (10.9.11 subnet) and the 'management' of the proxmox server in vlan1 on the 10.9.0 subnet.

Should I just not use a 'container' for vmbr0 in the first place is that usually only for VM networks? And put the 10.9.0.10 address that I wanted for this server directly on bond0.1 ?

Hi,
pve need vmbr0 for comunication (if you want to use a cluster).
I had no experience with bonds, but my normal config with vlans work very well. But I use only IP at the host on vmbr0 - all other bridges are for the VMs only (but I'm sure that ip on the host will work too).
Here my (shorten) running config:

auto lo
iface lo inet loopback

auto eth0
iface eth0 inet static
        address  0.0.0.0
        netmask  0.0.0.0

auto eth0.20
iface eth0.20 inet static
        address  0.0.0.0
        netmask  0.0.0.0

auto eth0.22
iface eth0.22 inet static
        address  0.0.0.0
        netmask  0.0.0.0

auto eth0.30
iface eth0.30 inet static
        address  0.0.0.0
        netmask  0.0.0.0

auto vmbr0
iface vmbr0 inet static
        address  172.20.4.11
        netmask  255.255.255.0
        gateway  172.20.4.254
        bridge_ports eth0.22
        bridge_stp off
        bridge_fd 0

auto vmbr20
iface vmbr20 inet manual
        bridge_ports eth0.20
        bridge_stp off
        bridge_fd 0

auto vmbr30
iface vmbr30 inet manual
        bridge_ports eth0.30
        bridge_stp off
        bridge_fd 0

Hope it's help.

Udo

 

[techguys]

Re: Proxmox Network configuration for bonded nics using LACP and also including VLANs

Thanks for the continued help.

So to simplify I got rid of the bond on eth0 and eth3 to try to get this sorted out. I've left eth0 in vlan1 and moved eth3 to vlan 11 (on the switch) which is what I want the VM network to be in... then I changed vmbr11 to:

auto vmbr
iface vmbr11 inet manual
bridge_ports eth3
bridge_stp off
bridge_fd 0

and then also:

auto vmbr11
iface vmbr11 inet static
address 10.9.11.253
netmask 255.255.255.0
bridge_ports eth3
bridge_stp off
bridge_fd 0

I can't get any network traffic to/from the VMs at all.

 

[techguys]

Re: Proxmox Network configuration for bonded nics using LACP and also including VLANs

Just to clarify... neither of the above 2 configurations worked for the VM's.

 

[udo]

Re: Proxmox Network configuration for bonded nics using LACP and also including VLANs

Thanks for the continued help.

So to simplify I got rid of the bond on eth0 and eth3 to try to get this sorted out. I've left eth0 in vlan1 and moved eth3 to vlan 11 (on the switch) which is what I want the VM network to be in... then I changed vmbr11 to:

auto vmbr
iface vmbr11 inet manual
bridge_ports eth3
bridge_stp off
bridge_fd 0

and then also:

auto vmbr11
iface vmbr11 inet static
address 10.9.11.253
netmask 255.255.255.0
bridge_ports eth3
bridge_stp off
bridge_fd 0

I can't get any network traffic to/from the VMs at all.

Hi,
at the first auto is the vmbr-number missing.
But to find the mistake try this:
First - you have restartet the proxmox-host after changing the interfaces-file?
if you define vmbr11 which are bridged to eth3 and you startet a VM, which are connectet to vmbr11 (btw. which os, nic?).
If you now ping from the vm to a machine in your network, you can do a tcpdump on the proxmoxhost on eth3 - and you must see the packets (both directions)!

Udo

 

[techguys]

Re: Proxmox Network configuration for bonded nics using LACP and also including VLANs

Hey Udo,

Thanks again... It's working now Must have had it right but it needed a reboot I guess.

I've got an ESXi and Hyper-V host for testing with other VM's and I can get to and from all VM's regardless of host now.

Now I've just got to get Jumbo Frames working on Proxmox, and then decide whether in the long run to use iSCSI or NFS!

Thanks.

 

[udo]

Re: Proxmox Network configuration for bonded nics using LACP and also including VLANs

Hey Udo,

Thanks again... It's working now Must have had it right but it needed a reboot I guess.

I've got an ESXi and Hyper-V host for testing with other VM's and I can get to and from all VM's regardless of host now.

Now I've just got to get Jumbo Frames working on Proxmox, and then decide whether in the long run to use iSCSI or NFS!

Thanks.

Hi,
fine that's works! For Jumbo frame on a seperate NIC:

auto eth1
iface eth1 inet static
    address  172.20.4.10
    netmask  255.255.255.0
    mtu 9000

eth1 in this case instead vmbrX, because it's for hostcomunication (drbd) and not for the VMs.

Udo

 

[techguys]

Re: Proxmox Network configuration for bonded nics using LACP and also including VLANs

Got it sorted with:
pre-up ifconfig eth1 mtu 9000
pre-up ifconfig eth2 mtu 9000

on the bond0 config...

works well

Thanks.