Bond + trunk with SAN

[kameleon]

I have successfully setup one of my proxmox servers to have all 4 nics bonded together with vlans passing it and then various vmbrX to represent the various interfaces connected to the vlans so I can pass them to the virtual machines. What I now need is to give one of the interfaces a static IP to connect to my SAN (Dell MD3200i). I must be typing something wrong because every time I add it to the /etc/network/interfaces and restart the network it refuses to connect to any network. I got all my pointers so far from here: http://bitbud.com/2010/01/07/trunks-bonds-bridges-and-vlans-in-proxmox/ and he states in there that he set vmbr30 to be his SAN traffic bridge. So I set mine similar. Below is my /etc/network/interfaces file with the offending section commented out:

auto lo
iface lo inet loopback

# interface GigabitEthernet5/45 on the core switch
auto eth0
iface eth0 inet manual

# interface GigabitEthernet5/46 on the core switch
auto eth1
iface eth1 inet manual

# interface GigabitEthernet5/47 on the core switch
auto eth2
iface eth2 inet manual

# interface GigabitEthernet5/48 on the core switch
auto eth3
iface eth3 inet manual

auto bond0
iface bond0 inet manual
slaves eth0 eth1 eth2 eth3
bond_miimon 100
bond_mode 4

auto bond0.1
iface bond0.1 inet manual
vlan-raw-device bond0

auto bond0.2
iface bond0.2 inet manual
vlan-raw-device bond0

auto bond0.3
iface bond0.3 inet manual
vlan-raw-device bond0

auto bond0.4
iface bond0.4 inet manual
vlan-raw-device bond0

auto bond0.5
iface bond0.5 inet manual
vlan-raw-device bond0

auto bond0.6
iface bond0.6 inet manual
vlan-raw-device bond0

auto bond0.7
iface bond0.7 inet manual
vlan-raw-device bond0

auto bond0.8
iface bond0.8 inet manual
vlan-raw-device bond0

auto bond0.9
iface bond0.9 inet manual
vlan-raw-device bond0

auto bond0.10
iface bond0.10 inet manual
vlan-raw-device bond0

auto bond0.11
iface bond0.11 inet manual
vlan-raw-device bond0

auto bond0.12
iface bond0.12 inet manual
vlan-raw-device bond0

auto bond0.13
iface bond0.13 inet manual
vlan-raw-device bond0

auto bond0.14
iface bond0.14 inet manual
vlan-raw-device bond0

auto bond0.15
iface bond0.15 inet manual
vlan-raw-device bond0

auto bond0.20
iface bond0.20 inet manual
vlan-raw-device bond0

auto vmbr0
iface vmbr0 inet static
        address 10.8.2.40
        netmask 255.255.255.0
        gateway 10.8.2.1
        bridge_ports bond0.2
        bridge_stp off
        bridge_fd 0

auto vmbr1
iface vmbr1 inet manual
bridge_ports bond0.1
bridge_stp off
bridge_fd 0

auto vmbr2
iface vmbr2 inet manual
bridge_ports bond0.2
bridge_stp off
bridge_fd 0

auto vmbr3
iface vmbr3 inet manual
bridge_ports bond0.3
bridge_stp off
bridge_fd 0

auto vmbr4
iface vmbr4 inet manual
bridge_ports bond0.4
bridge_stp off
bridge_fd 0

auto vmbr5
iface vmbr5 inet manual
bridge_ports bond0.5
bridge_stp off
bridge_fd 0

auto vmbr6
iface vmbr6 inet manual
bridge_ports bond0.6
bridge_stp off
bridge_fd 0

auto vmbr7
iface vmbr7 inet manual
bridge_ports bond0.7
bridge_stp off
bridge_fd 0

auto vmbr8
iface vmbr8 inet manual
bridge_ports bond0.8
bridge_stp off
bridge_fd 0

auto vmbr9
iface vmbr9 inet manual
bridge_ports bond0.9
bridge_stp off
bridge_fd 0

auto vmbr10
iface vmbr10 inet manual
bridge_ports bond0.10
bridge_stp off
bridge_fd 0

auto vmbr11
iface vmbr11 inet manual
bridge_ports bond0.11
bridge_stp off
bridge_fd 0

auto vmbr12
iface vmbr12 inet manual
bridge_ports bond0.12
bridge_stp off
bridge_fd 0

auto vmbr13
iface vmbr13 inet manual
bridge_ports bond0.13
bridge_stp off
bridge_fd 0

auto vmbr14
iface vmbr14 inet manual
bridge_ports bond0.14
bridge_stp off
bridge_fd 0

auto vmbr15
iface vmbr15 inet manual
bridge_ports bond0.15
bridge_stp off
bridge_fd 0

#auto vmbr20
#iface vmbr20 inet static
#address 192.168.130.50
#netmask 255.255.0.0
#bridge_ports bond0.20
#bridge_stp off
#bridge_fd 0

My ip ranges are 10.8.X.1 where the X is the # of the vlan with the exception of vlan20 which is the san traffic and is on the 192.168.X.X subnet.


Any pointers are welcome.

 

[udo]

Hi,
i'm wonder, that you use in vmbr0 and vmbr2 bond0.2 - is there an typo (perhaps bond0.20)? Normaly you can't bond one device to different bridges.

Do you see any hint with

brctl show
ip route

Udo

 

[kameleon]

That did not occur to me but makes sense. I just had the vmbr0 on vlan2 but really want it on vlan1. Vlan20 is the san vlan so the management interface can't be there either. Once I return to work on Monday I will try removing the offending vmbrX that is on same range as vmbr0. Thanks for the input.

 

[udo]

Hi,
do you think it's an good idea to use normal traffic and san-traffic on the same bond?
Because you can't use differents MTUs.

Whats about:
eth0+1 -> bond0 -> vlans -> vmbr0-vmbrX
eth2+3 -> bond1 -> storage-access with mtu 9000 (no bridge, san-access for host only (or need your VM direct san-access?))

Udo

 

[kameleon]

Unfortunately our core switch does not support jumbo frames so the mtu 9000 is not a factor.

 

[udo]

Unfortunately our core switch does not support jumbo frames so the mtu 9000 is not a factor.

Oh,
bad luck...

 

[kameleon]

Tell me about it. Being a small government agency with no IT budget has its drawbacks.

 

[kameleon]

I did a little tweaking and got it to work, even with the SAN vlan ip enabled! I was unable to get the vmbr0 to do anything on vlan1 so I left it on vlan2 but took the vmbr2 out so there was only one reference to bond0.2 and all was fine in networking world. Thanks for the pointers. Now just to get the SAN to talk to the machines via iscsi.

 

[udo]

I did a little tweaking and got it to work, even with the SAN vlan ip enabled! I was unable to get the vmbr0 to do anything on vlan1 so I left it on vlan2 but took the vmbr2 out so there was only one reference to bond0.2 and all was fine in networking world. Thanks for the pointers. Now just to get the SAN to talk to the machines via iscsi.

Hi,
i hear about trouble with mixing vlan-tagging and untagged traffic - it's looks like, that's with vlan-tagging on the default-vlan 1 it's the same?! ( i use other vlans - vlan 1 is an dummy vlan at my configuration...).

Or use anybody successfull vlan 1 beside other vlans?

Udo