VLAN Weirdness

[bubbletrouble]

Hi guys,

I'm trying to integrate Proxmox with a Juniper switch. I have other ubuntu hosts set up in a similar fashion but the behavior under Proxmox is as of yet inexplicable to me.

I have defined eth0 as a trunk port with a single management vlan, however I see only sporadic traffic passing back and forth with tcpdump, nor can I hit any of the layer 3 IP addresses on that subnet. It's like its being black holed. The hardware is identical to my other linux hosts, and I've verified that the interface works in non trunked mode.

Here is my interfaces config
auto lo
iface lo inet loopback

iface eth0 inet manual

auto eth0.100
iface eth0.100 inet static
address 0.0.0.0
netmask 0.0.0.0

iface eth1 inet manual

iface vmbr0 inet manual
bridge_ports none
bridge_stp off
bridge_fd 0

auto vmbr1
iface vmbr1 inet static
address 172.16.1.2
netmask 255.255.255.0
bridge_ports eth1
bridge_stp off
bridge_fd 0

auto vmbr100
iface vmbr100 inet static
address 172.16.100.106
netmask 255.255.255.0
bridge_ports eth0.100
bridge_stp off
bridge_fd 0


------------------------------------------
The vlan shows up
VLAN Dev name | VLAN ID
Name-Type: VLAN_NAME_TYPE_RAW_PLUS_VID_NO_PAD
eth0.100 | 100 | eth0

and the bridge

bridge name bridge id STP enabled interfaces
vmbr1 8000.0030489e7dd7 no eth1
vmbr100 8000.0030489e7dd6 no eth0.100

but thats where my luck ends. There aren't any iptables rules, no port based authentication or layer two access controls that would stop traffic from working on this vlan. tcpdump -vvv -i eth0.100 occasionally returns L2 data like STP, and brctl showmacs vmbr100 shows l2 mac addresses of hosts on that mangement segment. However, the management ip address assigned to the bridge on that vlan cannot be pinged or otherwise contacted at L3.

I also tried upgrading to the newer kernel 2.6.35-1-pve. I am running the igb kernel module on an 82576 Intel ethernet card.

I am open to troubleshooting or other suggestions.


Thanks.

 

[udo]

...
I have defined eth0 as a trunk port with a single management vlan, however I see only sporadic traffic passing back and forth with tcpdump, nor can I hit any of the layer 3 IP addresses on that subnet. It's like its being black holed. The hardware is identical to my other linux hosts, and I've verified that the interface works in non trunked mode.
...

Hi,
you have defined a trunk on the switch-side and use only one NIC (eth0) on the host-side?
If you use trunking, you should have bondX on the host-side (don't know if it's work with vlan-tagging).

Your config vor vmbr100 should be right but why you don't have an ip on vmbr0? Proxmox use vmbr0 for cluster-communication and AFAIK if there no ip then eth0!! Perhaps this is the problem (mixed mode between default vlan and tagged vlans are not working).

Normaly you need only use vlan-tagging for vlan 100 on the switch-port-side and vmbr100 should work.

Udo

 

[bubbletrouble]

Hey,


I was hoping to use my management vlan 100 for proxmox cluster communication, that being said the reason vmbr0 doesn't have an ip address is that there is no default vlan tagging on the interface. The switch is 802.1q trunking 1 vlan, 'management' with vlan-id 100

I went back and brought up vmbr0 just to see if that had any effect, but it doesn't seem to matter. It's also important to note that even if I bring down the bridges and just assign an ip to eth0.100, it still doesn't work. The only thing that seems to work is non trunked. The host does have eth0 and eth1 interfaces though, both of which I would like to export a range of vlans.

I also went back and verified that the mac address of the nic was being seen by the switch.. looks ok.

I was wondering if there might be some obscure spanning tree issue.. it seems like a pretty straight forward thing though. Anyway, thanks, any more ideas?

 

[udo]

Hey,


I was hoping to use my management vlan 100 for proxmox cluster communication, that being said the reason vmbr0 doesn't have an ip address is that there is no default vlan tagging on the interface. The switch is 802.1q trunking 1 vlan, 'management' with vlan-id 100

I went back and brought up vmbr0 just to see if that had any effect, but it doesn't seem to matter. It's also important to note that even if I bring down the bridges and just assign an ip to eth0.100, it still doesn't work. The only thing that seems to work is non trunked. The host does have eth0 and eth1 interfaces though, both of which I would like to export a range of vlans.

I also went back and verified that the mac address of the nic was being seen by the switch.. looks ok.

I was wondering if there might be some obscure spanning tree issue.. it seems like a pretty straight forward thing though. Anyway, thanks, any more ideas?

Hi,
there must be something wrong on the switch-side!
Try this config (than you use vmbr0 for cluster-communication on eth0.100):

auto lo
iface lo inet loopback

auto eth0
iface eth0 inet static
        address  0.0.0.0
        netmask  0.0.0.0

auto eth0.100
iface eth0.100 inet static
    address 0.0.0.0
    netmask 0.0.0.0

iface eth1 inet manual

auto vmbr0
iface vmbr0 inet static
    address 172.16.100.106
    netmask 255.255.255.0
    bridge_ports eth0.100
    bridge_stp off
    bridge_fd 0

auto vmbr1
iface vmbr1 inet static
    address 172.16.1.2
    netmask 255.255.255.0
    bridge_ports eth1
    bridge_stp off
    bridge_fd 0

I don't know juniper but trunking with one port make no sense for me...
I would say:
Port N default vid 1
Port N 802.1q 100

Udo

 

[bubbletrouble]

Thanks for the help so far.


I've tried this config, but still no cigar. It definitely feels like it might be a switch problem, but I have the identical port config for tons of other non proxmox hosts and have no issues whatsoever. Is it fair to say that the network, bridge etc configuration is only being done by the default debian network scripts? Is there anything else going on? Later today I'll try multiple vlans and a default vlan on that interface and see if it changes anything. I am thinking of just wiping the host with a straight ubuntu install and seeing if the problem persists, if there is a bug in proxmox it would be great to isolate and fix it.

 

[udo]

Thanks for the help so far.


I've tried this config, but still no cigar. It definitely feels like it might be a switch problem, but I have the identical port config for tons of other non proxmox hosts and have no issues whatsoever. Is it fair to say that the network, bridge etc configuration is only being done by the default debian network scripts? Is there anything else going on? Later today I'll try multiple vlans and a default vlan on that interface and see if it changes anything. I am thinking of just wiping the host with a straight ubuntu install and seeing if the problem persists, if there is a bug in proxmox it would be great to isolate and fix it.

Hi,
I think it's not a bug in proxmox (pve use only standard debian tools). I use (pve) bridges with vlan tagging since over two years in production without any problems (unused default vlan, manny tagged vlans).

Udo

 

[bubbletrouble]

Udo,


I didn't mean to imply that there was definitely an issue with Proxmox, only that i wasn't to rule it out. Anyway, to that end I re imaged the machine as a standard Ubuntu server and as you expected the problem still persisted.

Based on your earlier comment about it being odd practice to export a single vlan over a trunk, I took another look at that and exporting multiple vlans over the trunk suddenly got everything working. I had never noticed this behavior on Linux until now, and am not entirely sure what the cause is. Regardless the issues are resolved. Thanks very much!