Yet another confused VLAN user.

[pashadee]

Hi guys,

I should start by saying that I have not had the pleasure of configuring vlans in the past, but recently am needing to, so I am hoping some of you good people with vlan experience can help me out a little. It will be greatly appreciated!

I am going to attach a PNG file with my current setup, and describe some of the things not mentioned in the diagram.

I have a 3 node Proxmox Cluster
On node 1 eth1 is plugged directly to the WAN and bridged to a pfsense VM.

pfsense vm has 3 interfaces (1-wan, 2-lan, 3-vlan10)

I am hoping that pfsense can server as my main router as well as provide vlan routing as I will be needing probably at least 5 different vlan groups.

As per some threads I have setup separate bridges on each node for each vlan (always using eth0 on each node). So I have vbridge10, vbridge20, vbridge30 on each proxmox node.

I setup pfsense, and a couple vms on node 1 for vlan10, exported proper bridge device to the machines, modified pfsense rules to allow the traffic and all is working wonderfully.

Only on node 1 though, if I launch a vm on node 2, attach vmbridge10 to it, do everything like on node 1, I can not access the pfsense router, or any vms on node 1.

I am guessing that it's possibly because of the switch? I am not sure... this is where I need some help.

The part I don't understand is if I will be needing say 5 vlan groups, and I indeed need the switch to tag a port for each vlan, I won't be able to achieve that as each of my proxmox hosts only has 2 nics (node 1 has only one available because of wan).

I am thinking there has to be away to achieve this virtually (not via physical switch).

My main network is 192.168.1.0/24 (lan interface on pfsense, also subnet that proxmox is on)
vlan10 is 10.10.0.0/24 (pfsense vlan10 interface)

Thanks in advance guys, much appreciated!

 

[udo]

Hi,
sometimes there are trouble with mixed mode - e.g. tagged and untagged traffic on one interface.
I use one interface with tagged only without trouble.

You can try to find with tcpdump the issue - if you do an "tcpdump -i eth0 host ip.of.pf.sense" on the proxmox node 1 and try to ping from node 2 you should see the packets.
If not, look with tcpdump on node 2.

Udo

 

[spirit]

Hi,
sometimes there are trouble with mixed mode - e.g. tagged and untagged traffic on one interface.
I use one interface with tagged only without trouble.

You can try to find with tcpdump the issue - if you do an "tcpdump -i eth0 host ip.of.pf.sense" on the proxmox node 1 and try to ping from node 2 you should see the packets.
If not, look with tcpdump on node 2.

Udo

Yes, mixing tagged and untagged vlans is broken currently on 2.6.32 kernel.

Seem to work on coming 3.10 kernel.

 

[mir]

Yes, mixing tagged and untagged vlans is broken currently on 2.6.32 kernel.

Seem to work on coming 3.10 kernel.

mixed vlan tagging works over a bond, at least it works for me

auto vmbr0
iface vmbr0 inet static
address 192.168.2.8
netmask 255.255.255.0
bridge_ports bond0
bridge_stp off
bridge_fd 0


auto vmbr10
iface vmbr10 inet manual
bridge_ports bond0.10
bridge_stp off
bridge_fd 0

As you can see I create the vlans directly on the bond.

 

[pashadee]

Hi Udo,

Thanks for your response. Because of my inexperience with vlan I might have misled you guy to think that I am tagging the vlan traffic, in fact I am not. After doing some reading on the forums, google, etc I learned that I can vlan my virtual infrastructure using bridges that bridge ethX.VLANID, so I liked this aproach and created bridges for each of my vlans, which I then assigned to the network devices of my VMs. So there is no tagged traffic, it shouldn't be mixed. So far I am still getting the same results as before, VMs on the same VLAN can ping each other as long as they are on the same Proxmox host. I used your tcpdump command suggest (thank you for that),

Proxmost Host 1 (where pfsense resides):
16:36:44.208990 ARP, Request who-has 10.10.0.1 tell 10.10.0.10, length 46

Promox Host 2 (where VM pinging resides):
16:36:44.203330 ARP, Request who-has 10.10.0.1 tell 10.10.0.10, length 28

So it appears that the ARP request does make it to the proper machine, but it doesn't seem to be reaching the VM (I watch pfsense firewall logs, and nothing is denied there, I have a rule there to allow anything destined from 10.10.0./24)

Sounds like it might be my bridge config that's messing up, I will post them in this reply maybe you notice something.

Host 1:
auto lo
iface lo inet loopback


iface eth0 inet manual


iface eth1 inet manual


auto vmbr0
iface vmbr0 inet static
address 192.168.1.230
netmask 255.255.255.0
gateway 192.168.1.254
bridge_ports eth0
bridge_stp off
bridge_fd 0


auto vmbr1
iface vmbr1 inet manual
bridge_ports eth1
bridge_stp off
bridge_fd 0


auto vmbr10
iface vmbr10 inet manual
bridge_ports eth0.10
bridge_stp off
bridge_fd 0


auto vmbr20
iface vmbr20 inet manual
bridge_ports eth0.20
bridge_stp off
bridge_fd 0


Host 2:
auto lo
iface lo inet loopback


iface eth0 inet manual


iface eth1 inet manual


auto vmbr0
iface vmbr0 inet static
address 192.168.1.231
netmask 255.255.255.0
gateway 192.168.1.254
bridge_ports eth0
bridge_stp off
bridge_fd 0


auto vmbr10
iface vmbr10 inet manual
bridge_ports eth0.10
bridge_stp off
bridge_fd 0


auto vmbr20
iface vmbr20 inet manual
bridge_ports eth0.20
bridge_stp off
bridge_fd 0


Thanks so much for taking the time!

Hi,
sometimes there are trouble with mixed mode - e.g. tagged and untagged traffic on one interface.
I use one interface with tagged only without trouble.

You can try to find with tcpdump the issue - if you do an "tcpdump -i eth0 host ip.of.pf.sense" on the proxmox node 1 and try to ping from node 2 you should see the packets.
If not, look with tcpdump on node 2.

Udo

 

[pashadee]

Hi mir,

Your bond0 is just a bond to eth0? or more than one eth devices?

mixed vlan tagging works over a bond, at least it works for me

auto vmbr0
iface vmbr0 inet static
address 192.168.2.8
netmask 255.255.255.0
bridge_ports bond0
bridge_stp off
bridge_fd 0


auto vmbr10
iface vmbr10 inet manual
bridge_ports bond0.10
bridge_stp off
bridge_fd 0

As you can see I create the vlans directly on the bond.

 

[mir]

Hi mir,

Your bond0 is just a bond to eth0? or more than one eth devices?

no, it's a bond over eth1 and eth2. eth0 is unused at the moment.

 

[pashadee]

I did a tcpdump on pfsense VM while pinging it from host 2 via

ping -I vmbr10 10.10.0.1

tcpdump was showing the ICMP echo request arriving on pfsense, it looks like the reply gets lost on the way back.

 

[pashadee]

Does it make sense for me to bond to eth0 only then? I'm only needing on device in my case, I don't need failover or load-balance in my case (even if i did, on host 1 I couldn't achive this as I only have 2 nics and 1 is for wan 1 for lan).

 

[udo]

... So there is no tagged traffic, it shouldn't be mixed...

auto vmbr0
iface vmbr0 inet static
address 192.168.1.230
netmask 255.255.255.0
gateway 192.168.1.254
bridge_ports eth0
bridge_stp off
bridge_fd 0

...
auto vmbr10
iface vmbr10 inet manual
bridge_ports eth0.10
bridge_stp off
bridge_fd 0

Hi,
but this is what i mean with mixed vlan (tagging or not tagged).
If you use also an vlan-interface for vmbr0 perhaps yout trouble is gone (of course must configured on the switch too).
Like this (don't know if it's work with vlan1 and your switch):

auto vmbr0
iface vmbr0 inet static
    address 192.168.1.230
    netmask 255.255.255.0
    gateway 192.168.1.254
    bridge_ports eth0.1
    bridge_stp off
    bridge_fd 0

auto vmbr10
iface vmbr10 inet manual
    bridge_ports eth0.10
    bridge_stp off
    bridge_fd 0

auto vmbr20
iface vmbr20 inet manual
    bridge_ports eth0.20
    bridge_stp off
    bridge_fd 0

Udo

 

[udo]

I did a tcpdump on pfsense VM while pinging it from host 2 via

ping -I vmbr10 10.10.0.1

tcpdump was showing the ICMP echo request arriving on pfsense, it looks like the reply gets lost on the way back.

Hi,
is the routing/netmask right on the pfsense?

Udo