VLANs won't work

achekalin

Member
Jun 17, 2011
74
0
6
My setup is simple, a PC with 1 CPU (decent desktop PC with AMD Phenom II X4 and 2 Realtek NICs).

One NIC connects to backup host, and work perfectly, but the othe one point to network and should support VLANs.

I set up default VLAN (1) and two other VLANs to test the situation:

Code:
eth0.7    Link encap:Ethernet  HWaddr 6c:62:6d:e7:ff:42
          inet6 addr: fe80::6e62:6dff:fee7:ff42/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:27 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:0 (0.0 B)  TX bytes:1806 (1.7 KiB)

eth0.90   Link encap:Ethernet  HWaddr 6c:62:6d:e7:ff:42
          inet6 addr: fe80::6e62:6dff:fee7:ff42/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:24 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:0 (0.0 B)  TX bytes:1680 (1.6 KiB)

vmbr7     Link encap:Ethernet  HWaddr 6c:62:6d:e7:ff:42
          inet addr:10.90.91.248  Bcast:10.90.91.255  Mask:255.255.255.0
          inet6 addr: fe80::6e62:6dff:fee7:ff42/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:9 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:0 (0.0 B)  TX bytes:594 (594.0 B)

vmbr90    Link encap:Ethernet  HWaddr 6c:62:6d:e7:ff:42
          inet addr:10.90.90.248  Bcast:10.90.90.255  Mask:255.255.255.0
          inet6 addr: fe80::6e62:6dff:fee7:ff42/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:6 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:0 (0.0 B)  TX bytes:468 (468.0 B)

I check the config:

Code:
cat /proc/net/vlan/eth0.7
eth0.7  VID: 7   REORDER_HDR: 1  dev->priv_flags: 1
         total frames received            0
          total bytes received            0
      Broadcast/Multicast Rcvd            0

      total frames transmitted           31
       total bytes transmitted         2070
            total headroom inc            0
           total encap on xmit            0
Device: eth0
INGRESS priority mappings: 0:0  1:0  2:0  3:0  4:0  5:0  6:0 7:0
 EGRESS priority mappings:

looks good, the same for VLAN 90.

Switch that this NIC connected to can see its MACs, but I can not ping any hosts within its own VLANs from Proxmon host.

Strange indeed, as such NICs are used in our network, and both switch and NICs are looking pretty well working.

All config is done via web GUI, and the machine was rebooted several time afterwards.

What can I do to make VLANs on Proxmox work?

Thank you in advance for the advice, I really need to make it work (and I can't believe Proxmox can't do that).
 
Please can you port the network config: /etc/network/interfaces

surely:

Code:
# network interface settings
auto lo
iface lo inet loopback

iface eth0 inet manual

auto eth1
iface eth1 inet static
        address  10.255.255.1
        netmask  255.255.255.252

auto vmbr0
iface vmbr0 inet static
        address  192.168.161.58
        netmask  255.255.255.0
        gateway  192.168.161.1
        bridge_ports eth0
        bridge_stp off
        bridge_fd 0

auto vmbr7
iface vmbr7 inet static
        address  10.90.91.248
        netmask  255.255.255.0
        bridge_ports eth0.7
        bridge_stp off
        bridge_fd 0

auto vmbr90
iface vmbr90 inet static
        address  10.90.90.248
        netmask  255.255.255.0
        bridge_ports eth0.90
        bridge_stp off
        bridge_fd 0

All's done by Proxmon.
 
surely:

Code:
# network interface settings
auto lo
iface lo inet loopback

iface eth0 inet manual

auto eth1
iface eth1 inet static
        address  10.255.255.1
        netmask  255.255.255.252

auto vmbr0
iface vmbr0 inet static
        address  192.168.161.58
        netmask  255.255.255.0
        gateway  192.168.161.1
        bridge_ports eth0
        bridge_stp off
        bridge_fd 0

auto vmbr7
iface vmbr7 inet static
        address  10.90.91.248
        netmask  255.255.255.0
        bridge_ports eth0.7
        bridge_stp off
        bridge_fd 0

auto vmbr90
iface vmbr90 inet static
        address  10.90.90.248
        netmask  255.255.255.0
        bridge_ports eth0.90
        bridge_stp off
        bridge_fd 0

All's done by Proxmon.
Hi,
vlan work without trouble for me, but i don't use mixed mode (with and without vlantagging).
Can you try to use for vmbr0 also as vlan (like eth0.2)?

I have also definition of the vlans in the interface-file:
Code:
auto eth0
iface eth0 inet static
    address  0.0.0.0
    netmask  0.0.0.0

auto eth0.20
iface eth0.20 inet static
    address  0.0.0.0
    netmask  0.0.0.0

auto eth0.25
iface eth0.25 inet static
    address  0.0.0.0
    netmask  0.0.0.0

auto eth0.30
iface eth0.30 inet static
    address  0.0.0.0
    netmask  0.0.0.0

...

Udo
 
Last edited:
...i don't use mixed mode (with and without vlantagging).
Can you try to use for vmbr0 also as vlan (like eth0.2)?

Will try to, but I got no idea why VLANs won't work regular way (1 VLAN untagged and many tagged) as it works perfectly on any Linux I see.

Code:
auto eth0
iface eth0 inet static
    address  0.0.0.0
    netmask  0.0.0.0

auto eth0.20
iface eth0.20 inet static
    address  0.0.0.0
    netmask  0.0.0.0

...

What's strange, my such a file (/etc/network/interfaces) is different, and mine was made by Proxmon GUI, I didn't edit it myself. Your refers to eth0.X, while mine refers to vmbrX, and if yours is made by GUI also then I don't know how did you manage it to create such a config.
 
Will try to, but I got no idea why VLANs won't work regular way (1 VLAN untagged and many tagged) as it works perfectly on any Linux I see.



What's strange, my such a file (/etc/network/interfaces) is different, and mine was made by Proxmon GUI, I didn't edit it myself. Your refers to eth0.X, while mine refers to vmbrX, and if yours is made by GUI also then I don't know how did you manage it to create such a config.
Hi,
the vmbrX is the same - i list only the first part...
Code:
...
auto vmbr20
iface vmbr20 inet manual
        bridge_ports eth0.20
        bridge_stp off
        bridge_fd 0
...
Udo
 
I mean I don't have any ethX definitions in /etc/network/interfaces.

Mine simple contains

Code:
auto vmbr0
iface vmbr0 inet static

and yours contains first

Code:
auto eth0.20 
iface eth0.20 inet static
     address  0.0.0.0
     netmask  0.0.0.0

and then
Code:
auto vmbr20 
iface vmbr20 inet manual
         bridge_ports eth0.20
         bridge_stp off
         bridge_fd 0


I do believe GUI should produce the same config outline, but mine is different from yours :(


Upd: I've just changed my config so it look like yours (first I defined eth0.X and given IPs to it, and then defined vmbrX's and refereed to these eth0.X) - no luck, as soon as I given reboot command and system reboot itself the file become just like it was before (like mine version, not yours), and VLANs won't work either.
 
Last edited:
I mean I don't have any ethX definitions in /etc/network/interfaces.

Mine simple contains

Code:
auto vmbr0
iface vmbr0 inet static

and yours contains first

Code:
auto eth0.20 
iface eth0.20 inet static
     address  0.0.0.0
     netmask  0.0.0.0

and then
Code:
auto vmbr20 
iface vmbr20 inet manual
         bridge_ports eth0.20
         bridge_stp off
         bridge_fd 0


I do believe GUI should produce the same config outline, but mine is different from yours :(


Upd: I've just changed my config so it look like yours (first I defined eth0.X and given IPs to it, and then defined vmbrX's and refereed to these eth0.X) - no luck, as soon as I given reboot command and system reboot itself the file become just like it was before (like mine version, not yours), and VLANs won't work either.
Hi,
two things:
1. I made the changes directly in the interface-file without gui (this should not say that the gui don't work, but i modify the interface-file on debian-systems since a long time manually).

2. If you use the gui, a new file will be created and during boot this file was moved to the interface-file. I guess you have a new file due the gui-configuration which overwrite your handmade config.

Just edit the config one time more and reboot - then the edited version should be active.

Udo
 
Just done that. Here is the config (just in case)

Code:
# network interface settings
auto lo
iface lo inet loopback

#iface eth0 inet manual

auto eth1
iface eth1 inet static
        address  10.255.255.1
        netmask  255.255.255.252

auto eth0
iface eth0 inet static
        address  192.168.161.58
        netmask  255.255.255.0
        gateway  192.168.161.1

auto eth0.7
iface eth0.7 inet static
        address  10.90.91.248
        netmask  255.255.255.0

auto eth0.90
iface eth0.90 inet static
        address  10.90.90.248
        netmask  255.255.255.0


auto vmbr0
iface vmbr0 inet static
        bridge_ports eth0
        bridge_stp off
        bridge_fd 0

auto vmbr7
iface vmbr7 inet static
        bridge_ports eth0.7
        bridge_stp off
        bridge_fd 0

auto vmbr90
iface vmbr90 inet static
        bridge_ports eth0.90
        bridge_stp off
        bridge_fd 0

So what I have now is (ifconfig):

Code:
eth0      Link encap:Ethernet  HWaddr 6c:62:6d:e7:ff:42
          inet addr:192.168.161.58  Bcast:192.168.161.255  Mask:255.255.255.0
          inet6 addr: fe80::6e62:6dff:fee7:ff42/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:705 errors:0 dropped:0 overruns:0 frame:0
          TX packets:199 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:63843 (62.3 KiB)  TX bytes:30238 (29.5 KiB)
          Interrupt:43 Base address:0x6000

eth1      Link encap:Ethernet  HWaddr 5c:d9:98:b4:de:89
          inet addr:10.255.255.1  Bcast:10.255.255.3  Mask:255.255.255.252
          UP BROADCAST MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)
          Interrupt:20 Base address:0xec00

eth0.7    Link encap:Ethernet  HWaddr 6c:62:6d:e7:ff:42
          inet addr:10.90.91.248  Bcast:10.90.91.255  Mask:255.255.255.0
          inet6 addr: fe80::6e62:6dff:fee7:ff42/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:43 errors:0 dropped:0 overruns:0 frame:0
          TX packets:6 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:2114 (2.0 KiB)  TX bytes:468 (468.0 B)

eth0.90   Link encap:Ethernet  HWaddr 6c:62:6d:e7:ff:42
          inet addr:10.90.90.248  Bcast:10.90.90.255  Mask:255.255.255.0
          inet6 addr: fe80::6e62:6dff:fee7:ff42/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:21 errors:0 dropped:0 overruns:0 frame:0
          TX packets:6 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:982 (982.0 B)  TX bytes:468 (468.0 B)

lo        Link encap:Local Loopback
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:16436  Metric:1
          RX packets:108 errors:0 dropped:0 overruns:0 frame:0
          TX packets:108 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:22362 (21.8 KiB)  TX bytes:22362 (21.8 KiB)

(no vmbr's!)

and when I try to create VM the GUI offers only NAT-mode network.
 
Looks like I've made it work. If I'm right, the solution is simple:

put the same IP both on eth0.X and on vmbrX interface that uses that eth0.X as bridge device.

I'll try to check that once again now, but as far it sounds too fantastic for me.
 
Just done that. Here is the config (just in case)

Code:
# network interface settings
auto lo
iface lo inet loopback

#iface eth0 inet manual

auto eth1
iface eth1 inet static
        address  10.255.255.1
        netmask  255.255.255.252

auto eth0
iface eth0 inet static
        address  192.168.161.58
        netmask  255.255.255.0
        gateway  192.168.161.1

auto eth0.7
iface eth0.7 inet static
        address  10.90.91.248
        netmask  255.255.255.0

auto eth0.90
iface eth0.90 inet static
        address  10.90.90.248
        netmask  255.255.255.0


auto vmbr0
iface vmbr0 inet static
        bridge_ports eth0
        bridge_stp off
        bridge_fd 0

auto vmbr7
iface vmbr7 inet static
        bridge_ports eth0.7
        bridge_stp off
        bridge_fd 0

auto vmbr90
iface vmbr90 inet static
        bridge_ports eth0.90
        bridge_stp off
        bridge_fd 0

So what I have now is (ifconfig):

Code:
eth0      Link encap:Ethernet  HWaddr 6c:62:6d:e7:ff:42
          inet addr:192.168.161.58  Bcast:192.168.161.255  Mask:255.255.255.0
          inet6 addr: fe80::6e62:6dff:fee7:ff42/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:705 errors:0 dropped:0 overruns:0 frame:0
          TX packets:199 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:63843 (62.3 KiB)  TX bytes:30238 (29.5 KiB)
          Interrupt:43 Base address:0x6000

eth1      Link encap:Ethernet  HWaddr 5c:d9:98:b4:de:89
          inet addr:10.255.255.1  Bcast:10.255.255.3  Mask:255.255.255.252
          UP BROADCAST MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)
          Interrupt:20 Base address:0xec00

eth0.7    Link encap:Ethernet  HWaddr 6c:62:6d:e7:ff:42
          inet addr:10.90.91.248  Bcast:10.90.91.255  Mask:255.255.255.0
          inet6 addr: fe80::6e62:6dff:fee7:ff42/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:43 errors:0 dropped:0 overruns:0 frame:0
          TX packets:6 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:2114 (2.0 KiB)  TX bytes:468 (468.0 B)

eth0.90   Link encap:Ethernet  HWaddr 6c:62:6d:e7:ff:42
          inet addr:10.90.90.248  Bcast:10.90.90.255  Mask:255.255.255.0
          inet6 addr: fe80::6e62:6dff:fee7:ff42/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:21 errors:0 dropped:0 overruns:0 frame:0
          TX packets:6 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:982 (982.0 B)  TX bytes:468 (468.0 B)

lo        Link encap:Local Loopback
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:16436  Metric:1
          RX packets:108 errors:0 dropped:0 overruns:0 frame:0
          TX packets:108 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:22362 (21.8 KiB)  TX bytes:22362 (21.8 KiB)

(no vmbr's!)

and when I try to create VM the GUI offers only NAT-mode network.
Hi,
but your config has not much to do with my posted version.
Whats about something like this (you need in this case vlan2 for vmbr0):
Code:
# network interface settings
auto lo
iface lo inet loopback

auto eth1
iface eth1 inet static
        address  10.255.255.1
        netmask  255.255.255.252

auto eth0
iface eth0 inet static
        address  0.0.0.0
        netmask  0.0.0.0

auto eth0.2
iface eth0.2 inet static
        address  0.0.0.0
        netmask  0.0.0.0


auto eth0.7
iface eth0.7 inet static
        address  0.0.0.0
        netmask  0.0.0.0

auto eth0.90
iface eth0.90 inet static
        address  0.0.0.0
        netmask  0.0.0.0

auto vmbr0
iface vmbr0 inet static
        bridge_ports eth0.2
        address  192.168.161.58
        netmask  255.255.255.0
        gateway  192.168.161.1
        bridge_stp off
        bridge_fd 0

auto vmbr7
iface vmbr7 inet static
        address  10.90.91.248
        netmask  255.255.255.0
        bridge_ports eth0.7
        bridge_stp off
        bridge_fd 0

auto vmbr90
iface vmbr90 inet static
        address  10.90.90.248
        netmask  255.255.255.0
        bridge_ports eth0.90
        bridge_stp off
        bridge_fd 0
BTW: Why you use IPs on vmbr7 + 90? It's only nessesary if the pve-host must be accessible from the two networks and there is no route to the vmbr0-network...

Udo
 
That is, you put 0.0.0.0 on all eth's interfaces? I've seen that before but decided you've done that for security (for not showing yours IPs).

Yes, I've done as you showed me and it looks like it works fine!

Thank you, now will try to use it inside VMs!

(Frankly maybe it worth to include such manual to wiki? Would be really time-saver!)
 
BTW: Why you use IPs on vmbr7 + 90? It's only nessesary if the pve-host must be accessible from the two networks and there is no route to the vmbr0-network...

Do you mean I can not put IPs on bridges that goes to VLAN's at all? I don't need it unless from VMs, and VMs will assign their own IPs anyway...

Upd: Noop, just tested that. If I remove IPs from vmbr definition then such an vmbr can not be chosen in GUI when I create new VM. Appears like I need vmbr to carry IP?
 
Last edited:
Do you mean I can not put IPs on bridges that goes to VLAN's at all? I don't need it unless from VMs, and VMs will assign their own IPs anyway...

Upd: Noop, just tested that. If I remove IPs from vmbr definition then such an vmbr can not be chosen in GUI when I create new VM. Appears like I need vmbr to carry IP?
Hi,
only vmbr0 need an IP-Address. All other bridges don't have an ip on my setup:
Code:
auto vmbr90
iface vmbr90 inet manual
        bridge_ports eth0.90
        bridge_stp off
        bridge_fd 0

auto vmbr91
iface vmbr91 inet manual
        bridge_ports eth0.91
        bridge_stp off
        bridge_fd 0

auto vmbr92
iface vmbr92 inet manual
        bridge_ports eth0.92
        bridge_stp off
        bridge_fd 0

auto vmbr93
iface vmbr93 inet manual
        bridge_ports eth0.93
        bridge_stp off
        bridge_fd 0
Works well to assign the vmbrX to the VMs.

Udo
 
Ok, here is the result of my findings:

---
iface eth0 inet manual

auto eth0.20
iface eth0.20 inet static
..address 0.0.0.0
..netmask 0.0.0.0

auto eth0.21
iface eth0.21 inet static
..address 0.0.0.0
..netmask 0.0.0.0

# etc for each VLAN we'll use

# then we'll need to define VMBRs

auto vmbr0
iface vmbr0 inet static
..address a.b.c.d
..netmask 255.255.255.0
..gateway a.b.c.1
..bridge_ports eth0
..bridge_stp off
..bridge_fd 0

auto vmbr20
iface vmbr20 inet manual
..bridge_ports eth0.20
..bridge_stp off
..bridge_fd 0

auto vmbr21
iface vmbr21 inet manual
..bridge_ports eth0.21
..bridge_stp off
..bridge_fd 0

# etc
---

In other words:
1. We'll define eth0 as manual, and vmbr0 as static.
2. Add eth0.X for each VLAN we'll use, and all these eth0.X will be as static, and vmbrX on it will be as manual. All eth0.X should carry ip=0.0.0.0/0 on it, vmbrX should not carry any IP (except for vmbr0).
3. The control panel IP and gateway will be set on vmbr0.

Looks like quest solution, isn't it? What's bad, Proxmox GUI won't let me do that from web panel.

I use 2.6.35 kernel, but I believe it won't affect the receipt above.
 
The story won't show its end... Now I got a lot of eth's and vmbr's (that's what I wanted), and if I want to allow some VM to communicate to VLAN 7 then I add network device linked to vmbr 7 to it and it should work...

But no, it won't!

I've just create linux based VM, attached several different vmbr's to it (it appears as eth0, 1 ... inside the VM, and I use mac's to be sure which eth referrers to which VLAN), but the only outside IPs I can ping from inside is IPs on default vlan (vlan 1, untagged).

I feel myself like a child (before Proxmox i was sure I know a lot about VLANs and VLANs on *nix systems in particular), now I'm not sure in anything :(

BTW, external switch (that my Proxmox box connected to) can see each of my virtual NICs mac addresses, so I think we was able to create something close to working scheme. But the traffic anyway won't go...
 
The story won't show its end... Now I got a lot of eth's and vmbr's (that's what I wanted), and if I want to allow some VM to communicate to VLAN 7 then I add network device linked to vmbr 7 to it and it should work...

But no, it won't!

I've just create linux based VM, attached several different vmbr's to it (it appears as eth0, 1 ... inside the VM, and I use mac's to be sure which eth referrers to which VLAN), but the only outside IPs I can ping from inside is IPs on default vlan (vlan 1, untagged).

I feel myself like a child (before Proxmox i was sure I know a lot about VLANs and VLANs on *nix systems in particular), now I'm not sure in anything :(

BTW, external switch (that my Proxmox box connected to) can see each of my virtual NICs mac addresses, so I think we was able to create something close to working scheme. But the traffic anyway won't go...
Hi,
you can ping IPs on the default-Vlan? This looks like your switch don't tagg the VLANS ( eg. vlan7).
You can try to use tcpdump (on eth0) to find the issue. If you ping from outside on vlan7 you must see the tagging on the package.
Also if you ping from inside. I guess you see tagging from inside and none-tagging from outside.

Udo
 
Ok, I've tested that.

tcpdump says that pings are go well from outside in and from inside out.

I did ping from outside PC (10.90.90.1) to IP (10.90.90.196) that was assigned by one of VM to its internal eth NIC, and I see "arp who-has" packets from outside PC to given IP ("who-has 10.90.90.196"), but I see no reply packets. So ping won't go :(
 
Hello, my 2 cents.
i think i got the same problem last year see my post http://forum.proxmox.com/threads/5175-vlan-problems?highlight=

one week search and test on a nic : untagged vlan + taggued vlan never work all together, only one off the two thinks (tag or not tagg) can work at a time.

at the end i make this : all vlan tagged ( include vlan 1) ----> it's work. but i have a switch that support vlan 1 tagging.

you can see that is udo who save me last year, one more time "thank udo"

bye

GrdNico
 
Last edited: