Q in Q on PVE 4

D

dgeist

Member
Feb 26, 2015
37
0
6
Hi, I'm trying to nest some VM vlans inside an existing 802.1q tag between my PVE hosts (basically Q-in-Q).

I've done this before with debian/redhat/etc. and have always used the "vconfig" utility in the network interface config block. Well, the vlan package is not included by default in the PVE installer and if I attempt to add it, there appears to be a conflict with pve-manager:

Code: 
root@psp6hypd01:~# apt-get install vlan
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following packages will be REMOVED:
  pve-manager
The following NEW packages will be installed:
  vlan
0 upgraded, 1 newly installed, 1 to remove and 0 not upgraded.
Need to get 0 B/36.5 kB of archives.
After this operation, 150 kB of additional disk space will be used.
Do you want to continue? [Y/n] n
Abort.

Is this a defect in package mgmt or some known legitimate conflict between the PVE package and the debian vlan package?

Thanks
Dan
 
spirit said:
vconfig is deprecated since years and should not be used anymore.

you can use ip command.

" ip link add link eth0 name eth0.100 type vlan id 100" for example
Click to expand...

Thanks for the info. I wasn't aware it had been deprecated.

Now, the way I understand it, the "normal" way of making interfaces for VMs which are logically carried on .1q tags is to identify a vlan tag via the UI on top of the bridge interface, so you have something that looks like this:
Code: 
---------------------------
auto bond0
iface bond0 inet manual
        slaves enp66s0f0 enp66s0f1
        mtu 9000
        bond_miimon 100
        bond_mode active-backup

auto vmbr1
iface vmbr1 inet manual
        bridge_ports bond0
        bridge_stp off
        bridge_fd 0
        up ifconfig vmbr1 up
------------------------------

root@psp6hypd01:~# brctl show | grep vmbr1
vmbr1		8000.a0369f49a218	no		bond0
vmbr1v1002		8000.a0369f49a218	no		bond0.1002
vmbr1v1003		8000.a0369f49a218	no		bond0.1003

Similarly, I have a definition for a Q-in-Q setup (i.e. tagged vlans inside the existing vlan 1000) like so:
Code: 
auto bond0.1000
iface bond0.1000 inet manual
        vlan-raw-device bond0
        up ip link add link bond0 name bond0.1000 type vlan id 1000

auto vmbr2
iface vmbr2 inet manual
        bridge_ports bond0.1000
        bridge_stp off
        bridge_fd 0
        up ifconfig vmbr2 up

if I try to use vmbr2 as the bridge and put vlan ID 2000 on it via the UI, I get the following error in the UI logs:
Code: 
Running as unit 100.scope.
no physical interface on bridge 'vmbr2'

The command line above does work which is good, but obviously I'd like to incorporate that into actual GUI options for a regular user of the system. Anyone know where I went wrong, either conceptually or syntactically?

Thanks.
Dan
 
I don't have QinQ hardware to test, but they are some news with proxmox 4.0, not yet documented.

We support now vlan aware bridge (802.1q).

Code: 
auto vmbr0
iface vmbr0 inet manual
        bridge_ports eth0
        bridge_stp off
        bridge_fd 0
        bridge-vids 2-4094
        bridge_vlan_aware yes

(so, vms with vlan tagged interfaces, will use directly vmbr0, without need of vmbr0v... trick)



Now for qinq, you can create a 802.1ad vlan interface with

ip link add link eth0 eth0.24 type vlan proto 802.1ad id 24

so, for example
Code: 
auto vmbr0
iface vmbr0 inet manual
        bridge_ports eth0.24
        bridge_stp off
        bridge_fd 0
        bridge-vids 2-4094
        bridge_vlan_aware yes

Should work.

Could be great if you can test it !




 
  • Like
Reactions: EuroDomenii
Hi, Spirit. I was able to try this configuration out and it doesn't seem to be working (or I'm doing something wrong):
Code: 
auto lo
iface lo inet loopback

auto bond0
iface bond0 inet manual
        slaves enp66s0f0 enp66s0f1
        mtu 9000
        bond_miimon 100
        bond_mode active-backup
        up ip link add link bond0 name bond0.1000 type vlan proto 802.1ad id 1000

auto vmbr0
iface vmbr0 inet static
        address  X.160.73.20
        netmask  255.255.255.0
        gateway  X.160.73.1
        bridge_ports eno1
        bridge_stp off
        bridge_fd 0

auto vmbr1
iface vmbr1 inet manual
        mtu 9000
        bridge_ports bond0
        bridge_stp off
        bridge_fd 0
        bridge-vids 2-4094
        bridge_vlan_aware yes

auto vmbr2
iface vmbr2 inet manual
        mtu 9000
        bridge_ports bond0.1000
        bridge_stp off
        bridge_fd 0
        bridge-vids 2-4094
        bridge_vlan_aware yes

I was able to see packets in/out on interfaces bond0 and bond0.1000 but none of the other sub-interfaces of each seem to be instantiated by the PVE vm initialization. Do I still need to declare a .1q tag for each of the sub-interfaces, like "up ip link add link bond0 name bond0.98 type vlan proto 802.1Q id 98" if I have a vlan on the switch passing tagged frames on vlan 98 of the fabric?

Dan
 
dgeist said:
I was able to see packets in/out on interfaces bond0 and bond0.1000 but none of the other sub-interfaces of each seem to be instantiated by the PVE vm initialization.
Click to expand...
mmm,i found a bug in our network script when then interface in the bridge is a tagged interface,
can you try

"bridge vlan add dev bond0 vid 2-4094"

This should allow vlans 802.1Q coming from bridge to pass through bond0 interface

Do I still need to declare a .1q tag for each of the sub-interfaces, like "up ip link add link bond0 name bond0.98 type vlan proto 802.1Q id 98" if I have a vlan on the switch passing tagged frames on vlan 98 of the fabric?
Click to expand...

No, if you configure vlans in vm configuration, they should be tagged (802.1q) on the bridge port.

"bridge vlan show -c" should display the list of bridge vlans/ports
 
Hmmm, where do I put this setting? In the network interfaces script? Do I run it on commandline? Before starting a VM or after?

I tried the "bridge vlan show -c" and got:
Code: 
root@psp6hypd01:~# bridge vlan show -c
Message truncated
port	vlan ids
eno1	 1 PVID Egress Untagged

Message truncated

The VMs continue to not start with any network connectivity. I can't even load one with access to vmbr0 (mgmt interface). Not sure what's going on. Can I provide some more debug information here?

Dan
 
dgeist said:
Hmmm, where do I put this setting? In the network interfaces script? Do I run it on commandline? Before starting a VM or after?

I tried the "bridge vlan show -c" and got:
Code: 
root@psp6hypd01:~# bridge vlan show -c
Message truncated
port    vlan ids
eno1     1 PVID Egress Untagged

Message truncated

The VMs continue to not start with any network connectivity. I can't even load one with access to vmbr0 (mgmt interface). Not sure what's going on. Can I provide some more debug information here?

Dan
Click to expand...

for the "Message truncated", they are a bug in iproute2 display currently when too much vlans are defined (2-4094), a patched version will be available soon.
you can reduce the number of vlans to test, bridge vlan show should display the vlans.


for the management interface, you can defined a tagged vlan mgmt interface like

Code: 
auto vmbr0.100
iface vmbr0.100 inet static
        address ...
        netmask ....

to defined a management ip in vlan 100.

I'm using 802.1Q like this wihtout any problem, on QinQ on top the bridge has not been tested.
 
Thanks for the explanation about the bug. Here's my current interface config. I'm not trying to get management over a tag at this point, I'm just trying to get any traffic passing to any vm and the mgt bridge is the only one that's been able to pass traffic. Can you tell me what I'm doing wrong here? There is still nothing getting passed to any vm.

Code: 
auto lo
iface lo inet loopback

auto bond0
iface bond0 inet manual
        slaves enp66s0f0 enp66s0f1
        mtu 9000
        bond_miimon 100
        bond_mode active-backup
        up ip link add link bond0 name bond0.1000 type vlan proto 802.1ad id 1000
        up bridge vlan add dev bond0 vid 1000-1004

auto vmbr0
iface vmbr0 inet static
        address  X.160.73.20
        netmask  255.255.255.0
        gateway  X.160.73.1
        bridge_ports eno1
        bridge_stp off
        bridge_fd 0

auto vmbr1
iface vmbr1 inet manual
        mtu 9000
        bridge_ports bond0
        bridge_stp off
        bridge_fd 0
        bridge-vids 2-4094
        bridge_vlan_aware yes
        up bridge vlan add dev bond0 vid 1000-1004


auto vmbr2
iface vmbr2 inet manual
        mtu 9000
        bridge_ports bond0.1000
        bridge_stp off
        bridge_fd 0
        bridge-vids 2-4094
        bridge_vlan_aware yes
        up bridge vlan add dev bond0.1000 vid 2000-2005

root@psp6hypd01:~# bridge vlan show -c
Message truncated
port    vlan ids
eno1     1 PVID Egress Untagged

Message truncated

root@psp6hypd01:~# brctl show
bridge name     bridge id               STP enabled     interfaces
vmbr0           8000.b82a72d45423       no              eno1
                                                        tap100i0
                                                        tap100i5
                                                        tap100i6
                                                        tap100i7
                                                        tap100i8
vmbr1           8000.a0369f49a218       no              bond0
                                                        tap100i1
                                                        tap100i2
                                                        tap100i4
                                                        tap100i9
vmbr2           8000.a0369f49a218       no              bond0.1000
                                                        tap100i3

root@psp6hypd01:~# cat /proc/net/vlan/config
VLAN Dev name    | VLAN ID
Name-Type: VLAN_NAME_TYPE_RAW_PLUS_VID_NO_PAD
bond0.1000     | 1000  | bond0

Thanks.
Dan
 
I found an error in the above that was instantiating too many vlan IDs. Once corrected, I get the following:

Code: 
root@psp6hypd01:~# bridge vlan show -c
port    vlan ids
eno1     1 PVID Egress Untagged

enp66s0f0        1 PVID Egress Untagged
         1000
         1001
         1002
         1003
         1004

enp66s0f0.1000   1 PVID Egress Untagged
         2000
         2001
         2002
         2003
         2004
         2005

vmbr0    1 PVID Egress Untagged

vmbr1    1 PVID Egress Untagged

vmbr2    1 PVID Egress Untagged

tap100i0         1 PVID Egress Untagged

tap100i1         1 Egress Untagged
         1002 PVID Egress Untagged

tap100i2         1 Egress Untagged
         1003 PVID Egress Untagged

tap100i3         1 Egress Untagged
         2000 PVID Egress Untagged

tap100i4         1 Egress Untagged
         1001 PVID Egress Untagged

tap100i5         1 PVID Egress Untagged

tap100i6         1 PVID Egress Untagged

tap100i7         1 PVID Egress Untagged

tap100i8         1 PVID Egress Untagged

tap100i9         1 Egress Untagged
         98 PVID Egress Untagged

I'm not sure the "Untagged" part is right... The frames should be coming into and leaving the hypervisor host tagged, but I want to have them appear "Untagged" to the guest VM. Still working on if I'm seeing actual traffic on the links...

Dan
 
Code: 
auto vmbr1
iface vmbr1 inet manual
        mtu 9000
        bridge_ports bond0
        bridge_stp off
        bridge_fd 0
        bridge-vids 1000-1004
        bridge_vlan_aware yes

should be enough to have traffic tagged with vlan (.1Q) 1000-1004, going out to the bond0 port.


(the up " up bridge vlan add dev bond0 vid 1000-1004", is not necessary in this case, because we correctly manage this, with the bridge-vids param).

But It must be done, if you use a tagged interface for bond. (It's a bug, I have sent a patch yesterday)

Code: 
auto vmbr1
iface vmbr1 inet manual
        mtu 9000
        bridge_ports bond0.10
        bridge_stp off
        bridge_fd 0
        bridge-vids 1000-1004
        bridge_vlan_aware yes
        up bridge vlan add dev bond0.10 vid 1000-1004




dgeist said:
I'm not sure the "Untagged" part is right... The frames should be coming into and leaving the hypervisor host tagged, but I want to have them appear "Untagged" to the guest VM. Still working on if I'm seeing actual traffic on the links...
Dan
Click to expand...

About the
"PVID Egress Untagged", that mean that the untagged traffic going out the interface, will be tagged with the vlan. (It's like a switchport access mode in cisco switch).
If your vm don't write tag in the os side,the packet will be tagged when going out from the vm to the bridge. (and untagged in the reverse way)
 
I've tried all manner of variants with this setup, but even if they show as being instantiated, none ever pass traffic to the upstream switched (or receive it), either with a standard tag or a multi-layered one. I've had to create vmbr interfaces for each logical vlan in which I want to have clients located, so to support a vm with interfaces in (switch) vlans 98 and 1000.2000, I used the following:
---------------------------------------------
auto bond0
iface bond0 inet manual
slaves enp66s0f0 enp66s0f1
mtu 9000
bond_miimon 100
bond_mode active-backup
up ip link add link bond0 name bond0.1000 type vlan id 1000

auto bond0.1000
iface bond0.1000 inet manual
mtu 9000
vlan-raw-device bond0
up ip link add link bond0.1000 name bond0.1000.2000 type vlan id 2000

auto bond0.1000.2000
iface bond0.1000.2000 inet manual
mtu 9000
vlan-raw-device bond0.1000

auto vmbr98
iface vmbr98 inet manual
bridge_ports bond0.98
bridge_stp off
bridge_fd 0
up ifconfig vmbr98 up

auto vmbr2000
iface vmbr2000 inet manual
bridge_ports bond0.1000.2000
bridge_stp off
bridge_fd 0
up bridge vlan add dev bond0.1000 vid 2000
up ifconfig vmbr2000 up
------------------------------------------------------------

Lots of other combinations were tried, but this was the only one that works.

Dan
 
dgeist said:
t), either with a standard tag
Click to expand...
Damn, that's strange it's work for me and for other peoples.
maybe it's a temporary kernel bug.

Anyway, thanks for sharing your config !

I could also try to improve the old way, when proxmox duplicate bridges with create upsream .

BTW, for Q-in-Q, I known they are different implementation, some vendor implement 802.1Q on top of 802.1Q,
other one 802.1Q on top of 802.1ad.

what is your switch vendor and implementation ? (as you don't specify 802.1ad nowhere)
 
Agreed that the behavior is strange (and possibly something wonky with a kernel patch gone awry, etc.). As for the switch fabric, the three PVE nodes (just one being used right now) have their diverse 10g connections through a pair of cisco nexus switches. I THINK the other "control" host (trusty) on which the pair member in the VSRX firewall cluster was running is connected via a Juniper MX switch which is then trunked to the Nexus pair.

I would guess they both support BOTH types of encapsulation. When I had "ad" configured for the outer portion, I could still see frames, but they were not correctly filtered down into the tagged sub-interfaces. I can check on the switch side with our networking folks to verify what state the interfaces and trunks on the switches are in presently. As I understand it, 802.1ad is more efficient if you know you are using it, but 802.1q is backwards (or forwards...) compatible, just a bit less efficient for the stack.

Any thoughts on how to dig into the virtio net compatibility issue I'm seeing?

Dan
 
I can make a patch for proxmox, if it's help, to manage this kind of bridge

auto vmbr98
iface vmbr98 inet manual
bridge_ports bond0.98
bridge_stp off
bridge_fd 0
Click to expand...


then if you define a vm with vlan 1000,
it will create automaticaly a

vmbr98v1000 with a bond0.98.1000 enslave.


(It's what we are doing historicaly, but we don't allow to use tagged interface in source bridge).


Could this help you ?
 
spirit said:
I can make a patch for proxmox, if it's help, to manage this kind of bridge




then if you define a vm with vlan 1000,
it will create automaticaly a

vmbr98v1000 with a bond0.98.1000 enslave.


(It's what we are doing historicaly, but we don't allow to use tagged interface in source bridge).


Could this help you ?
Click to expand...

That might make management a lot easier, yes. Currently, the interfaces just show as "unknown". I don't want to alter default behavior, or course, but adding flexibility might be nice.

Dan
 
dgeist said:
That might make management a lot easier, yes. Currently, the interfaces just show as "unknown". I don't want to alter default behavior, or course, but adding flexibility might be nice.

Dan
Click to expand...

I'll try to make some patch next week, if you have sometime to test them, it could be great :)
 
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom
Back