network if tx buffer issues with tagged qinq and untagged

[immo]

HI,

it took a while to get some idea about the issue..
Setup:
Proxmox 4.4
a VM "router" with 2 network ifs.
if1 - vmbr250
if2 - vmbr208
second vm "hostA" with single if
if1 - vmbr250v1001
third vm "hostB" with single if
if1 - vmbr250v1001

The router vm use internally vlans 1001 and 1002 and the second if to setup routing betwen them.

1. Issue
vm router doesnt see the hostA and HostB as long they are at the same cluster host
2. Issue
if the vm HostA and B migrated to a second cluster host proxmox still has the interfaces vmbr250v1001 and 2002 on the original cluster host existing. They are not cleaned up fully.
ip link set vmbr250v1002 down
ip link delete vmbr250v1002
ip link delete eth3.250.1002
cleans it up.
3. on the target host i do get tx buffer issues as soon I access the vm. It looks like some frame handling is not done correctly

How can I setup the networking in a way router and Host can exists on the same hosts?

 

[Alwin]

1. Issue
vm router doesnt see the hostA and HostB as long they are at the same cluster host

I don't understand.

2. Issue
if the vm HostA and B migrated to a second cluster host proxmox still has the interfaces vmbr250v1001 and 2002 on the original cluster host existing. They are not cleaned up fully.
ip link set vmbr250v1002 down
ip link delete vmbr250v1002
ip link delete eth3.250.1002
cleans it up.

In this setup, it is unknown if a VM wouldn't come back and connect to that interface. To automate your manual procedure, a lot of things have to be checked before and after, that needs unnecessary resources.

3. on the target host i do get tx buffer issues as soon I access the vm. It looks like some frame handling is not done correctly

Has maybe to do with current setup.

A simpler approach, in my opinion, would be to go with a vlan aware bridge, then you add your VLAN tag to the VM config and the router is the only one that needs to handle different VLANs on its interface. All PVE hosts need to have the same network setup, to migrate all of the VMs to the same host. If you need migration to different PVE hosts, then you also need to make sure that the VLANs are reachable from all PVE hosts.

 

[immo]

I don't understand.

Me too ;-)
Expectation would be that if a host which is connected to a vlan sub interface of an Ethernet if which is shared with a second vm would see all broadcast frames for this vlan.

if host A connected to vmbr250v1001 send s a untagged frame, the frame will be tagged with 1001 as soon as the frame exits the bridge. (vmbr250v1001 has two interfaces connected. One is the tap of the VM and one is eth3.250.1001)
Do to the fact that eth3.250.1001 is a logical part of eth3.250 the frame should be forwarded also to the bridge vmbr250 and therefore seen by the hosts connected to this vm.

May be this assumptions that the frame is forwarded to vmbr250 too is wrong cos the eth interface eth3.250.1001 is already a part of the "HW Interface". This could explain the behavior.

Your proposal to use a vlan aware bridge points to ovs which we tried to prevent but seems to be the only way.

 

[Alwin]

Your proposal to use a vlan aware bridge points to ovs which we tried to prevent but seems to be the only way.

The linux bridge has the ability to be vlan aware, set "bridge_vlan_aware yes" in /etc/network/inerfaces in the bridge section, see also man bridge.

 

[immo]

but I guess this will not change the behavior on the same host.
cos proxmox create the vmbr250v1001 automatically and add eth3.250 to this bridge as soon as I set the vlan in the networking part of the vm network add bridge part

 

[Alwin]

I am not quite sure, what you want to achieve, as I am guessing from your subject line,that you would like to do vxlan. So then you would need a vxlan bridge underneath the vmbr250. https://vincent.bernat.im/en/blog/2017-vxlan-linux

From your description it sound more like, you want be able to migrate your router VM independently to you other VMs and these VMs should use the router for network access. In that case, having a bridge only connecting the router and other VMs with a vlan aware bridge would already be sufficient.

 

[immo]

openvswitch seems to be the solution for issue 1 and 2

 

[immo]

It has nothing to do with vxlan. We just seperate a couple of testbeds containing real hw devices and vms over a company wide vlan network. We just got two vlans for our task. Therefore we use the outer vlan as VMAN and the inner one per testbed.

 

[Alwin]

Thread on the forum about 802.1ad: https://forum.proxmox.com/threads/q-in-q-on-pve-4.23531/
Here are some notes on the top of the commit: https://git.kernel.org/pub/scm/linu.../?id=8ad227ff89a7e6f05d07cd0acfd95ed3a24450ca

Maybe someone else has the equipment and expertise for such a setup. As far as I can see, support for 802.1ad is in the kernel, it should be possible to configure linux networking for it.

 

[immo]

Let me try this if the next node is available... just a few days far away