openvswitch bridge VLAN configuration

[Lionel H]

Hi everyone,

I'm trying to achieve the following scenario:

+-------+
|       |  192.168.1.1/24                                   +----------------------------------------------------+
|  CPE  +-------------------------+                         |                   +------------------------------+ |
|       |                         |                         |                   |             VM               | |
+-------+                         |                         |       VLAN101  +--+----+                         | |
                                                            |      +---------+ ens19 | 192.168.1.2             | |
+-------+                 +----------------+    Trunk port  |      |         +--+----+                         | |
|       |  192.168.2.1/24 |                +----------------+---+  | VLAN201    |                              | |
|  CPE  +-----------------+     SWITCH     |------------| vmbr1 +---------+  +--+----+                         | |
|       |                 |                +----------------+---+  |      +--+ ens20 | 192.168.2.2             | |
+-------+                 +-------+--------+                |      |         +--+----+                         | |
                                  |                         |      |            |                              | |
+-------+                         |                         |      | VLAN301 +--+----+                         | |
|       |  192.168.3.1/24         |                         |      +---------+ ens21 | 192.168.3.2             | |
|  CPE  +-------------------------+                         |                +-------+                         | |
|       |                                                   |                   |------------------------------+ |
+-------+                                                   +----------------------------------------------------+

I need within the same VM to interact with differents CPEs. On the VM, each network interface is putted in a specific network namespace, I didn't mention on the picture to keep it simple.

The connection is working but not as expected because I see some packet loss.
When doing a tcpdump on vmbr1 while pinging 8.8.8.8 I see the reply coming back but never see the request:

tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on vmbr1, link-type EN10MB (Ethernet), capture size 262144 bytes

11:11:26.561793 c8:cd:72:99:99:93 > 02:ce:55:77:46:01, ethertype IPv4 (0x0800), length 100: 8.8.8.8 > 192.168.2.57: ICMP echo reply, id 1106, seq 1, length 64
11:11:27.581802 c8:cd:72:99:99:93 > 02:ce:55:77:46:01, ethertype IPv4 (0x0800), length 100: 8.8.8.8 > 192.168.2.57: ICMP echo reply, id 1106, seq 2, length 64
11:11:28.606044 c8:cd:72:99:99:93 > 02:ce:55:77:46:01, ethertype IPv4 (0x0800), length 100: 8.8.8.8 > 192.168.2.57: ICMP echo reply, id 1106, seq 3, length 64
11:11:29.629867 c8:cd:72:99:99:93 > 02:ce:55:77:46:01, ethertype IPv4 (0x0800), length 100: 8.8.8.8 > 192.168.2.57: ICMP echo reply, id 1106, seq 4, length 64
11:11:30.653591 c8:cd:72:99:99:93 > 02:ce:55:77:46:01, ethertype IPv4 (0x0800), length 100: 8.8.8.8 > 192.168.2.57: ICMP echo reply, id 1106, seq 5, length 64

Here are the configurations:

7: vmbr1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000
    link/ether 00:24:9b:15:06:02 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::224:9bff:fe15:602/64 scope link 
       valid_lft forever preferred_lft forever

^ no ip because the ip will be handled by the differents interfaces in the VM.

Openswitch configuration:

root@pve-master:~# ovs-vsctl show
c5f561b0-b3d3-409a-9531-db590aa8480c
    Bridge "vmbr1"
        Port "vmbr1"
            Interface "vmbr1"
                type: internal
        Port "enx00249b150602"
            Interface "enx00249b150602"
        Port "tap100i1"
            tag: 101
            Interface "tap100i1"
    ovs_version: "2.7.0"

enx00249b150602 is my second network card connected to the trunk port in the switch.

I've setup the interface using the GUI from proxmox which gives me the following config for my network interfaces:

net1: virtio=02:CE:55:77:46:01,bridge=vmbr1,queues=1,tag=101

In the VM I see sometimes the reply back, sometimes it gets lost.

Do you have an idea about what causing this issue? Do I need to apply some ebtables magic to make it works?

 

[Richard]

tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on vmbr1, link-type EN10MB (Ethernet), capture size 262144 bytes

11:11:26.561793 c8:cd:72:99:99:93 > 02:ce:55:77:46:01, ethertype IPv4 (0x0800), length 100: 8.8.8.8 > 192.168.2.57: ICMP echo reply, id 1106, seq 1, length 64
11:11:27.581802 c8:cd:72:99:99:93 > 02:ce:55:77:46:01, ethertype IPv4 (0x0800), length 100: 8.8.8.8 > 192.168.2.57: ICMP echo reply, id 1106, seq 2, length 64
11:11:28.606044 c8:cd:72:99:99:93 > 02:ce:55:77:46:01, ethertype IPv4 (0x0800), length 100: 8.8.8.8 > 192.168.2.57: ICMP echo reply, id 1106, seq 3, length 64
11:11:29.629867 c8:cd:72:99:99:93 > 02:ce:55:77:46:01, ethertype IPv4 (0x0800), length 100: 8.8.8.8 > 192.168.2.57: ICMP echo reply, id 1106, seq 4, length 64
11:11:30.653591 c8:cd:72:99:99:93 > 02:ce:55:77:46:01, ethertype IPv4 (0x0800), length 100: 8.8.8.8 > 192.168.2.57: ICMP echo reply, id 1106, seq 5, length 64

Where/what is 192.168.2.57- wrong entry in a route table somewhere?

In the VM I see sometimes the reply back, sometimes it gets lost.

Post an example about such a packet (which has been tracked by tcpdump, if possible).

IP configuration is defined in file /etc/network/interface - how looks that file like?

 

[Lionel H]

Where/what is 192.168.2.57- wrong entry in a route table somewhere?

It's the IP assigned via DHCP on the interface bound to the VLAN in the guest.

Post an example about such a packet (which has been tracked by tcpdump, if possible).

On the bridge, now I see that the Arp reply does not seems to reach the interface in the guest VM... See attached pcap.
rename ping_vmbr1.txt to ping_vmbr1.pcap as the forum does not allow pcap extension..

IP configuration is defined in file /etc/network/interface - how looks that file like?

It's pretty simple, because I want that vmbr1 acting as a trunk and forward traffic to the correct interface in the VM:

auto lo
iface lo inet loopback

iface enp2s0 inet manual

allow-vmbr1 enx00249b150602
iface enx00249b150602 inet manual
    ovs_type OVSPort
    ovs_bridge vmbr1

auto vmbr0
iface vmbr0 inet static
    address  10.3.76.69
    netmask  255.255.255.0
    gateway  10.3.76.1
    bridge_ports enp2s0
    bridge_stp off
    bridge_fd 0

auto vmbr1
iface vmbr1 inet manual
    ovs_type OVSBridge
    ovs_ports enx00249b150602

enx00249b150602 here is a usb to ethernet dongle adapter. This is the port that is directly connected in the trunk port on the switch.

vmbr0 is the control bridge connected on the control network (not relevant in the above config).

EDIT:
When trying to acquire an ip address, the reply is sent but has not vlan tag:

08:15:47.717097 02:ce:55:77:46:01 > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 346: vlan 101, p 0, ethertype IPv4, 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, Request from 02:ce:55:77:46:01, length 300
08:15:47.717230 02:ce:55:77:46:01 > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 348: vlan 101, p 1, ethertype IPv4, 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, Request from 02:ce:55:77:46:01, length 300
08:15:47.721401 ac:64:62:6c:9c:4f > 02:ce:55:77:46:01, ethertype IPv4 (0x0800), length 592: 192.168.2.254.67 > 255.255.255.255.68: BOOTP/DHCP, Reply, length 548

Do you have an idea why ? Might be something dumb but I don't see what's going on there.

 

[Lionel H]

UP.

 

[Richard]

vmbr0 is the control bridge connected on the control network (not relevant in the above config).

I'm not quite sure about that. I have the impression that somewere ther are asymmetric routes. How look the namespaces (resprectively the global space) in the VM like regarding IP configuration?

EDIT:
When trying to acquire an ip address, the reply is sent but has not vlan tag:

08:15:47.717097 02:ce:55:77:46:01 > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 346: vlan 101, p 0, ethertype IPv4, 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, Request from 02:ce:55:77:46:01, length 300
08:15:47.717230 02:ce:55:77:46:01 > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 348: vlan 101, p 1, ethertype IPv4, 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, Request from 02:ce:55:77:46:01, length 300
08:15:47.721401 ac:64:62:6c:9c:4f > 02:ce:55:77:46:01, ethertype IPv4 (0x0800), length 592: 192.168.2.254.67 > 255.255.255.255.68: BOOTP/DHCP, Reply, length 548

Do you have an idea why ? Might be something dumb but I don't see what's going on there.

That happens typically when the NIC does not support VLAN tags.