VM sending out on two VLANs - one of which isn't assigned to VM

S

SlothCroissant

Active Member
Feb 26, 2019
15
0
41
34
I'm seeing a very weird behavior on one of my VMs where packets appear to be duplicated on the PVE host and sent out on multiple VLANs.

Expected traffic flow: VM (VLAN102 in PVE) > PVE Host Bridge (Vlan-aware) > Cisco switch (on VLAN102)
Observed traffic flow: DHCP broadcasts from the VM are being seen *twice* on the host, coming from VLAN 102 & VLAN 101. tcpdump shows that traffic seems to be leaving two VM NICs at once somehow.

Environment:

PVE host net config:

Code: 
root@pve01:~# cat /etc/network/interfaces
auto lo
iface lo inet loopback

auto bond0
iface bond0 inet manual
        bond-slaves ens7f0 ens7f1 ens8f0 ens8f1 ens6
        bond-primary ens6
        bond-mode active-backup
        bond-miimon 100

auto vmbr0
iface vmbr0 inet static
        address 10.1.10.31/24
        gateway 10.1.10.1
        bridge-ports bond0
        bridge-stp off
        bridge-fd 0
        bridge-vlan-aware yes
        bridge-vids 101 102 103 104 10 20 30

iface enx02e0ec3012d3 inet manual
iface ens8f0 inet manual
iface ens8f1 inet manual
iface ens7f0 inet manual
iface ens7f1 inet manual
iface ens6 inet manual

VM config (note that I manually set the NIC MAC addresses close for testing, it made no difference):
Code: 
root@pve01:~# cat /etc/pve/qemu-server/2002.conf
agent: 1
boot: order=scsi0
cores: 4
cpu: host
machine: q35
memory: 8192
meta: creation-qemu=6.1.0,ctime=1646798395
name: ubnt01-sec
net0: virtio=9A:A8:28:E9:07:E9,bridge=vmbr0,tag=102
net1: virtio=9A:A8:28:E9:07:E8,bridge=vmbr0,tag=104
numa: 0
onboot: 1
ostype: l26
scsi0: sdb:2002/vm-2002-disk-0.qcow2,size=16G
scsihw: virtio-scsi-pci
smbios1: uuid=917ff8b4-886c-4cda-b43b-877390015084
sockets: 4
tablet: 0
vmgenid: 7706953c-2eb1-40fc-b5fe-d3f02b0e1c6d
root@pve01:~#


When I take a tcpdump on eth0 (aka net0 from the PVE perspective), I see traffic as expected:

Code: 
ryanb@ubnt01-sec:~$ sudo tcpdump -vnei eth0 port 67
tcpdump: listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes
12:19:07.005998 9a:a8:28:e9:07:e9 > ff:ff:ff:ff:ff:ff, ethertype IPv4 (0x0800), length 342: (tos 0x10, ttl 128, id 0, offset 0, flags [none], proto UDP (17), length 328)
    0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, Request from 9a:a8:28:e9:07:e9, length 300, xid 0xe1cf282a, Flags [none]
          Client-Ethernet-Address 9a:a8:28:e9:07:e9
          Vendor-rfc1048 Extensions
            Magic Cookie 0x63825363
            DHCP-Message Option 53, length 1: Discover
            Hostname Option 12, length 10: "ubnt01-sec"
            Parameter-Request Option 55, length 7:
              Subnet-Mask, BR, Default-Gateway, Domain-Name-Server
              Classless-Static-Route, Domain-Name, MTU
12:19:09.907056 9a:a8:28:e9:07:e9 > ff:ff:ff:ff:ff:ff, ethertype IPv4 (0x0800), length 342: (tos 0x10, ttl 128, id 0, offset 0, flags [none], proto UDP (17), length 328)
    0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, Request from 9a:a8:28:e9:07:e9, length 300, xid 0xe1cf282a, secs 2, Flags [none]
          Client-Ethernet-Address 9a:a8:28:e9:07:e9
          Vendor-rfc1048 Extensions
            Magic Cookie 0x63825363
            DHCP-Message Option 53, length 1: Discover
            Hostname Option 12, length 10: "ubnt01-sec"
            Parameter-Request Option 55, length 7:
              Subnet-Mask, BR, Default-Gateway, Domain-Name-Server
              Classless-Static-Route, Domain-Name, MTU
12:19:14.379247 9a:a8:28:e9:07:e9 > ff:ff:ff:ff:ff:ff, ethertype IPv4 (0x0800), length 342: (tos 0x10, ttl 128, id 0, offset 0, flags [none], proto UDP (17), length 328)
    0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, Request from 9a:a8:28:e9:07:e9, length 300, xid 0xe1cf282a, secs 7, Flags [none]
          Client-Ethernet-Address 9a:a8:28:e9:07:e9
          Vendor-rfc1048 Extensions
            Magic Cookie 0x63825363
            DHCP-Message Option 53, length 1: Discover
            Hostname Option 12, length 10: "ubnt01-sec"
            Parameter-Request Option 55, length 7:
              Subnet-Mask, BR, Default-Gateway, Domain-Name-Server
              Classless-Static-Route, Domain-Name, MTU

However on the host (specifically on bond0, which should cover all traffic leaving the host), I see the packets doubled - ONLY difference being the VLAN ID:
Code: 
tcpdump: listening on bond0, link-type EN10MB (Ethernet), snapshot length 262144 bytes
12:20:39.471148 9a:a8:28:e9:07:e9 > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 346: vlan 102, p 0, ethertype IPv4 (0x0800), (tos 0x10, ttl 128, id 0, offset 0, flags [none], proto UDP (17), length 328)
    0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, Request from 9a:a8:28:e9:07:e9, length 300, xid 0x54cd95b, Flags [none]
          Client-Ethernet-Address 9a:a8:28:e9:07:e9
          Vendor-rfc1048 Extensions
            Magic Cookie 0x63825363
            DHCP-Message (53), length 1: Discover
            Hostname (12), length 10: "ubnt01-sec"
            Parameter-Request (55), length 7:
              Subnet-Mask (1), BR (28), Default-Gateway (3), Domain-Name-Server (6)
              Classless-Static-Route (121), Domain-Name (15), MTU (26)
12:20:39.471312 9a:a8:28:e9:07:e9 > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 346: vlan 101, p 0, ethertype IPv4 (0x0800), (tos 0x10, ttl 128, id 0, offset 0, flags [none], proto UDP (17), length 328)
    0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, Request from 9a:a8:28:e9:07:e9, length 300, xid 0x54cd95b, Flags [none]
          Client-Ethernet-Address 9a:a8:28:e9:07:e9
          Vendor-rfc1048 Extensions
            Magic Cookie 0x63825363
            DHCP-Message (53), length 1: Discover
            Hostname (12), length 10: "ubnt01-sec"
            Parameter-Request (55), length 7:
              Subnet-Mask (1), BR (28), Default-Gateway (3), Domain-Name-Server (6)
              Classless-Static-Route (121), Domain-Name (15), MTU (26)
12:20:42.930851 9a:a8:28:e9:07:e9 > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 346: vlan 102, p 0, ethertype IPv4 (0x0800), (tos 0x10, ttl 128, id 0, offset 0, flags [none], proto UDP (17), length 328)
    0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, Request from 9a:a8:28:e9:07:e9, length 300, xid 0x54cd95b, secs 3, Flags [none]
          Client-Ethernet-Address 9a:a8:28:e9:07:e9
          Vendor-rfc1048 Extensions
            Magic Cookie 0x63825363
            DHCP-Message (53), length 1: Discover
            Hostname (12), length 10: "ubnt01-sec"
            Parameter-Request (55), length 7:
              Subnet-Mask (1), BR (28), Default-Gateway (3), Domain-Name-Server (6)
              Classless-Static-Route (121), Domain-Name (15), MTU (26)
12:20:42.930996 9a:a8:28:e9:07:e9 > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 346: vlan 101, p 0, ethertype IPv4 (0x0800), (tos 0x10, ttl 128, id 0, offset 0, flags [none], proto UDP (17), length 328)
    0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, Request from 9a:a8:28:e9:07:e9, length 300, xid 0x54cd95b, secs 3, Flags [none]
          Client-Ethernet-Address 9a:a8:28:e9:07:e9
          Vendor-rfc1048 Extensions
            Magic Cookie 0x63825363
            DHCP-Message (53), length 1: Discover
            Hostname (12), length 10: "ubnt01-sec"
            Parameter-Request (55), length 7:
              Subnet-Mask (1), BR (28), Default-Gateway (3), Domain-Name-Server (6)
              Classless-Static-Route (121), Domain-Name (15), MTU (26)
12:20:46.917851 9a:a8:28:e9:07:e9 > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 346: vlan 102, p 0, ethertype IPv4 (0x0800), (tos 0x10, ttl 128, id 0, offset 0, flags [none], proto UDP (17), length 328)
    0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, Request from 9a:a8:28:e9:07:e9, length 300, xid 0x54cd95b, secs 7, Flags [none]
          Client-Ethernet-Address 9a:a8:28:e9:07:e9
          Vendor-rfc1048 Extensions
            Magic Cookie 0x63825363
            DHCP-Message (53), length 1: Discover
            Hostname (12), length 10: "ubnt01-sec"
            Parameter-Request (55), length 7:
              Subnet-Mask (1), BR (28), Default-Gateway (3), Domain-Name-Server (6)
              Classless-Static-Route (121), Domain-Name (15), MTU (26)
12:20:46.917986 9a:a8:28:e9:07:e9 > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 346: vlan 101, p 0, ethertype IPv4 (0x0800), (tos 0x10, ttl 128, id 0, offset 0, flags [none], proto UDP (17), length 328)
    0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, Request from 9a:a8:28:e9:07:e9, length 300, xid 0x54cd95b, secs 7, Flags [none]
          Client-Ethernet-Address 9a:a8:28:e9:07:e9
          Vendor-rfc1048 Extensions
            Magic Cookie 0x63825363
            DHCP-Message (53), length 1: Discover
            Hostname (12), length 10: "ubnt01-sec"
            Parameter-Request (55), length 7:
              Subnet-Mask (1), BR (28), Default-Gateway (3), Domain-Name-Server (6)
              Classless-Static-Route (121), Domain-Name (15), MTU (26)

VLAN 101 is tied to another VM (id: 2001) on the host. If I take specific captures on each VM's tap interface, I can see that traffic is somehow leaving.... both VMs? Am I looking at this right?

Code: 
root@pve01:~# tcpdump -v -nei any port 67
tcpdump: data link type LINUX_SLL2
tcpdump: listening on any, link-type LINUX_SLL2 (Linux cooked v2), snapshot length 262144 bytes
12:22:59.873857 tap2002i0 B   ifindex 391 9a:a8:28:e9:07:e9 ethertype IPv4 (0x0800), length 348: (tos 0x10, ttl 128, id 0, offset 0, flags [none], proto UDP (17), length 328)
    0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, Request from 9a:a8:28:e9:07:e9, length 300, xid 0xf5ace076, secs 3, Flags [none]
          Client-Ethernet-Address 9a:a8:28:e9:07:e9
          Vendor-rfc1048 Extensions
            Magic Cookie 0x63825363
            DHCP-Message (53), length 1: Discover
            Hostname (12), length 10: "ubnt01-sec"
            Parameter-Request (55), length 7:
              Subnet-Mask (1), BR (28), Default-Gateway (3), Domain-Name-Server (6)
              Classless-Static-Route (121), Domain-Name (15), MTU (26)
12:22:59.874252 tap2001i0 Out ifindex 375 9a:a8:28:e9:07:e9 ethertype IPv4 (0x0800), length 348: (tos 0x10, ttl 128, id 0, offset 0, flags [none], proto UDP (17), length 328)
    0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, Request from 9a:a8:28:e9:07:e9, length 300, xid 0xf5ace076, secs 3, Flags [none]
          Client-Ethernet-Address 9a:a8:28:e9:07:e9
          Vendor-rfc1048 Extensions
            Magic Cookie 0x63825363
            DHCP-Message (53), length 1: Discover
            Hostname (12), length 10: "ubnt01-sec"
            Parameter-Request (55), length 7:
              Subnet-Mask (1), BR (28), Default-Gateway (3), Domain-Name-Server (6)
              Classless-Static-Route (121), Domain-Name (15), MTU (26)
12:23:01.114134 tap2002i0 B   ifindex 391 9a:a8:28:e9:07:e9 ethertype IPv4 (0x0800), length 348: (tos 0x10, ttl 128, id 0, offset 0, flags [none], proto UDP (17), length 328)
    0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, Request from 9a:a8:28:e9:07:e9, length 300, xid 0x60b2d204, Flags [none]
          Client-Ethernet-Address 9a:a8:28:e9:07:e9
          Vendor-rfc1048 Extensions
            Magic Cookie 0x63825363
            DHCP-Message (53), length 1: Discover
            Hostname (12), length 10: "ubnt01-sec"
            Parameter-Request (55), length 7:
              Subnet-Mask (1), BR (28), Default-Gateway (3), Domain-Name-Server (6)
              Classless-Static-Route (121), Domain-Name (15), MTU (26)
12:23:01.114481 tap2001i0 Out ifindex 375 9a:a8:28:e9:07:e9 ethertype IPv4 (0x0800), length 348: (tos 0x10, ttl 128, id 0, offset 0, flags [none], proto UDP (17), length 328)
    0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, Request from 9a:a8:28:e9:07:e9, length 300, xid 0x60b2d204, Flags [none]
          Client-Ethernet-Address 9a:a8:28:e9:07:e9
          Vendor-rfc1048 Extensions
            Magic Cookie 0x63825363
            DHCP-Message (53), length 1: Discover
            Hostname (12), length 10: "ubnt01-sec"
            Parameter-Request (55), length 7:
              Subnet-Mask (1), BR (28), Default-Gateway (3), Domain-Name-Server (6)

Here's vm2001's config, for ref:
Code: 
root@pve01:~# cat /etc/pve/qemu-server/2001.conf
agent: 1
boot: order=scsi0;net0
cores: 4
cpu: host
machine: q35
memory: 8192
meta: creation-qemu=6.1.0,ctime=1646798395
name: ubnt01-pri
net0: virtio=E6:DC:1F:70:F8:43,bridge=vmbr0,tag=101
net1: virtio=EE:AD:89:6A:BF:81,bridge=vmbr0,tag=103
numa: 0
onboot: 1
ostype: l26
scsi0: nvme1:2001/vm-2001-disk-0.qcow2,size=16G
scsihw: virtio-scsi-pci
smbios1: uuid=e45ef6e2-95fd-4cd0-a111-35fc0ce7572a
sockets: 4
tablet: 0
vmgenid: 9e14c0cd-fcaa-4379-94fd-a2e66a037cce

And tcpdump-ing on vm2001 (which is in an entirely different VLAN, and should never see vm2002's broadcast traffic), I see the traffic - only *once* however, so it's either inbound as it's receiving the broadcast somehow, or it's outbound and it is *somehow* originating the exact same packet at the exact same time.

Thoughts? I'm at a loss that either 1) vm2001 on VLAN101 is receiving vm2002's VLAN102 tagged traffic, or 2) vlan2001 is somehow sending the exact same traffic, causing dupes to show on the host.

Only thing to consider is that these were cloned VMs - I installed VyOS 1.3 on a base VM, then cloned it twice to create two separate VyOS routers for my environment. They should be logically separated by the VLANs. I tried removing and re-adding the NICs on vm2002 to no avail.

Let me know if I can provide any other detail.
 
Nudging this, as I haven't been able to use my secondary ISP router for several days now due to bad VLAN tagging behavior. Can anyone provide any insight into my configuration or ideas on what may be causing my woes?
 
Okay, i'll nudge this one more time before flat out giving up on VLANs in Proxmox :(

If nothing else, can someone tell me if they see any issues with my networking configuration?

```
root@pve01:~# cat /etc/network/interfaces
auto lo
iface lo inet loopback

auto bond0
iface bond0 inet manual
bond-slaves ens7f0 ens7f1 ens8f0 ens8f1 ens6
bond-primary ens6
bond-mode active-backup
bond-miimon 100

auto vmbr0
iface vmbr0 inet static
address 10.1.10.31/24
gateway 10.1.10.1
bridge-ports bond0
bridge-stp off
bridge-fd 0
bridge-vlan-aware yes
bridge-vids 101 102 103 104 10 20 30

iface enx02e0ec3012d3 inet manual
iface ens8f0 inet manual
iface ens8f1 inet manual
iface ens7f0 inet manual
iface ens7f1 inet manual
iface ens6 inet manual
```

Should I be making a bridge for every VLAN? Or is vlan-aware good enough? My goal is to make a L2 VLAN (aka the interface doesn't need a routed IP, just tag the frame and send it on its way)
 
Last edited:
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom