[SOLVED] VM with VLAN tagged interface not working

D

dsorrent

New Member
Jul 3, 2023
3
1
3
I am trying to create a VM with a trunk port and create a tagged interfaces within the VM. I have a 3 node Proxmox cluster deployed. Each node has the same NIC configurations (with different IPs configured on vmbr0):

Code: 
auto lo
iface lo inet loopback

auto eno3
iface eno3 inet manual
        mtu 9000

auto eno4
iface eno4 inet manual
        mtu 9000

auto bond0
iface bond0 inet manual
        bond-slaves eno3 eno4
        bond-miimon 100
        bond-mode 802.3ad
        bond-xmit-hash-policy layer2+3
        mtu 9000

auto vmbr0
iface vmbr0 inet static
        address 172.16.128.50/16
        gateway 172.16.0.1
        bridge-ports bond0
        bridge-stp off
        bridge-fd 0
        bridge-vlan-aware yes
        bridge-vids 2-4094

auto vmbr30
iface vmbr30 inet manual
        bridge-ports vlan30
        bridge-stp off
        bridge-fd 0
        bridge-vlan-aware yes
        bridge-vids 2-4094

auto vlan30
iface vlan30 inet manual
        vlan-raw-device bond0

I am deploying 3 Virtual Machines, one to each node to test configuration. To ensure my VLAN is configured correctly on the switch side, I created vlan30 & vmbr30 and I created the following VMs:

Hypervisor50 (172.16.128.50) hosts Rocky Linux instance 192.168.30.30
Hypervisor60 (172.16.128.60) hosts Vyos instance 192.168.30.1
Hypervisor70 (172.16.128.70) hosts Rocky Linux instance 192.168.30.20

When all 3 instances are connected to vmbr30, I can ping fine between all of them. Everything works as expected.

I then re-configured the 2 Linux instances to use vmbr0 and rebooted them with the intent to tag the VLAN within the VM.
I left the Vyos instance directly plumbed to vmbr30.

When I create a tagged interface for vlan 30 in the Linux VMs, I can't ping any instance from any other instance. This is an attempt to ping from 192.168.30.20 (Hypervisor70) to 192.168.30.1 (Hypervisor60):

1688409889065.png

I do not see ICMP packets at all anywhere in any tcpdumps on the hypervisors. What I am seeing is:

tcpdump on the bond on all 3 hypervisors:
tcpdump -i bond0 -nn -e vlan 30

Hypervisor50:
Code: 
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on bond0, link-type EN10MB (Ethernet), snapshot length 262144 bytes
15:13:22.379433 e6:f2:62:88:90:2c > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 64: vlan 30, p 0, ethertype ARP (0x0806), Request who-has 192.168.30.1 tell 192.168.30.20, length 46
15:13:23.437063 e6:f2:62:88:90:2c > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 64: vlan 30, p 0, ethertype ARP (0x0806), Request who-has 192.168.30.1 tell 192.168.30.20, length 46
15:13:24.461036 e6:f2:62:88:90:2c > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 64: vlan 30, p 0, ethertype ARP (0x0806), Request who-has 192.168.30.1 tell 192.168.30.20, length 46
15:13:25.485370 e6:f2:62:88:90:2c > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 64: vlan 30, p 0, ethertype ARP (0x0806), Request who-has 192.168.30.1 tell 192.168.30.20, length 46
15:13:26.509062 e6:f2:62:88:90:2c > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 64: vlan 30, p 0, ethertype ARP (0x0806), Request who-has 192.168.30.1 tell 192.168.30.20, length 46
15:13:27.533174 e6:f2:62:88:90:2c > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 64: vlan 30, p 0, ethertype ARP (0x0806), Request who-has 192.168.30.1 tell 192.168.30.20, length 46
^C
6 packets captured
6 packets received by filter
0 packets dropped by kernel

Hypervisor60:
Code: 
listening on bond0, link-type EN10MB (Ethernet), snapshot length 262144 bytes
15:13:22.387542 e6:f2:62:88:90:2c > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 64: vlan 30, p 0, ethertype ARP (0x0806), Request who-has 192.168.30.1 tell 192.168.30.20, length 46
15:13:22.387714 1a:eb:58:ad:e0:95 > e6:f2:62:88:90:2c, ethertype 802.1Q (0x8100), length 46: vlan 30, p 0, ethertype ARP (0x0806), Reply 192.168.30.1 is-at 1a:eb:58:ad:e0:95, length 28
15:13:23.445178 e6:f2:62:88:90:2c > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 64: vlan 30, p 0, ethertype ARP (0x0806), Request who-has 192.168.30.1 tell 192.168.30.20, length 46
15:13:23.445298 1a:eb:58:ad:e0:95 > e6:f2:62:88:90:2c, ethertype 802.1Q (0x8100), length 46: vlan 30, p 0, ethertype ARP (0x0806), Reply 192.168.30.1 is-at 1a:eb:58:ad:e0:95, length 28
15:13:24.469134 e6:f2:62:88:90:2c > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 64: vlan 30, p 0, ethertype ARP (0x0806), Request who-has 192.168.30.1 tell 192.168.30.20, length 46
15:13:24.469313 1a:eb:58:ad:e0:95 > e6:f2:62:88:90:2c, ethertype 802.1Q (0x8100), length 46: vlan 30, p 0, ethertype ARP (0x0806), Reply 192.168.30.1 is-at 1a:eb:58:ad:e0:95, length 28
15:13:25.493462 e6:f2:62:88:90:2c > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 64: vlan 30, p 0, ethertype ARP (0x0806), Request who-has 192.168.30.1 tell 192.168.30.20, length 46
15:13:25.493616 1a:eb:58:ad:e0:95 > e6:f2:62:88:90:2c, ethertype 802.1Q (0x8100), length 46: vlan 30, p 0, ethertype ARP (0x0806), Reply 192.168.30.1 is-at 1a:eb:58:ad:e0:95, length 28
15:13:26.517170 e6:f2:62:88:90:2c > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 64: vlan 30, p 0, ethertype ARP (0x0806), Request who-has 192.168.30.1 tell 192.168.30.20, length 46
15:13:26.517397 1a:eb:58:ad:e0:95 > e6:f2:62:88:90:2c, ethertype 802.1Q (0x8100), length 46: vlan 30, p 0, ethertype ARP (0x0806), Reply 192.168.30.1 is-at 1a:eb:58:ad:e0:95, length 28
15:13:27.541283 e6:f2:62:88:90:2c > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 64: vlan 30, p 0, ethertype ARP (0x0806), Request who-has 192.168.30.1 tell 192.168.30.20, length 46
15:13:27.541605 1a:eb:58:ad:e0:95 > e6:f2:62:88:90:2c, ethertype 802.1Q (0x8100), length 46: vlan 30, p 0, ethertype ARP (0x0806), Reply 192.168.30.1 is-at 1a:eb:58:ad:e0:95, length 28
^C
12 packets captured
12 packets received by filter
0 packets dropped by kernel

Hypervisor70:
Code: 
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on bond0, link-type EN10MB (Ethernet), snapshot length 262144 bytes
15:13:22.382357 e6:f2:62:88:90:2c > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 46: vlan 30, p 0, ethertype ARP (0x0806), Request who-has 192.168.30.1 tell 192.168.30.20, length 28
15:13:22.382580 1a:eb:58:ad:e0:95 > e6:f2:62:88:90:2c, ethertype 802.1Q (0x8100), length 64: vlan 30, p 0, ethertype ARP (0x0806), Reply 192.168.30.1 is-at 1a:eb:58:ad:e0:95, length 46
15:13:23.439895 e6:f2:62:88:90:2c > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 46: vlan 30, p 0, ethertype ARP (0x0806), Request who-has 192.168.30.1 tell 192.168.30.20, length 28
15:13:23.440160 1a:eb:58:ad:e0:95 > e6:f2:62:88:90:2c, ethertype 802.1Q (0x8100), length 64: vlan 30, p 0, ethertype ARP (0x0806), Reply 192.168.30.1 is-at 1a:eb:58:ad:e0:95, length 46
15:13:24.463872 e6:f2:62:88:90:2c > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 46: vlan 30, p 0, ethertype ARP (0x0806), Request who-has 192.168.30.1 tell 192.168.30.20, length 28
15:13:24.464177 1a:eb:58:ad:e0:95 > e6:f2:62:88:90:2c, ethertype 802.1Q (0x8100), length 64: vlan 30, p 0, ethertype ARP (0x0806), Reply 192.168.30.1 is-at 1a:eb:58:ad:e0:95, length 46
15:13:25.488206 e6:f2:62:88:90:2c > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 46: vlan 30, p 0, ethertype ARP (0x0806), Request who-has 192.168.30.1 tell 192.168.30.20, length 28
15:13:25.488502 1a:eb:58:ad:e0:95 > e6:f2:62:88:90:2c, ethertype 802.1Q (0x8100), length 64: vlan 30, p 0, ethertype ARP (0x0806), Reply 192.168.30.1 is-at 1a:eb:58:ad:e0:95, length 46
15:13:26.511896 e6:f2:62:88:90:2c > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 46: vlan 30, p 0, ethertype ARP (0x0806), Request who-has 192.168.30.1 tell 192.168.30.20, length 28
15:13:26.512316 1a:eb:58:ad:e0:95 > e6:f2:62:88:90:2c, ethertype 802.1Q (0x8100), length 64: vlan 30, p 0, ethertype ARP (0x0806), Reply 192.168.30.1 is-at 1a:eb:58:ad:e0:95, length 46
15:13:27.536006 e6:f2:62:88:90:2c > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 46: vlan 30, p 0, ethertype ARP (0x0806), Request who-has 192.168.30.1 tell 192.168.30.20, length 28
15:13:27.536491 1a:eb:58:ad:e0:95 > e6:f2:62:88:90:2c, ethertype 802.1Q (0x8100), length 64: vlan 30, p 0, ethertype ARP (0x0806), Reply 192.168.30.1 is-at 1a:eb:58:ad:e0:95, length 46
^C
12 packets captured
12 packets received by filter
0 packets dropped by kernel

tcpdump on the bridge on all 3 hypervisors

Code: 
Hypervisor50 & Hypervisor70 (hosting Linux VMs & trying to pass trunk):
tcpdump -i vmbr0 -nn -e vlan 30

Hypervisor60 (Hosting Vyos VM and bridging a VLAN tagged interface on the hypervisor):
tcpdump -i vmbr30 -nn -e

Hypervisor50:

Nothing

Hypervisor60:

Code: 
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on vmbr30, link-type EN10MB (Ethernet), snapshot length 262144 bytes
15:18:41.830904 e6:f2:62:88:90:2c > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: Request who-has 192.168.30.1 tell 192.168.30.20, length 46
15:18:41.831143 1a:eb:58:ad:e0:95 > e6:f2:62:88:90:2c, ethertype ARP (0x0806), length 42: Reply 192.168.30.1 is-at 1a:eb:58:ad:e0:95, length 28
15:18:42.869372 e6:f2:62:88:90:2c > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: Request who-has 192.168.30.1 tell 192.168.30.20, length 46
15:18:42.869586 1a:eb:58:ad:e0:95 > e6:f2:62:88:90:2c, ethertype ARP (0x0806), length 42: Reply 192.168.30.1 is-at 1a:eb:58:ad:e0:95, length 28
15:18:43.893525 e6:f2:62:88:90:2c > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: Request who-has 192.168.30.1 tell 192.168.30.20, length 46
15:18:43.893741 1a:eb:58:ad:e0:95 > e6:f2:62:88:90:2c, ethertype ARP (0x0806), length 42: Reply 192.168.30.1 is-at 1a:eb:58:ad:e0:95, length 28
15:18:44.917466 e6:f2:62:88:90:2c > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: Request who-has 192.168.30.1 tell 192.168.30.20, length 46
15:18:44.917670 1a:eb:58:ad:e0:95 > e6:f2:62:88:90:2c, ethertype ARP (0x0806), length 42: Reply 192.168.30.1 is-at 1a:eb:58:ad:e0:95, length 28
15:18:45.941512 e6:f2:62:88:90:2c > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: Request who-has 192.168.30.1 tell 192.168.30.20, length 46
15:18:45.941677 1a:eb:58:ad:e0:95 > e6:f2:62:88:90:2c, ethertype ARP (0x0806), length 42: Reply 192.168.30.1 is-at 1a:eb:58:ad:e0:95, length 28
15:18:46.965445 e6:f2:62:88:90:2c > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: Request who-has 192.168.30.1 tell 192.168.30.20, length 46
15:18:46.965696 1a:eb:58:ad:e0:95 > e6:f2:62:88:90:2c, ethertype ARP (0x0806), length 42: Reply 192.168.30.1 is-at 1a:eb:58:ad:e0:95, length 28
^C
12 packets captured
12 packets received by filter
0 packets dropped by kernel

Hypervisor70:

Code: 
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on vmbr0, link-type EN10MB (Ethernet), snapshot length 262144 bytes
15:18:21.648543 e6:f2:62:88:90:2c > 33:33:00:00:00:02, ethertype 802.1Q (0x8100), length 66: vlan 30, p 0, ethertype IPv6 (0x86dd), fe80::5399:8f70:b17b:5292 > ff02::2: ICMP6, router solicitation, length 8
15:18:41.825616 e6:f2:62:88:90:2c > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 46: vlan 30, p 0, ethertype ARP (0x0806), Request who-has 192.168.30.1 tell 192.168.30.20, length 28
15:18:42.864086 e6:f2:62:88:90:2c > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 46: vlan 30, p 0, ethertype ARP (0x0806), Request who-has 192.168.30.1 tell 192.168.30.20, length 28
15:18:43.888252 e6:f2:62:88:90:2c > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 46: vlan 30, p 0, ethertype ARP (0x0806), Request who-has 192.168.30.1 tell 192.168.30.20, length 28
15:18:44.912184 e6:f2:62:88:90:2c > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 46: vlan 30, p 0, ethertype ARP (0x0806), Request who-has 192.168.30.1 tell 192.168.30.20, length 28
15:18:45.936221 e6:f2:62:88:90:2c > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 46: vlan 30, p 0, ethertype ARP (0x0806), Request who-has 192.168.30.1 tell 192.168.30.20, length 28
15:18:46.960162 e6:f2:62:88:90:2c > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 46: vlan 30, p 0, ethertype ARP (0x0806), Request who-has 192.168.30.1 tell 192.168.30.20, length 28
^C
7 packets captured
7 packets received by filter
0 packets dropped by kernel

I have been searching online for what I might have mis-configured but from everything I read, it sounds like this should work? Any help is greatly appreciated!
 
chris.23lo said:
can share the following?

# ip a | grep bond

# brctl show

cheers
Click to expand...

Sure, thanks!

Hypervisor50:

Code: 
# ip a | grep bond
2: eno3: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 9000 qdisc mq master bond0 state UP group default qlen 1000
3: eno4: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 9000 qdisc mq master bond0 state UP group default qlen 1000
4: bond0: <BROADCAST,MULTICAST,MASTER,UP,LOWER_UP> mtu 9000 qdisc noqueue master vmbr0 state UP group default qlen 1000
6: vlan30@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc noqueue master vmbr30 state UP group default qlen 1000

# brctl show
bridge name     bridge id               STP enabled     interfaces
fwbr103i0               8000.de82cbfa01f6       no              fwln103i0
                                                        tap103i0
vmbr0           8000.1e815fc8325e       no              bond0
                                                        fwpr103p0
vmbr30          8000.1e815fc8325e       no              vlan30
#

Hypervisor60:

Code: 
# ip a | grep bond
2: eno3: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 9000 qdisc mq master bond0 state UP group default qlen 1000
3: eno4: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 9000 qdisc mq master bond0 state UP group default qlen 1000
4: bond0: <BROADCAST,MULTICAST,MASTER,UP,LOWER_UP> mtu 9000 qdisc noqueue master vmbr0 state UP group default qlen 1000
6: vlan30@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc noqueue master vmbr30 state UP group default qlen 1000


# brctl show
bridge name     bridge id               STP enabled     interfaces
vmbr0           8000.76d5b1650fa5       no              bond0
                                                        tap104i0
vmbr30          8000.76d5b1650fa5       no              tap104i1
                                                        vlan30
#

Hypervisor70:

Code: 
# ip a | grep bond
2: eno3: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 9000 qdisc mq master bond0 state UP group default qlen 1000
3: eno4: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 9000 qdisc mq master bond0 state UP group default qlen 1000
4: bond0: <BROADCAST,MULTICAST,MASTER,UP,LOWER_UP> mtu 9000 qdisc noqueue master vmbr0 state UP group default qlen 1000
6: vlan30@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc noqueue master vmbr30 state UP group default qlen 1000


# brctl show
bridge name     bridge id               STP enabled     interfaces
vmbr0           8000.7a8646a74e22       no              bond0
                                                        tap105i0
vmbr30          8000.7a8646a74e22       no              vlan30
#
 
remove vlan aware from vmbr30.
don't mix tagging on both nic && vlan-aware bridge.


something like: (no need to create a vlan30 interface, you can directly use bond0.30)

Code: 
auto vmbr30
iface vmbr30 inet manual
        bridge-ports bond0.30
        bridge-stp off
        bridge-fd 0

also if you mix vmbr0 vlan-aware (taging on bridge), and another brigde like vmbr30 with tagging on the interface,

the vlan30 will never reach vmbr0, because bond0.30 is "catching" all traffic for vlan30.


The best way is to use only the vmbr0 bridge, and do all the tags at the vm nic level.
just keep
Code: 
auto vmbr0
iface vmbr0 inet static
        address 172.16.128.50/16
        gateway 172.16.0.1
        bridge-ports bond0
        bridge-stp off
        bridge-fd 0
        bridge-vlan-aware yes
        bridge-vids 2-4094

and configure all tags in the vm nic options.
 
Last edited:
Thanks, that makes sense. Since I only created vlan30 & vmbr30 to test the switch, I removed them completely and once I cleared some ARP caches, everything is now working as desired.

Thanks again!
 
  • Like
Reactions: spirit
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom