I am trying to create a VM with a trunk port and create a tagged interfaces within the VM. I have a 3 node Proxmox cluster deployed. Each node has the same NIC configurations (with different IPs configured on vmbr0):
I am deploying 3 Virtual Machines, one to each node to test configuration. To ensure my VLAN is configured correctly on the switch side, I created vlan30 & vmbr30 and I created the following VMs:
Hypervisor50 (172.16.128.50) hosts Rocky Linux instance 192.168.30.30
Hypervisor60 (172.16.128.60) hosts Vyos instance 192.168.30.1
Hypervisor70 (172.16.128.70) hosts Rocky Linux instance 192.168.30.20
When all 3 instances are connected to vmbr30, I can ping fine between all of them. Everything works as expected.
I then re-configured the 2 Linux instances to use vmbr0 and rebooted them with the intent to tag the VLAN within the VM.
I left the Vyos instance directly plumbed to vmbr30.
When I create a tagged interface for vlan 30 in the Linux VMs, I can't ping any instance from any other instance. This is an attempt to ping from 192.168.30.20 (Hypervisor70) to 192.168.30.1 (Hypervisor60):
![1688409889065.png 1688409889065.png](https://forum.proxmox.com/data/attachments/52/52235-7c78ac885cd7bf7b0744225a2c94f2e1.jpg)
I do not see ICMP packets at all anywhere in any tcpdumps on the hypervisors. What I am seeing is:
tcpdump on the bond on all 3 hypervisors:
Hypervisor50:
Hypervisor60:
Hypervisor70:
tcpdump on the bridge on all 3 hypervisors
Hypervisor50:
Nothing
Hypervisor60:
Hypervisor70:
I have been searching online for what I might have mis-configured but from everything I read, it sounds like this should work? Any help is greatly appreciated!
Code:
auto lo
iface lo inet loopback
auto eno3
iface eno3 inet manual
mtu 9000
auto eno4
iface eno4 inet manual
mtu 9000
auto bond0
iface bond0 inet manual
bond-slaves eno3 eno4
bond-miimon 100
bond-mode 802.3ad
bond-xmit-hash-policy layer2+3
mtu 9000
auto vmbr0
iface vmbr0 inet static
address 172.16.128.50/16
gateway 172.16.0.1
bridge-ports bond0
bridge-stp off
bridge-fd 0
bridge-vlan-aware yes
bridge-vids 2-4094
auto vmbr30
iface vmbr30 inet manual
bridge-ports vlan30
bridge-stp off
bridge-fd 0
bridge-vlan-aware yes
bridge-vids 2-4094
auto vlan30
iface vlan30 inet manual
vlan-raw-device bond0
I am deploying 3 Virtual Machines, one to each node to test configuration. To ensure my VLAN is configured correctly on the switch side, I created vlan30 & vmbr30 and I created the following VMs:
Hypervisor50 (172.16.128.50) hosts Rocky Linux instance 192.168.30.30
Hypervisor60 (172.16.128.60) hosts Vyos instance 192.168.30.1
Hypervisor70 (172.16.128.70) hosts Rocky Linux instance 192.168.30.20
When all 3 instances are connected to vmbr30, I can ping fine between all of them. Everything works as expected.
I then re-configured the 2 Linux instances to use vmbr0 and rebooted them with the intent to tag the VLAN within the VM.
I left the Vyos instance directly plumbed to vmbr30.
When I create a tagged interface for vlan 30 in the Linux VMs, I can't ping any instance from any other instance. This is an attempt to ping from 192.168.30.20 (Hypervisor70) to 192.168.30.1 (Hypervisor60):
![1688409889065.png 1688409889065.png](https://forum.proxmox.com/data/attachments/52/52235-7c78ac885cd7bf7b0744225a2c94f2e1.jpg)
I do not see ICMP packets at all anywhere in any tcpdumps on the hypervisors. What I am seeing is:
tcpdump on the bond on all 3 hypervisors:
tcpdump -i bond0 -nn -e vlan 30
Hypervisor50:
Code:
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on bond0, link-type EN10MB (Ethernet), snapshot length 262144 bytes
15:13:22.379433 e6:f2:62:88:90:2c > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 64: vlan 30, p 0, ethertype ARP (0x0806), Request who-has 192.168.30.1 tell 192.168.30.20, length 46
15:13:23.437063 e6:f2:62:88:90:2c > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 64: vlan 30, p 0, ethertype ARP (0x0806), Request who-has 192.168.30.1 tell 192.168.30.20, length 46
15:13:24.461036 e6:f2:62:88:90:2c > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 64: vlan 30, p 0, ethertype ARP (0x0806), Request who-has 192.168.30.1 tell 192.168.30.20, length 46
15:13:25.485370 e6:f2:62:88:90:2c > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 64: vlan 30, p 0, ethertype ARP (0x0806), Request who-has 192.168.30.1 tell 192.168.30.20, length 46
15:13:26.509062 e6:f2:62:88:90:2c > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 64: vlan 30, p 0, ethertype ARP (0x0806), Request who-has 192.168.30.1 tell 192.168.30.20, length 46
15:13:27.533174 e6:f2:62:88:90:2c > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 64: vlan 30, p 0, ethertype ARP (0x0806), Request who-has 192.168.30.1 tell 192.168.30.20, length 46
^C
6 packets captured
6 packets received by filter
0 packets dropped by kernel
Hypervisor60:
Code:
listening on bond0, link-type EN10MB (Ethernet), snapshot length 262144 bytes
15:13:22.387542 e6:f2:62:88:90:2c > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 64: vlan 30, p 0, ethertype ARP (0x0806), Request who-has 192.168.30.1 tell 192.168.30.20, length 46
15:13:22.387714 1a:eb:58:ad:e0:95 > e6:f2:62:88:90:2c, ethertype 802.1Q (0x8100), length 46: vlan 30, p 0, ethertype ARP (0x0806), Reply 192.168.30.1 is-at 1a:eb:58:ad:e0:95, length 28
15:13:23.445178 e6:f2:62:88:90:2c > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 64: vlan 30, p 0, ethertype ARP (0x0806), Request who-has 192.168.30.1 tell 192.168.30.20, length 46
15:13:23.445298 1a:eb:58:ad:e0:95 > e6:f2:62:88:90:2c, ethertype 802.1Q (0x8100), length 46: vlan 30, p 0, ethertype ARP (0x0806), Reply 192.168.30.1 is-at 1a:eb:58:ad:e0:95, length 28
15:13:24.469134 e6:f2:62:88:90:2c > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 64: vlan 30, p 0, ethertype ARP (0x0806), Request who-has 192.168.30.1 tell 192.168.30.20, length 46
15:13:24.469313 1a:eb:58:ad:e0:95 > e6:f2:62:88:90:2c, ethertype 802.1Q (0x8100), length 46: vlan 30, p 0, ethertype ARP (0x0806), Reply 192.168.30.1 is-at 1a:eb:58:ad:e0:95, length 28
15:13:25.493462 e6:f2:62:88:90:2c > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 64: vlan 30, p 0, ethertype ARP (0x0806), Request who-has 192.168.30.1 tell 192.168.30.20, length 46
15:13:25.493616 1a:eb:58:ad:e0:95 > e6:f2:62:88:90:2c, ethertype 802.1Q (0x8100), length 46: vlan 30, p 0, ethertype ARP (0x0806), Reply 192.168.30.1 is-at 1a:eb:58:ad:e0:95, length 28
15:13:26.517170 e6:f2:62:88:90:2c > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 64: vlan 30, p 0, ethertype ARP (0x0806), Request who-has 192.168.30.1 tell 192.168.30.20, length 46
15:13:26.517397 1a:eb:58:ad:e0:95 > e6:f2:62:88:90:2c, ethertype 802.1Q (0x8100), length 46: vlan 30, p 0, ethertype ARP (0x0806), Reply 192.168.30.1 is-at 1a:eb:58:ad:e0:95, length 28
15:13:27.541283 e6:f2:62:88:90:2c > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 64: vlan 30, p 0, ethertype ARP (0x0806), Request who-has 192.168.30.1 tell 192.168.30.20, length 46
15:13:27.541605 1a:eb:58:ad:e0:95 > e6:f2:62:88:90:2c, ethertype 802.1Q (0x8100), length 46: vlan 30, p 0, ethertype ARP (0x0806), Reply 192.168.30.1 is-at 1a:eb:58:ad:e0:95, length 28
^C
12 packets captured
12 packets received by filter
0 packets dropped by kernel
Hypervisor70:
Code:
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on bond0, link-type EN10MB (Ethernet), snapshot length 262144 bytes
15:13:22.382357 e6:f2:62:88:90:2c > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 46: vlan 30, p 0, ethertype ARP (0x0806), Request who-has 192.168.30.1 tell 192.168.30.20, length 28
15:13:22.382580 1a:eb:58:ad:e0:95 > e6:f2:62:88:90:2c, ethertype 802.1Q (0x8100), length 64: vlan 30, p 0, ethertype ARP (0x0806), Reply 192.168.30.1 is-at 1a:eb:58:ad:e0:95, length 46
15:13:23.439895 e6:f2:62:88:90:2c > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 46: vlan 30, p 0, ethertype ARP (0x0806), Request who-has 192.168.30.1 tell 192.168.30.20, length 28
15:13:23.440160 1a:eb:58:ad:e0:95 > e6:f2:62:88:90:2c, ethertype 802.1Q (0x8100), length 64: vlan 30, p 0, ethertype ARP (0x0806), Reply 192.168.30.1 is-at 1a:eb:58:ad:e0:95, length 46
15:13:24.463872 e6:f2:62:88:90:2c > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 46: vlan 30, p 0, ethertype ARP (0x0806), Request who-has 192.168.30.1 tell 192.168.30.20, length 28
15:13:24.464177 1a:eb:58:ad:e0:95 > e6:f2:62:88:90:2c, ethertype 802.1Q (0x8100), length 64: vlan 30, p 0, ethertype ARP (0x0806), Reply 192.168.30.1 is-at 1a:eb:58:ad:e0:95, length 46
15:13:25.488206 e6:f2:62:88:90:2c > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 46: vlan 30, p 0, ethertype ARP (0x0806), Request who-has 192.168.30.1 tell 192.168.30.20, length 28
15:13:25.488502 1a:eb:58:ad:e0:95 > e6:f2:62:88:90:2c, ethertype 802.1Q (0x8100), length 64: vlan 30, p 0, ethertype ARP (0x0806), Reply 192.168.30.1 is-at 1a:eb:58:ad:e0:95, length 46
15:13:26.511896 e6:f2:62:88:90:2c > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 46: vlan 30, p 0, ethertype ARP (0x0806), Request who-has 192.168.30.1 tell 192.168.30.20, length 28
15:13:26.512316 1a:eb:58:ad:e0:95 > e6:f2:62:88:90:2c, ethertype 802.1Q (0x8100), length 64: vlan 30, p 0, ethertype ARP (0x0806), Reply 192.168.30.1 is-at 1a:eb:58:ad:e0:95, length 46
15:13:27.536006 e6:f2:62:88:90:2c > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 46: vlan 30, p 0, ethertype ARP (0x0806), Request who-has 192.168.30.1 tell 192.168.30.20, length 28
15:13:27.536491 1a:eb:58:ad:e0:95 > e6:f2:62:88:90:2c, ethertype 802.1Q (0x8100), length 64: vlan 30, p 0, ethertype ARP (0x0806), Reply 192.168.30.1 is-at 1a:eb:58:ad:e0:95, length 46
^C
12 packets captured
12 packets received by filter
0 packets dropped by kernel
tcpdump on the bridge on all 3 hypervisors
Code:
Hypervisor50 & Hypervisor70 (hosting Linux VMs & trying to pass trunk):
tcpdump -i vmbr0 -nn -e vlan 30
Hypervisor60 (Hosting Vyos VM and bridging a VLAN tagged interface on the hypervisor):
tcpdump -i vmbr30 -nn -e
Hypervisor50:
Nothing
Hypervisor60:
Code:
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on vmbr30, link-type EN10MB (Ethernet), snapshot length 262144 bytes
15:18:41.830904 e6:f2:62:88:90:2c > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: Request who-has 192.168.30.1 tell 192.168.30.20, length 46
15:18:41.831143 1a:eb:58:ad:e0:95 > e6:f2:62:88:90:2c, ethertype ARP (0x0806), length 42: Reply 192.168.30.1 is-at 1a:eb:58:ad:e0:95, length 28
15:18:42.869372 e6:f2:62:88:90:2c > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: Request who-has 192.168.30.1 tell 192.168.30.20, length 46
15:18:42.869586 1a:eb:58:ad:e0:95 > e6:f2:62:88:90:2c, ethertype ARP (0x0806), length 42: Reply 192.168.30.1 is-at 1a:eb:58:ad:e0:95, length 28
15:18:43.893525 e6:f2:62:88:90:2c > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: Request who-has 192.168.30.1 tell 192.168.30.20, length 46
15:18:43.893741 1a:eb:58:ad:e0:95 > e6:f2:62:88:90:2c, ethertype ARP (0x0806), length 42: Reply 192.168.30.1 is-at 1a:eb:58:ad:e0:95, length 28
15:18:44.917466 e6:f2:62:88:90:2c > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: Request who-has 192.168.30.1 tell 192.168.30.20, length 46
15:18:44.917670 1a:eb:58:ad:e0:95 > e6:f2:62:88:90:2c, ethertype ARP (0x0806), length 42: Reply 192.168.30.1 is-at 1a:eb:58:ad:e0:95, length 28
15:18:45.941512 e6:f2:62:88:90:2c > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: Request who-has 192.168.30.1 tell 192.168.30.20, length 46
15:18:45.941677 1a:eb:58:ad:e0:95 > e6:f2:62:88:90:2c, ethertype ARP (0x0806), length 42: Reply 192.168.30.1 is-at 1a:eb:58:ad:e0:95, length 28
15:18:46.965445 e6:f2:62:88:90:2c > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: Request who-has 192.168.30.1 tell 192.168.30.20, length 46
15:18:46.965696 1a:eb:58:ad:e0:95 > e6:f2:62:88:90:2c, ethertype ARP (0x0806), length 42: Reply 192.168.30.1 is-at 1a:eb:58:ad:e0:95, length 28
^C
12 packets captured
12 packets received by filter
0 packets dropped by kernel
Hypervisor70:
Code:
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on vmbr0, link-type EN10MB (Ethernet), snapshot length 262144 bytes
15:18:21.648543 e6:f2:62:88:90:2c > 33:33:00:00:00:02, ethertype 802.1Q (0x8100), length 66: vlan 30, p 0, ethertype IPv6 (0x86dd), fe80::5399:8f70:b17b:5292 > ff02::2: ICMP6, router solicitation, length 8
15:18:41.825616 e6:f2:62:88:90:2c > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 46: vlan 30, p 0, ethertype ARP (0x0806), Request who-has 192.168.30.1 tell 192.168.30.20, length 28
15:18:42.864086 e6:f2:62:88:90:2c > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 46: vlan 30, p 0, ethertype ARP (0x0806), Request who-has 192.168.30.1 tell 192.168.30.20, length 28
15:18:43.888252 e6:f2:62:88:90:2c > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 46: vlan 30, p 0, ethertype ARP (0x0806), Request who-has 192.168.30.1 tell 192.168.30.20, length 28
15:18:44.912184 e6:f2:62:88:90:2c > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 46: vlan 30, p 0, ethertype ARP (0x0806), Request who-has 192.168.30.1 tell 192.168.30.20, length 28
15:18:45.936221 e6:f2:62:88:90:2c > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 46: vlan 30, p 0, ethertype ARP (0x0806), Request who-has 192.168.30.1 tell 192.168.30.20, length 28
15:18:46.960162 e6:f2:62:88:90:2c > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 46: vlan 30, p 0, ethertype ARP (0x0806), Request who-has 192.168.30.1 tell 192.168.30.20, length 28
^C
7 packets captured
7 packets received by filter
0 packets dropped by kernel
I have been searching online for what I might have mis-configured but from everything I read, it sounds like this should work? Any help is greatly appreciated!