VLAN aware with multiple nodes

P

pizza

Renowned Member
Nov 7, 2015
96
8
73
When on a single proxmox node the VLAN aware setting is enabled on vmbr0, you can use VLAN tags with VM's on that.

But when you have multiple nodes in a cluster, does VLAN tags also work on other nodes? So vm's with the same VLAN tag can communicate when they run on several nodes?
 
pizza said:
But when you have multiple nodes in a cluster, does VLAN tags also work on other nodes? So vm's with the same VLAN tag can communicate when they run on several nodes?
Click to expand...
Your whole infrastructure needs to be VLAN aware, traffic is tagged with the corresponding VLAN and will travel from one node to the other. If you configure a VLAN on the VM and are not using a VLAN-aware bridge, each VLAN will become its own interface connected to a bridge with only that VLAN.

Code: 
vlan-aware bridge: eno1 <-> vmbr0 <-> tap100i0
non-aware bridge: eno1.10 <-> vmbr0v10 <-> tap100i0
 
I configured the switchport from mode access (was vlan20) to trunk and configured the Proxmox nodes to use vlan 20 voor vmbr0.

iface vmbr0 inet static
address 192.168.89.232
netmask 24
gateway 192.168.89.1
bridge-ports enp3s0f0.20
bridge-stp off
bridge-fd 0
bridge-vlan-aware yes
bridge-vids 2-4094

On both nodes I created 2 containers with VLAN tag=200 under Network Settings. Now the containers can reach each other across the nodes.

If I want to allow only vlan 200-300 for example, the solution is to set bridge-vids 200-300?
 
Last edited:
pizza said:
I configured the switchport from mode access (was vlan20) to trunk and configured the Proxmox nodes to use vlan 20 voor vmbr0.

iface vmbr0 inet static
address 192.168.89.232
netmask 24
gateway 192.168.89.1
bridge-ports enp3s0f0.20
bridge-stp off
bridge-fd 0
bridge-vlan-aware yes
bridge-vids 2-4094
Click to expand...

That's strange it's working with "bridge-ports enp3s0f0.20"

if the vlan 20 is only for proxmox ip management, the correct way is:

Code: 
iface vmbr0 inet manual
        bridge-ports enp3s0f0
        bridge-stp off
        bridge-fd 0
        bridge-vlan-aware yes
        bridge-vids 2-4094

auto iface vmbr0.20
iface vmbr0.20 inet static
        address  192.168.89.232
        netmask  24
        gateway  192.168.89.1


If I want to allow only vlan 200-300 for example, the solution is to set bridge-vids 200-300?
Click to expand...
yes
 
  • Like
Reactions: ebiss
its not strange, its just another way todo it.
it should not make any difference either way
 
Under 5.4 this works perfectly, but after upgrading to 6.1 I have connection problems when vm's are on different nodes.

I can ping the vm's, ssh and telnet to port 80 for example. But with curl of wget for example I cannot get any connection (hangs on request) but tcpdump shows traffic and firewalling is not activated.

When the vm's are on the same node, everything works.


tcpdump -i vmbr0 vlan 500 (vm's are on different nodes and vlan setting=500)

10:18:11.880698 IP 192.168.1.2.45214 > 192.168.1.1.http: Flags , seq 2419906062, win 64240, options [mss 1460,sackOK,TS val 583789432 ecr 0,nop,wscale 7], length 0
10:18:11.880968 IP 192.168.1.1.http > 192.168.1.2.45214: Flags [S.], seq 3088704958, ack 2419906063, win 65160, options [mss 1460,sackOK,TS val 4079919146 ecr 583789432,nop,wscale 7], length 0
10:18:11.881001 IP 192.168.1.2.45214 > 192.168.1.1.http: Flags [.], ack 1, win 502, options [nop,nop,TS val 583789433 ecr 4079919146], length 0
10:18:11.881063 IP 192.168.1.2.45214 > 192.168.1.1.http: Flags [P.], seq 1:76, ack 1, win 502, options [nop,nop,TS val 583789433 ecr 4079919146], length 75: HTTP: GET / HTTP/1.1
10:18:11.881202 IP 192.168.1.1.http > 192.168.1.2.45214: Flags [.], ack 76, win 509, options [nop,nop,TS val 4079919146 ecr 583789433], length 0
10:18:11.881599 IP 192.168.1.1.http > 192.168.1.2.45214: Flags [P.], seq 10137:11174, ack 76, win 509, options [nop,nop,TS val 4079919147 ecr 583789433], length 1037: HTTP
10:18:11.881614 IP 192.168.1.2.45214 > 192.168.1.1.http: Flags [.], ack 1, win 502, options [nop,nop,TS val 583789433 ecr 4079919146,nop,nop,sack 1 {10137:11174}], length 0
10:18:16.885252 IP 192.168.1.1.http > 192.168.1.2.45214: Flags [F.], seq 11174, ack 76, win 509, options [nop,nop,TS val 4079924150 ecr 583789433], length 0
10:18:16.885290 IP 192.168.1.2.45214 > 192.168.1.1.http: Flags [.], ack 1, win 502, options [nop,nop,TS val 583794437 ecr 4079919146,nop,nop,sack 1 {10137:11175}], length 0
10:18:16.897184 ARP, Request who-has 192.168.1.1 tell 192.168.1.2, length 28
10:18:16.897280 ARP, Reply 192.168.1.1 is-at 2a:b5:60:d9:02:cc (oui Unknown), length 42
10:18:46.914183 IP 192.168.1.1.http > 192.168.1.2.45214: Flags [R.], seq 11175, ack 76, win 509, options [nop,nop,TS val 4079954179 ecr 583794437], length 0
 
Last edited:
IPv6 gives also the same problem:

10:24:48.115563 IP6 2a02:1c0:5::aaa:1.52934 > 2a02:1c0:5::aaa:2.http: Flags [P.], seq 1:84, ack 1, win 507, options [nop,nop,TS val 3869184249 ecr 1147001774], length 83: HTTP: GET / HTTP/1.1
10:24:48.115628 IP6 2a02:1c0:5::aaa:2.http > 2a02:1c0:5::aaa:1.52934: Flags [.], ack 84, win 502, options [nop,nop,TS val 1147001774 ecr 3869184249], length 0
10:24:48.129199 IP6 2a02:1c0:5::aaa:2.http > 2a02:1c0:5::aaa:1.52934: Flags [.], seq 1:1429, ack 84, win 502, options [nop,nop,TS val 1147001788 ecr 3869184249], length 1428: HTTP: HTTP/1.1 200 OK
10:24:48.337165 IP6 2a02:1c0:5::aaa:2.http > 2a02:1c0:5::aaa:1.52934: Flags [.], seq 1:1429, ack 84, win 502, options [nop,nop,TS val 1147001996 ecr 3869184249], length 1428: HTTP: HTTP/1.1 200 OK
10:24:48.769162 IP6 2a02:1c0:5::aaa:2.http > 2a02:1c0:5::aaa:1.52934: Flags [.], seq 1:1429, ack 84, win 502, options [nop,nop,TS val 1147002428 ecr 3869184249], length 1428: HTTP: HTTP/1.1 200 OK
10:24:49.605194 IP6 2a02:1c0:5::aaa:2.http > 2a02:1c0:5::aaa:1.52934: Flags [.], seq 1:1429, ack 84, win 502, options [nop,nop,TS val 1147003264 ecr 3869184249], length 1428: HTTP: HTTP/1.1 200 OK
10:24:51.265180 IP6 2a02:1c0:5::aaa:2.http > 2a02:1c0:5::aaa:1.52934: Flags [.], seq 1:1429, ack 84, win 502, options [nop,nop,TS val 1147004924 ecr 3869184249], length 1428: HTTP: HTTP/1.1 200 OK
10:24:53.120204 IP6 2a02:1c0:5::aaa:2.http > 2a02:1c0:5::aaa:1.52934: Flags [F.], seq 11174, ack 84, win 502, options [nop,nop,TS val 1147006779 ecr 3869184249], length 0
10:24:53.120314 IP6 2a02:1c0:5::aaa:1.52934 > 2a02:1c0:5::aaa:2.http: Flags [.], ack 1, win 507, options [nop,nop,TS val 3869189254 ecr 1147001774,nop,nop,sack 1 {9997:11175}], length 0
10:24:53.120362 IP6 2a02:1c0:5::aaa:2.http > 2a02:1c0:5::aaa:1.52934: Flags [.], seq 1:1429, ack 84, win 502, options [nop,nop,TS val 1147006779 ecr 3869189254], length 1428: HTTP: HTTP/1.1 200 OK
10:24:53.325136 IP6 2a02:1c0:5::aaa:2.http > 2a02:1c0:5::aaa:1.52934: Flags [.], seq 1:1429, ack 84, win 502, options [nop,nop,TS val 1147006984 ecr 3869189254], length 1428: HTTP: HTTP/1.1 200 OK
10:24:53.761140 IP6 2a02:1c0:5::aaa:2.http > 2a02:1c0:5::aaa:1.52934: Flags [.], seq 1:1429, ack 84, win 502, options [nop,nop,TS val 1147007420 ecr 3869189254], length 1428: HTTP: HTTP/1.1 200 OK
10:24:54.593142 IP6 2a02:1c0:5::aaa:2.http > 2a02:1c0:5::aaa:1.52934: Flags [.], seq 1:1429, ack 84, win 502, options [nop,nop,TS val 1147008252 ecr 3869189254], length 1428: HTTP: HTTP/1.1 200 OK
10:24:56.261206 IP6 2a02:1c0:5::aaa:2.http > 2a02:1c0:5::aaa:1.52934: Flags [.], seq 1:1429, ack 84, win 502, options [nop,nop,TS val 1147009920 ecr 3869189254], length 1428: HTTP: HTTP/1.1 200 OK


root@node1:/etc/network/if-up.d# tcpdump -i enp3s0f1 -nn -e vlan | grep 'vlan 500'
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on enp3s0f1, link-type EN10MB (Ethernet), capture size 262144 bytes
11:19:17.234298 32:19:12:94:4b:13 > 2a:b5:60:d9:02:cc, ethertype 802.1Q (0x8100), length 82: vlan 10, p 0, ethertype 802.1Q, vlan 500, p 0, ethertype IPv4, 192.168.1.2.49578 > 192.168.1.1.80: Flags , seq 4187764128, win 64240, options [mss 1460,sackOK,TS val 587454786 ecr 0,nop,wscale 7], length 0
11:19:17.234342 2a:b5:60:d9:02:cc > 32:19:12:94:4b:13, ethertype 802.1Q (0x8100), length 82: vlan 10, p 0, ethertype 802.1Q, vlan 500, p 0, ethertype IPv4, 192.168.1.1.80 > 192.168.1.2.49578: Flags [S.], seq 1909612345, ack 4187764129, win 65160, options [mss 1460,sackOK,TS val 4083584500 ecr 587454786,nop,wscale 7], length 0
11:19:17.234446 32:19:12:94:4b:13 > 2a:b5:60:d9:02:cc, ethertype 802.1Q (0x8100), length 74: vlan 10, p 0, ethertype 802.1Q, vlan 500, p 0, ethertype IPv4, 192.168.1.2.49578 > 192.168.1.1.80: Flags [.], ack 1, win 502, options [nop,nop,TS val 587454786 ecr 4083584500], length 0
11:19:17.234548 32:19:12:94:4b:13 > 2a:b5:60:d9:02:cc, ethertype 802.1Q (0x8100), length 149: vlan 10, p 0, ethertype 802.1Q, vlan 500, p 0, ethertype IPv4, 192.168.1.2.49578 > 192.168.1.1.80: Flags [P.], seq 1:76, ack 1, win 502, options [nop,nop,TS val 587454786 ecr 4083584500], length 75: HTTP: GET / HTTP/1.1
11:19:17.234592 2a:b5:60:d9:02:cc > 32:19:12:94:4b:13, ethertype 802.1Q (0x8100), length 74: vlan 10, p 0, ethertype 802.1Q, vlan 500, p 0, ethertype IPv4, 192.168.1.1.80 > 192.168.1.2.49578: Flags [.], ack 76, win 509, options [nop,nop,TS val 4083584500 ecr 587454786], length 0
11:19:17.234876 2a:b5:60:d9:02:cc > 32:19:12:94:4b:13, ethertype 802.1Q (0x8100), length 1522: vlan 10, p 0, ethertype 802.1Q, vlan 500, p 0, ethertype IPv4, 192.168.1.1.80 > 192.168.1.2.49578: Flags [.], seq 1:1449, ack 76, win 509, options [nop,nop,TS val 4083584500 ecr 587454786], length 1448: HTTP: HTTP/1.1 200 OK
11:19:17.234878 2a:b5:60:d9:02:cc > 32:19:12:94:4b:13, ethertype 802.1Q (0x8100), length 1522: vlan 10, p 0, ethertype 802.1Q, vlan 500, p 0, ethertype IPv4, 192.168.1.1.80 > 192.168.1.2.49578: Flags [.], seq 1449:2897, ack 76, win 509, options [nop,nop,TS val 4083584500 ecr 587454786], length 1448: HTTP
11:19:17.234878 2a:b5:60:d9:02:cc > 32:19:12:94:4b:13, ethertype 802.1Q (0x8100), length 1522: vlan 10, p 0, ethertype 802.1Q, vlan 500, p 0, ethertype IPv4, 192.168.1.1.80 > 192.168.1.2.49578: Flags [.], seq 2897:4345, ack 76, win 509, options [nop,nop,TS val 4083584500 ecr 587454786], length 1448: HTTP
11:19:17.234879 2a:b5:60:d9:02:cc > 32:19:12:94:4b:13, ethertype 802.1Q (0x8100), length 1522: vlan 10, p 0, ethertype 802.1Q, vlan 500, p 0, ethertype IPv4, 192.168.1.1.80 > 192.168.1.2.49578: Flags [.], seq 4345:5793, ack 76, win 509, options [nop,nop,TS val 4083584500 ecr 587454786], length 1448: HTTP
11:19:17.234880 2a:b5:60:d9:02:cc > 32:19:12:94:4b:13, ethertype 802.1Q (0x8100), length 1522: vlan 10, p 0, ethertype 802.1Q, vlan 500, p 0, ethertype IPv4, 192.168.1.1.80 > 192.168.1.2.49578: Flags [.], seq 5793:7241, ack 76, win 509, options [nop,nop,TS val 4083584500 ecr 587454786], length 1448: HTTP
11:19:17.234889 2a:b5:60:d9:02:cc > 32:19:12:94:4b:13, ethertype 802.1Q (0x8100), length 1522: vlan 10, p 0, ethertype 802.1Q, vlan 500, p 0, ethertype IPv4, 192.168.1.1.80 > 192.168.1.2.49578: Flags [.], seq 7241:8689, ack 76, win 509, options [nop,nop,TS val 4083584500 ecr 587454786], length 1448: HTTP
 
Last edited:
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom
Back