Proxmox VLAN ID problem

[VisionSoul]

Hi, hope you all well,

I'm writing, because i have a strange issue when trying to setup vlans on proxmox. My problem is that any vlan id above 999 (4 digit vlan id's ) simply doesn't work. I tested connecting a physical machine to my switch and give it vlan id for ex.. 1000 and everything goes just fine. it receives ip the correct ip, no problem. If i try the same on proxmox vm, it does get connectivity. Any vlan id below 1000 works just fine, only above doesn't.

Anyone here have any suggestions or using vlans id's above 999?

Thanks in advance!

Best Regards!

 

[spirit]

can you send your /etc/network/interfaces ?

do you use vlan aware bridge ? if not, the vlan tag is set on <interface>.<vlan>. The character limits is 14, and if you have a a long interface name like enp0s3dddfe..... it could be too long.

 

[VisionSoul]

can you send your /etc/network/interfaces ?

do you use vlan aware bridge ? if not, the vlan tag is set on <interface>.<vlan>. The character limits is 14, and if you have a a long interface name like enp0s3dddfe..... it could be too long.

Hi,

Yes, because otherwise, below that wouldn't work. My problem is only when trying to use vlan id of 1000 or above. All other vlans ids below 1000 works just fine!

Here's my config:

root@vlab:~# cat /etc/network/interfaces
# network interface settings; autogenerated
# Please do NOT modify this file directly, unless you know what
# you're doing.
#
# If you want to manage parts of the network configuration manually,
# please utilize the 'source' or 'source-directory' directives to do
# so.
# PVE will preserve these directives, but will NOT read its network
# configuration from sourced files, so do not attempt to move any of
# the PVE managed interfaces into external files!

auto lo
iface lo inet loopback

iface eno1 inet manual

auto enp2s0f1
iface enp2s0f1 inet manual

auto enp2s0f0
iface enp2s0f0 inet manual

auto enp6s0
iface enp6s0 inet manual

auto enp6s0d1
iface enp6s0d1 inet manual

iface enp132s0 inet manual

auto bond1
iface bond1 inet manual
bond-slaves enp6s0 enp6s0d1
bond-miimon 100
bond-mode 802.3ad
bond-xmit-hash-policy layer2+3
#VMs BOND

auto vmbr0
iface vmbr0 inet static
address 192.168.2.50/24
gateway 192.168.2.1
bridge-ports enp132s0
bridge-stp off
bridge-fd 0
bridge-vlan-aware yes
bridge-vids 2-4094
#MGMT & Backups

auto vmbr1
iface vmbr1 inet static
address 10.10.20.0/24
bridge-ports bond1
bridge-stp off
bridge-fd 0
bridge-vlan-aware yes
bridge-vids 2-4094
#LAB Network



Thanks

 

[Tekuno-Kage]

Hi,

Yes, because otherwise, below that wouldn't work. My problem is only when trying to use vlan id of 1000 or above. All other vlans ids below 1000 works just fine!

Here's my config:

root@vlab:~# cat /etc/network/interfaces
# network interface settings; autogenerated
# Please do NOT modify this file directly, unless you know what
# you're doing.
#
# If you want to manage parts of the network configuration manually,
# please utilize the 'source' or 'source-directory' directives to do
# so.
# PVE will preserve these directives, but will NOT read its network
# configuration from sourced files, so do not attempt to move any of
# the PVE managed interfaces into external files!

auto lo
iface lo inet loopback

iface eno1 inet manual

auto enp2s0f1
iface enp2s0f1 inet manual

auto enp2s0f0
iface enp2s0f0 inet manual

auto enp6s0
iface enp6s0 inet manual

auto enp6s0d1
iface enp6s0d1 inet manual

iface enp132s0 inet manual

auto bond1
iface bond1 inet manual
bond-slaves enp6s0 enp6s0d1
bond-miimon 100
bond-mode 802.3ad
bond-xmit-hash-policy layer2+3
#VMs BOND

auto vmbr0
iface vmbr0 inet static
address 192.168.2.50/24
gateway 192.168.2.1
bridge-ports enp132s0
bridge-stp off
bridge-fd 0
bridge-vlan-aware yes
bridge-vids 2-4094
#MGMT & Backups

auto vmbr1
iface vmbr1 inet static
address 10.10.20.0/24
bridge-ports bond1
bridge-stp off
bridge-fd 0
bridge-vlan-aware yes
bridge-vids 2-4094
#LAB Network



Thanks

Look like you don't declare a vlan taggin for the host. Unless are you tagging the traffic on a VM...

If you expect that the host send tagged traffic, you need to declare it.
Example:

auto ceph_clstr
iface ceph_clstr inet static
        address 10.196.51.11/24
        mtu 9000
        vlan-id 51
        vlan-raw-device vmbr1
#Virtual SAN Replication | CEPH Cluster

auto ceph_pblc
iface ceph_pblc inet static
        address 10.196.52.11/24
        mtu 9000
        vlan-id 52
        vlan-raw-device vmbr1
#Virtual SAN Access | CEPH Public

 

[spirit]

Hi,

Yes, because otherwise, below that wouldn't work. My problem is only when trying to use vlan id of 1000 or above. All other vlans ids below 1000 works just fine!

Here's my config:

root@vlab:~# cat /etc/network/interfaces
# network interface settings; autogenerated
# Please do NOT modify this file directly, unless you know what
# you're doing.
#
# If you want to manage parts of the network configuration manually,
# please utilize the 'source' or 'source-directory' directives to do
# so.
# PVE will preserve these directives, but will NOT read its network
# configuration from sourced files, so do not attempt to move any of
# the PVE managed interfaces into external files!

auto lo
iface lo inet loopback

iface eno1 inet manual

auto enp2s0f1
iface enp2s0f1 inet manual

auto enp2s0f0
iface enp2s0f0 inet manual

auto enp6s0
iface enp6s0 inet manual

auto enp6s0d1
iface enp6s0d1 inet manual

iface enp132s0 inet manual

auto bond1
iface bond1 inet manual
bond-slaves enp6s0 enp6s0d1
bond-miimon 100
bond-mode 802.3ad
bond-xmit-hash-policy layer2+3
#VMs BOND

auto vmbr0
iface vmbr0 inet static
address 192.168.2.50/24
gateway 192.168.2.1
bridge-ports enp132s0
bridge-stp off
bridge-fd 0
bridge-vlan-aware yes
bridge-vids 2-4094
#MGMT & Backups

auto vmbr1
iface vmbr1 inet static
address 10.10.20.0/24
bridge-ports bond1
bridge-stp off
bridge-fd 0
bridge-vlan-aware yes
bridge-vids 2-4094
#LAB Network



Thanks

Your config is fine. Are you sure than vlans > 1000 are declared on your physical switch ?

 

[VisionSoul]

Look like you don't declare a vlan taggin for the host. Unless are you tagging the traffic on a VM...

If you expect that the host send tagged traffic, you need to declare it.
Example:

auto ceph_clstr
iface ceph_clstr inet static
        address 10.196.51.11/24
        mtu 9000
        vlan-id 51
        vlan-raw-device vmbr1
#Virtual SAN Replication | CEPH Cluster

auto ceph_pblc
iface ceph_pblc inet static
        address 10.196.52.11/24
        mtu 9000
        vlan-id 52
        vlan-raw-device vmbr1
#Virtual SAN Access | CEPH Public

Hi, yes i'm putting vlan id directly on VM, no need on the host. Bridge is already vlan aware. Either way i tested different scenarios, and same result. Changing for ex Test Vlan (vid: 1000) does not work. Changing it to vid: 999 just works.

I don't really understand, since as states on etc/networks/interfaces file bridge is allowing vlans from 2-4094.

Best Regards!

 

[VisionSoul]

Your config is fine. Are you sure than vlans > 1000 are declared on your physical switch ?

Hi,

Yes they are. As i said above, for ex. Test Vlan (vid: 1000) not working., Changing it to 999, starts to work. It grabs right away dhcp lease and works just fine. Anything above vid 999 don't work.

By the way, connecting a physical machine to a access port on the switch on vlan lets say 1000 works just fine. Only vm's on proxmox above vid 999 don't work.

Best Regards!

 

[VisionSoul]

Ok, so new update,

After further testing ( i was testing with round number ( vid 100, vid 1000 ) and assumed that was working with 3 digits vids all the way up to 999. I was wrong. Now i was experimenting with other vids, and it only accepts up to vid 125. If i try vid 126 or anything above it doesn't grab dhcp lease.....

This is very strange...

I attached 2 screenshots with vid 100 and the other with vid 1000 for compare. The result of the vid1000 is the same for all the other above vid 125.

This is some kinda of malfunction on proxmox because testing with a physical machine it grabs dhcp leases just fine. ( tested with raspberrypi) (3rd SS)

Any suggestions of whats is going on here?

 

[spirit]

After further testing ( i was testing with round number ( vid 100, vid 1000 ) and assumed that was working with 3 digits vids all the way up to 999. I was wrong. Now i was experimenting with other vids, and it only accepts up to vid 125. If i try vid 126 or anything above it doesn't grab dhcp lease.....

what is your nic model ? Some nic don't support more than X vlans. (as they are doing vlan offloading). I have see this on mellanox connecx-3 for example.

As you are using vlan aware with "bridge-vids 2-4094", it's trying to allocated all vlan at boot from 2,3,4,5...until 128.

you could try to configure manually the vlan like in "bridge-vids 2,6,10-50,...".

 

[VisionSoul]

what is your nic model ? Some nic don't support more than X vlans. (as they are doing vlan offloading). I have see this on mellanox connecx-3 for example.

As you are using vlan aware with "bridge-vids 2-4094", it's trying to allocated all vlan at boot from 2,3,4,5...until 128.

you could try to configure manually the vlan like in "bridge-vids 2,6,10-50,...".

Hi,

yes bridges are vlan-aware.

Yes my cards are mellanox connectx-3's. I have one single an one dual. Maybe that's the problem then. What dual 10gb sfp+ nics do you recommend?

Thanks!

 

[spirit]

Hi,

yes bridges are vlan-aware.

Yes my cards are mellanox connectx-3's. I have one single an one dual. Maybe that's the problem then. What dual 10gb sfp+ nics do you recommend?

Thanks!

connectx-4 is limited to 512 vlans, not sure about connectx-5.

They both working really fine.

 

[VisionSoul]

connectx-4 is limited to 512 vlans, not sure about connectx-5.

They both working really fine.

Hi,

Ok i'll try to find one at good ebay price.

Thanks for the recommendations!

Best regards!