PVE Hosts and VMs on VLANS issue [Solved]

[alexinux]

Hey all,

This might be something dumb i missed while configuring VLANs on the host but here it goes...
I have 5 VLANs on my network (10=PCs, 20=Laptops, 30=Printers, 40=Servers & 50=Management). I have 3 swiches and 3 PVEs on VLAN 40 and all the VMs on the same VLAN 40 inside the hosts since they are servers as well (Domain controller, File Server, etc.). I had the ports on the switch to access but with VLAN 40 and I had not configured the VLAN Aware at the bridge since everything work. Forward to this past week and now I have to create VMs that are not servers with VLAN 10 within the PVEs for testing configurations instead of using physical hardware and I can't ping the VMs inside the host. The changes I did are as follows:

- On the switches I configured the ports that previously were "access VLAN 40" to trunks passing all VLANS.
- On the PVEs I configured VMBR0 to be VLAN Aware
- I tagged all servers withing the PVEs with 40 as that is the VLAN they should be
- I created VMBR0.40 on each of the PVEs and gave it their respective IPs for each host
- I removed the hosts IP configuration from VMBR0 as it is supposed to be attached to the newly created VMBR0.40

Now i can ping everything on the network minus the Server VMs (VLAN 40) and the Test VM (VLAN 10) within the PVEs. Was there something I missed? It worked before the trunking of the ports because I guess everything inside the hosts was VLAN 40 and the hosts as well. All VMs are attached to VMBR0 now and the PVEs are attached to VMBR0.40.... is this ok? Do I have to create more VLAN intefaces for communication between VLANs or VMs and the network?

Attached is what I have for one of the PVEs but the rest of the have the same configuration, only diference is the IP for each PVE.

 

[dj423]

As long as you are setting the tag on the vm interfaces Example:

net0: virtio=XX:34:12:XX:0A:F1,bridge=vmbr0,tag=40

you should be able to pass tagged packets between pve hosts and vm nodes if you are just doing standard inter-vlan routing. I could see maybe some route issue to vlan10, but vlan40 should have no issues if the switch is filtering vlan packets correctly. Are your 3 switches connected via trunk interfaces or stacked? If your switch allows you to monitor the layer 2 traffic you can dive deeper into checking vlan tags, but your config looks similar to mine FWIW.
HTH

 

[alexinux]

As long as you are setting the tag on the vm interfaces Example:

net0: virtio=XX:34:12:XX:0A:F1,bridge=vmbr0,tag=40

you should be able to pass tagged packets between pve hosts and vm nodes if you are just doing standard inter-vlan routing. I could see maybe some route issue to vlan10, but vlan40 should have no issues if the switch is filtering vlan packets correctly. Are your 3 switches connected via trunk interfaces or stacked? If your switch allows you to monitor the layer 2 traffic you can dive deeper into checking vlan tags, but your config looks similar to mine FWIW.
HTH

The 3 switches are connected via trunks, all have VLAN50 as the native VLAN and they pass VLANs on the network so I have no issue there. I can poing all devices on al VLANs in the network but not the VMs inside the PVEs. I can ping the PVEs on VLAN 40 with the new configuration not even the PVEs can ping the inside VMs. I didnt reboot the PVEs when I made the changes just did "systemctl restart networking".... I don't recall seeing you had to reboot for all changes to take effect.

 

[alexinux]

As long as you are setting the tag on the vm interfaces Example:

net0: virtio=XX:34:12:XX:0A:F1,bridge=vmbr0,tag=40

you should be able to pass tagged packets between pve hosts and vm nodes if you are just doing standard inter-vlan routing. I could see maybe some route issue to vlan10, but vlan40 should have no issues if the switch is filtering vlan packets correctly. Are your 3 switches connected via trunk interfaces or stacked? If your switch allows you to monitor the layer 2 traffic you can dive deeper into checking vlan tags, but your config looks similar to mine FWIW.
HTH

So I have this on all the VMs withing the PVEs and each PVE has a VMBR0.40 interface which puts the host on the VLAN 40 that I want it in. The VMBR0 then changes to be the main interface for everything to get out but does not need an IP address as it will act as a trunk with the "VLAN Aware" option. Is this correct? or do I have to create more VMBR0.xx interfaces for the different VMs that will be hosted inside the PVEs?

 

[dj423]

Are any VM's passing tagged traffic through the bridge can you tell? Since you are able to ping the pve nodes, at least the tagged management interface would seem to work correct on vlan40, so your VM's (at least on vlan40) should pass tagged packets as well.

 

[alexinux]

Are any VM's passing tagged traffic through the bridge can you tell? Since you are able to ping the pve nodes, at least the tagged management interface would seem to work correct on vlan40, so your VM's (at least on vlan40) should pass tagged packets as well.

I've logged in into all VMs to ping out and they can't even ping the host they are hosted in nor the VMs within the host.

 

[dj423]

Can they ping the gateway, 40.1?

 

[alexinux]

Can they ping the gateway, 40.1?

Before the VMBR0.40 adn the port on the switch went from access to trunk they did ping the gateway. After I put the port on the switch as a trunk and created the VMBR0.40 for the host IP they cant.

 

[dj423]

Ah ok. So I would investigate the switch setup on the trunk config as well as the bond. Are you tagging the interfaces on the switch, or tagging the bonded interface? Wish I was more help, but have not seen those symptoms running bonded nics on the PVE hosts as trunk ports in my endeavors. If the management interface (vmbr0.40) is able to pass traffic, I don't see why the VM's wont as long as you are setting them to tagged (tag=40) on the interface as above.

 

[alexinux]

Ah ok. So I would investigate the switch setup on the trunk config as well as the bond. Are you tagging the interfaces on the switch, or tagging the bonded interface? Wish I was more help, but have not seen those symptoms running bonded nics on the PVE hosts as trunk ports in my endeavors. If the management interface (vmbr0.40) is able to pass traffic, I don't see why the VM's wont as long as you are setting them to tagged (tag=40) on the interface as above.

Yes i forgot to mention the Bonded interfaces. Each PVE has 2 NICs which I bonded together for more bandwidth. They worked without issue when they were access ports on the switch and the switches also had LACP on the "bonded" ports towards the PVEs. I will reboot all the PVEs to see if the allow traffic from the VMs. Its rare to me as I followed the guidance for using VLANs on Proxmox but for some reason had this outcome.

 

[dj423]

Well your config looks correct to me as for pve, now the switch, that's a whole other rabbit hole to confirm. Might throw a test machine on the switch port to test passing tagged traffic through the bond, or a known good config for other ports to make sure your switching is configured correct.

FWIW I use a mikrotik crs317 for core switching duties, and I make sure to set the vlan filtering to only allow tagged traffic on the trunks, then just make sure I tag everything ingress/egress from the pve host bonded trunk ports. The native vlan in my case is pvid=1, which is usually the native vlan on most cisco/MikroTik gear in my experience.

My bonds/VLAN99 as example:

/interface bonding
add mode=802.3ad name=pve-bond1 slaves=sfp-sfpplus13,sfp-sfpplus14 transmit-hash-policy=layer-2-and-3
add mode=802.3ad name=pve-bond2 slaves=sfp-sfpplus15,sfp-sfpplus16 transmit-hash-policy=layer-2-and-3
add mode=802.3ad name=pve-bond3 slaves=sfp-sfpplus9,sfp-sfpplus10 transmit-hash-policy=layer-2-and-3


/interface bridge port
add bridge=bridge frame-types=admit-only-vlan-tagged interface=pve-bond1

/interface bridge vlan
add bridge=bridge tagged="bridge,ether1,sfp-sfpplus1,sfp-sfpplus2,sfp-sfpplus3,sfp-sfpplus4,sfp-sfpplus5,sf\
    p-sfpplus6,sfp-sfpplus7,sfp-sfpplus8,sfp-sfpplus9,sfp-sfpplus10,sfp-sfpplus11,sfp-sfpplus12,pve-bond1,p\
    ve-bond2,pve-bond3" vlan-ids=99

# corosync vlan
add bridge=bridge comment=PVE tagged=\
    bridge,pve-bond1,pve-bond2,pve-bond3,sfp-sfpplus1,sfp-sfpplus2,sfp-sfpplus3,sfp-sfpplus4 vlan-ids=45

 

[alexinux]

Issue resolved at least for the servers and containers with other VLANs

We use Cisco Switches and the following worked on their config and the PVEs config:

#### If you are using LACP/Bond Channels ####
Interface Port-Channel1
description <your description for this Port-Channel>
switchport trunk native VLAN <your native vlan, 50 is our mgmt vlan>
switchport trunk allowed vlan <10,20,30,40,50> ###put only the vlans in your network, we removed the 2-4096 config as per security policy
switchport mode trunk

#### On the physical ports we used this although I don't know if its redundant if configured already did it for the Port-Channel
###since we are using multiple ports for the connections; otherwise just use single port configs
Interface Range GigabitEthernet 1/0/23-24
switchport trunk native vlan 50
switchport trunk allowed vlan <10,20,30,40,50>
switchport mode trunk
channel-protocol lacp
channel-group 1 mode active
spanning-tree portfast
spanning-tree bpduguard enable
spanning-tree guard root
ip verify source
ip dhcp snooping trust

### On the PVEs we have the following for their ports:
/etc/network/interfaces

auto lo
iface lo inet loopback

###NIC 1
auto eno0np0
iface eno0np0 inet manual

###NIC 12
auto enp96s0f1np1
iface enp96s0f1np1 inet manual

###NIC 13
auto enp134s0
iface enp134s0 inet manual

auto bond0
iface bond0 inet manual
bond-slaves eno0np0 enp96s0f1np1 enp134s0
bond-miimon 100
bond-mode 802.3ad
bond-xmit-hash-policy layer2+3

auto vmbr0
iface vmbr0 inet manual
bridge-ports bond0
bridge-stp off
bridge-fd 0
bridge-vlan-aware yes
bridge-vids 10 20 30 40 50

auto vmbr0.40
iface vmbr0.40 inet static
addressc<our ip address and /mask>
gateway <our gateway>
dns-nameserver <our dns ip>