VLANs broke in PVE-7.2-5?

[TechTy]

I have a VLAN (VLAN2) that I recently setup to separate all my hosts that are accessible from outside my network from the internal hosts. Everything was working fine until I made two changes (not sure which one broke the VLAN). The first change was upgrading my PVE-7.0-2 (or something close to that version) to PVE-7.2-5. The second change was removing all my DHCP leases but then realizing that my switch had a DHCP IP and that was now gone. I noticed this when I noticed I couldn't access any of my machines on VLAN 2. I couldn't get the switch to take an IP so I reset the switch to factory default. Since these two changes I haven't been able to get any traffic to flow on VLAN 2. I have tried assigning static IPs and also tried DHCP and there seems to be no connection between my VMs and the firewall. The switch config was pretty simple before and the only setting I question is whether 802.1Q should be enabled or disabled or does it matter at all? (see screenshot 1).

I haven't changed any of the settings on the firewall and I can ping both the firewall and my PVE server on VLAN 2 so I know the network is working and I assume the switch isn't causing issues because I can connect with my server on VLAN 2 but none of my VLAN 2 machines are getting an IP address and even with static IPs they can't reach anything, and can't be reached on any network when all was working fine before. My firewall is a pfsense firewall and the switch is a Netgear GS308E. I will include some screenshots of my configs. Any help is greatly appreciated. Thanks in advance.

 

[TechTy]

help?

 

[aaron]

Can you post the contents of the /etc/network/interface file of the Proxmox VE server inside [code][/code] tags?

cat /etc/network/interfaces

And please the config of one such VM?

qm config <vmid>

 

[TechTy]

Interface:

# network interface settings; autogenerated
# Please do NOT modify this file directly, unless you know what
# you're doing.
#
# If you want to manage parts of the network configuration manually,
# please utilize the 'source' or 'source-directory' directives to do
# so.
# PVE will preserve these directives, but will NOT read its network
# configuration from sourced files, so do not attempt to move any of
# the PVE managed interfaces into external files!

auto lo
iface lo inet loopback

iface enp1s0f0 inet manual

iface enp1s0f1 inet manual

auto vmbr0
iface vmbr0 inet static
    address 10.40.40.4/24
    gateway 10.40.40.1
    bridge-ports enp1s0f0
    bridge-stp off
    bridge-fd 0
    bridge-vlan-aware yes
    bridge-vids 2-4094

auto vmbr1
iface vmbr1 inet manual
    bridge-ports none
    bridge-stp off
    bridge-fd 0
    bridge-vlan-aware yes
    bridge-vids 2-4094
#PXEBridge

auto vlan2
iface vlan2 inet static
    address 10.40.1.4/24
    vlan-raw-device enp1s0f0
#VLAN2

qm config 107:

boot: order=scsi0
cores: 1
memory: 4608
name: Nextcloud
net0: virtio=4E:88:9D:C6:8C:31,bridge=vmbr0,firewall=1,tag=2
numa: 0
onboot: 1
ostype: l26
scsi0: myZFS:vm-107-disk-0,size=32G
scsi1: myZFS:vm-107-disk-1,size=1T
smbios1: uuid=de6668c5-9026-4438-a9ff-07e341eee0ce
sockets: 4
vga: qxl
vmgenid: ab15cc55-9739-41f8-9bd4-d9286e431660

 

[aaron]

The guests don't have a VLAN configured themselves right? Just a matching IP address on the interface they see?

The Proxmox VE host can communicate with other physical boxes on the same VLAN IIUC?

What makes me a bit suspicious (without knowing the netgear interface too well) is that VLAN 2 doesn't seem to have any port associated? It should be defined as a tagged VLAN on the ports on the switch where you need it.

 

[TechTy]

I tried some troubleshooting with changing the tagged status of VLANs on my switch and rebooted the PVE server after most of the changes to try and find the correct configuration but I don't really know that much about how VLANs are supposed to be configured for tagging on the switch side (I know how to set them up on the firewall side) and I'm pretty much guessing at this point. The firewall is on port 1 and the server is on port 2. The default VLAN is VLAN 1 and the "Public Network" VLAN is VLAN 2. I can also no longer ping the PVE on the VLAN 2 address which I could do before. Could you provide some general guidance for how to setup the VLANs to tagging to point me in the correct direction? I will try to do some organized troubleshooting tomorrow and provide all the configurations I tried and the outcomes of each configuration. Thanks.

 

[TechTy]

I finally got it to work again. I ended up changing the VLAN 2 tag to VLAN 4 to start over with my coniguration. I didn't setup the PVE to use the VLAN in the network config since I don't want my PVE to have an IP on VLAN 4. I set the switch VLAN 4 config to tag port 1 (connected to the firewall) and 2 and left all the VLAN 1 ports as untagged. Everything is working as expected now. Thank you for all your help.

 

[aaron]

Even if you did manage to get it working, I think a small explanation would be good, if other people find this thread in the future.

VLANs can usually be configured in a few ways. They can be untagged, which means, that the whole port is part of a VLAN (without manually changing anything, this is usually VLAN 1). Then there can be multiple VLANs configured as tagged VLANs. The difference is, that on an untagged VLAN, the client connecting to the switch is not aware of any vlan tags on the network packets. For tagged vlans, the client needs to add the vlan tags to the network packets. This makes it possible to have multiple different vlans on a single port on the switch.

Switches usually need to be configured accordingly to accept vlan tags. If you only want tagged traffic on a port, without any untagged packets being accepted, it is usually known as trunking. Used mostly to connect different parts of the network to pipe through many vlans. But even then, one usually needs to configure which vlan IDs are accepted.

 

[cwathan]

To provide even deeper network information... Switches, even on trunked ports, still accept untagged traffic. That traffic is dumped into the "default VLAN." There are network management protocols that do not function within VLANs. But, I probably shouldn't confuse the OP. I just wanted to let people know that there will still be untagged traffic flow on TRUNK ports, in MANY cases, to the default VLAN, and why.

 

[TechTy]

Thank you for the excellent explanations! It is very helpful.

 

[John Driessen]

For my understanding:
When I have a port on the managed switch on which I want to use traffic of multiple vlans (the Proxmox Server is on one of them, containing VMs in the other vlans) , then I need to tag all these vlans on that port on the managed switch?
And then in Proxmox (having an IP in the main vlan (the PVID) it would be sufficient to provide the tag of the vlan for the VMs and then it should be working?
Additional info: I have a unmanaged switch connected to that port on the managed switch. The Proxmox Server, but also other pc's and laptops anre connected to the unmanaged switch. These other pcs and laptops will sending untagged traffic, so will end up in the PVID (main vlan)

 

[aaron]

When I have a port on the managed switch on which I want to use traffic of multiple vlans (the Proxmox Server is on one of them, containing VMs in the other vlans) , then I need to tag all these vlans on that port on the managed switch?

Yes. So that the switch knows which VLANs should be available on that port, and if they are tagged or untagged. There can only be on untagged VLAN per port.

And then in Proxmox (having an IP in the main vlan (the PVID) it would be sufficient to provide the tag of the vlan for the VMs and then it should be working?

On the Proxmox VE side, you have a few options on how to set the VLANs. The mgmt IP could be configured on a Linux VLAN interface for example. The VLANs can be tagged on the individual guest NIC. You could also configure multiple vmbr interfaces that use the dot notation in their bridge port. Or use a Linux VLAN interface as bridge port.
You could also make the vmbr interface "vlan aware". With that setting, vlan tags set within the guests will work as well.

And with the SDN you have a few options more

Additional info: I have a unmanaged switch connected to that port on the managed switch. The Proxmox Server, but also other pc's and laptops anre connected to the unmanaged switch. These other pcs and laptops will sending untagged traffic, so will end up in the PVID (main vlan)

Not sure with tagged VLAN traffic will work through such a switch.