[SOLVED] New host NIC breaks VLAN for only some VMs

[kblock]

Hi all, I've got a weird problem here that has had me scratching my head for a while and I am running out of ideas.

I added a new NIC to my host in order to get some dedicated NICs for opnsense - this all went fine, though I had to change the host network configuration to point to enp6s0 instead of enp5s0.

Since then, I have a VM and a couple of LXCs that just will not work on that VLAN - most work totally fine, but these few specific workloads just won't take.

I have tried a few things:

Completely redoing the network/vlan configuration on my switch, firewall, and proxmox itself
Removing the NIC from the VM and re-adding it
Removing the VLAN tagging immediately makes the systems work and they happily pick up an IP from that VLAN
Adding the VLAN tag back is as if it just isn't connected - ip a shows as "state down"
Setting a static IP doesn't make a difference when VLAN tagged

I am clearly missing something, and the fact that most LXCs happily pick up a DHCP address from that VLAN makes me think it is an OS problem more than a proxmox or broader network problem - but I can't think where to look next.

Any suggestions greatly appreciated!

Details of one failing LXC here:

root@nfs ~# cat /etc/os-release
PRETTY_NAME="Debian GNU/Linux 11 (bullseye)"
NAME="Debian GNU/Linux"
VERSION_ID="11"
VERSION="11 (bullseye)"
VERSION_CODENAME=bullseye
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"
root@nfs ~# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eth0@if139: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether b2:97:f3:90:b6:44 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 10.0.1.190/24 brd 10.0.1.255 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::b097:f3ff:fe90:b644/64 scope link
       valid_lft forever preferred_lft forever
152: eth1@if153: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether f2:a0:0f:96:24:07 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet6 fe80::f0a0:fff:fe96:2407/64 scope link
       valid_lft forever preferred_lft forever


lxc.conf:
arch: amd64
cores: 1
features: nesting=1
hostname: nfs
memory: 512
mp0: /core/data,mp=/core/data
net0: name=eth0,bridge=vmbr0,firewall=1,hwaddr=B2:97:F3:90:B6:44,ip=dhcp,type=veth
net1: name=eth1,bridge=vmbr0,firewall=1,hwaddr=F2:A0:0F:96:24:07,ip=dhcp,tag=2,type=veth
onboot: 1
ostype: debian
rootfs: apps:102/vm-102-disk-0.raw,size=8G
swap: 512
unprivileged: 1

 

[kblock]

More information, specifically with LXC 102

I can see the following on the DHCP server logs, the earliest two are after an ifup eth0, the following 4 are after clearing the vlan tag and doing the same (note, I have removed eth1 and changed the MAC address of eth0)

2023-10-29T16:39:33 Informational dhcpd DHCPACK on 10.0.1.190 to da:e2:c0:5e:80:5c via igb1
2023-10-29T16:39:33 Informational dhcpd DHCPREQUEST for 10.0.1.190 (10.0.1.1) from da:e2:c0:5e:80:5c via igb1
2023-10-29T16:39:33 Informational dhcpd DHCPOFFER on 10.0.1.190 to da:e2:c0:5e:80:5c via igb1
2023-10-29T16:39:32 Informational dhcpd DHCPDISCOVER from da:e2:c0:5e:80:5c via igb1
2023-10-29T16:39:00 Informational dhcpd DHCPOFFER on 10.0.100.62 to da:e2:c0:5e:80:5c via vlan02
2023-10-29T16:39:00 Informational dhcpd DHCPDISCOVER from da:e2:c0:5e:80:5c via vlan02

 

[kblock]

Alrighty, got somewhere and managed to get it working.

This old thread had a different way of running VLANs - https://forum.proxmox.com/threads/proxmox-opnsense-vlan.90634/

Effectively what I have done is all on the proxmox host networking, and is as follows:

1. Added new Linux VLAN pointing to enp6s0.2
2. Added new Linux Bridge explicitly using that new VLAN as the port
3. Updated all VMs (and containers) to the new Linux Bridge AND set them to untagged

There might be more efficient ways, but at least this both works and makes some logical sense.