[SOLVED] Proxmox VM combining tagged and untagged VLAN traffic (host behind Unifi switch)

[whydoentitwork]

Hello all!

After 3 days of digging, I decided to ask directly:

My proxmox host is connected to a Unifi switch port with a native VLAN 1610, also allowing tagged VLAN traffic for IDs 3510, 1620, ...
That means the proxmox host is automatically in the 1610 VLAN.

Setting up single-NIC VMs has been a breeze:
I configured vmbr0 as VLAN-aware and added a VLAN tag (3510 or 1620) to the VM's NIC to put the VM into the respective network.
Or I left the VLAN tag in the NIC settings blank to automatically put the VM into the native 1610 network.

For the last three days I tried a slightly more complicated setup:

Background:
I want to run a DNS server (bind9 docker container in a proxmox VM), which listens on all networks (1610, 1620, 3510, ...) and manages both my local DNS zone resolution and forwarding.
I do not want to host multiple DNS servers in every VLAN network and I also cannot use my firewall's DNS capabilities.
I need the DNS forwarding logic to go via the same network interface that the DNS request came from (eg. client from 1610 VLAN making DNS request -> DNS server forwarding via 1610 VLAN to upstream DNS servers), because every network has different properties (like privacy VPN).
To that end I came up with a proxmox VM that is attached to multiple NICs: one untagged one (which should automatically be tagged with 1610 by the Unifi Switch port), one tagged with 3510, one tagged with 1620, ...
Then, in this VM I would like to bind the DNS server to all those NICs' interfaces and serve requests.

My first question: Is this a stupid approach and is there an obviously better way?

For troubleshooting I scaled the setup down to a VM having only two NICs: one VLAN-tagged NIC (3510) one and one untagged one (whose traffic should be VLAN 1610 at the switch port).
Here I realized that the tagged NIC did not work as expected.
While a VM with two tagged NICs (3510 and 1620) could send and receive traffic via both interfaces, the VM with one untagged and one tagged NIC could not properly use the tagged NIC.
Using tcpdump inside the VM showed incoming ARP requests on the tagged NIC, but I couldn't send out anything, not even ping the gateway.
Each NIC did receive a valid and correct DHCP lease (which I configured statically on my home firewall), however.

I am very new to proxmox, so am I missing something obvious?

 

[ce3rd]

hi, should work like you did it -> i mean vmbr0 vlan aware and then create extra tagged pve-nics for the VM.
another option is 1 vlan aware bridge for the VM at PVE side and tag the vlan interfaces inside the VM...

 

[whydoentitwork]

Yes, I mean connecting one VLAN-tagged NIC to a VM works as expected, the VM is then in the respective VLAN.
Connecting one untagged NIC to a VM also works, the VM is then in the native 1610 VLAN, same as the proxmox host.
What also works is connecting multiple VLAN-tagged NICs to the same VM (in this case I don't understand the logic by which the default gateway is chosen, but that can be fixed).

What doesn't work is connecting both a VLAN-tagged NIC and an untagged NIC to the same VM.

I will try setting it up today again and report any insights.
I really need a working multi-subnet DNS server in order to combine per-subnet forwarding (to prevent DNS leaks in privacy VPN) with local DNS names.
My plan is to then set up local TLS for my DNS zone (whose domain I own) without having to import self-signed certificates on all devices.

EDIT:

Maybe relevant to know: I have even more VLANs than the specified ones above and I am trying to use Alma Linux as guest OS.
Might it be a problem where Alma Linux cannot handle ca. 6 VLAN-backed NICs and one untagged NIC (so total 7 NICs)?
If my understanding is correct, the guest shouldn't actually know about the VLAN tags, it just sees different network interfaces, the tagging is done by the proxmox host.
Surely, for a production Linux OS like Alma Linux 7 NICs shouldn't cause any issues, right?

EDIT 2:

For troubleshooting I rebooted the proxmox host - no change.

I also set up a new Alma Linux VM with the default GUI packages (no minimal server install).
I connected two tagged NICs.
Alma linux Gnome GUI settings show both interfaces, both have a valid DHCP lease.
When I disable one of them in the GUI settings the other one does not automatically get used - is this normal?
For some reason I cannot even get this setup to work anymore - only one interface works, using `ip a` they look similar (both UP, both valid DHCP lease etc.), but I can only get traffic through one.
I also tried using `nmcli con mod <INTERFACE1> ipv4.never-default no` on the non-functional interface and `nmcli con mod <INTERFACE2> ipv4.never-default yes` on the other one, teared both down and up again (`nmcli con down` / `nmcli con up`) - no success.

EDIT 3:

I removed the functional NIC from the Alma Linux VM (VLAN tag 815) and restarted the VM.
Still no outgoing network traffic (pinging the gateway), but tcpdump inside the guest showed that incoming ping requests (sent from my admin machine to the VM's IP) did come in and actually also got a response.

Doing a packet capture on my router / firewall for the VLAN subnet the VM is on did not show any ICMP echo requests being sent from the guest.
Instead, occasionally an ARP request came from it:

<TIME> ARP, Request who-has 192.168.16.1 tell 192.168.16.53, length 42
<TIME> ARP, Reply 192.168.16.1 is-at <ROUTER_MAC>, length 28

A packet capture on the non-functional interface using wireshark from within the guest shows my ARP requests / responses and my ping echo requests sent from the guest, but no ICMP echo responses.

A packet capture on the proxmox host tap interface corresponding to the faulty VM NIC did also show the ARP requests / responses and my ping echo requests sent from the guest, but no ICMP echo responses.

It seems like the guest VM is able to communicate fine (traffic leaves the NIC and ARP requests get responses).
Also, the proxmox tap interface shows all packets as expected.
There are just no replies to ICMP requests / `curl`.
Only ARP requests / responses come through and those are also the only ones seen by my router.
The proxmox firewall and IP filter / MAC filter are disabled for the guest.

What am I missing?

EDIT 4:

I tried an isolated experiment:
- Alma Linux with two tagged NICs, one worked one did not
- disconnected working NIC
- other one started working
- reattached old NIC
- both worked

I'm sorry for the lengthy post but WHAT

EDIT 5:

Everything becomes incredibly inconsistent once two NICs are added.
Single-NIC VMs work as expected.

 

[ce3rd]

hi, just tried alma VM with 2 tagged nics - worked like it should alma -> pve host -> hp switch with tagged vlans -> another pve host -> vm

[root@localhost ~]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host noprefixroute
       valid_lft forever preferred_lft forever
2: ens18: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether bc:24:11:65:08:72 brd ff:ff:ff:ff:ff:ff
    altname enp0s18
    altname enxbc2411650872
    inet 192.168.131.102/24 brd 192.168.131.255 scope global dynamic noprefixroute ens18
       valid_lft 6831sec preferred_lft 6831sec
    inet6 fe80::be24:11ff:fe65:872/64 scope link noprefixroute
       valid_lft forever preferred_lft forever
3: ens19: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether bc:24:11:b1:c8:04 brd ff:ff:ff:ff:ff:ff
    altname enp0s19
    altname enxbc2411b1c804
    inet 9.13.14.195/29 scope global ens19
       valid_lft forever preferred_lft forever

vlan 3:
[root@localhost ~]# ping 192.168.131.1
PING 192.168.131.1 (192.168.131.1) 56(84) Bytes an Daten.
64 Bytes von 192.168.131.1: icmp_seq=1 ttl=64 Zeit=0.389 ms
64 Bytes von 192.168.131.1: icmp_seq=2 ttl=64 Zeit=0.553 ms
^C
--- 192.168.131.1 Ping-Statistiken ---
2 Pakete übertragen, 2 empfangen, 0% packet loss, time 1025ms
rtt min/avg/max/mdev = 0.389/0.471/0.553/0.082 ms

vlan 50
[root@localhost ~]# ping 9.13.14.193
PING 9.13.14.193 (9.13.14.193) 56(84) Bytes an Daten.
64 Bytes von 9.13.14.193: icmp_seq=1 ttl=64 Zeit=0.340 ms
64 Bytes von 9.13.14.193: icmp_seq=2 ttl=64 Zeit=0.363 ms
64 Bytes von 9.13.14.193: icmp_seq=3 ttl=64 Zeit=0.331 ms
^C
--- 9.13.14.193 Ping-Statistiken ---
3 Pakete übertragen, 3 empfangen, 0% packet loss, time 2065ms
rtt min/avg/max/mdev = 0.331/0.344/0.363/0.013 ms

it's a pve 9.1.5 and alma 10.1 VM - quick setup - worked out of the box - maybe you've got a switch problem ;-) or maybe a loop situation?

you can try without switch if you make 2 VMs on vmbr0 (vlan aware) with 2 nics on each machine and 2 vlans tagged you are NOT using at your switch, so that the vlans only exist on pve. if that works, your problem is outside PVE ;-)

 

[shanreich]

Can you post your network configuration on the affected host as well as the VM configuration of an affected VM? Preferably when the system is in the erroneous state:

cat /etc/network/interfaces
cat /etc/network/interfaces.d/sdn

ip a

qm config <VMID>

 

[whydoentitwork]

Can you post your network configuration on the affected host as well as the VM configuration of an affected VM? Preferably when the system is in the erroneous state:

cat /etc/network/interfaces
cat /etc/network/interfaces.d/sdn

ip a

qm config <VMID>

Yesterday, I changed my proxmox network configuration before I got your response.
I did this because I read online that it is not good practice to have a proxmox PVID that is not equal the native unifi swtich port's VLAN ID.
If I am understanding correctly, having the default proxmox vmbr0 PVID 1 and a unifi native switch port VLAN of 1610 effectively transparently translates proxmox's VLAN 1 to unifi's VLAN 1610, potentially causing security issues due to VLAN leakage.
Since yesterday my network configuration looks like this:

/etc/network/interfaces:

auto lo
iface lo inet loopback

iface nic0 inet manual

auto vmbr0
iface vmbr0 inet manual
    bridge-ports nic0
    bridge-stp off
    bridge-fd 0
    bridge-vlan-aware yes
    bridge-vids 2-4094
    bridge-pvid 1610

auto vmbr0.1610
iface vmbr0.1610 inet static
    address 172.16.10.99/24
    gateway 172.16.10.1

source /etc/network/interfaces.d/*

/etc/network/interfaces.d/sdn is empty.


I am unsure if this change is relevant to my issue.
Since the change, I had to edit untagged VM NICs to VLAN tag 1610 to make them get a DHCP lease on the correct VLAN (which I did not expect).
Before, my network configuration was basically default, with the singe exception that `bridge-vids 2-4094` and `bridge-vlan-aware yes` was added under the vmbr0 configuration (standard VLAN-aware vmbr0 settings).


Concerning the VMs, I set up three for testing: Two with two tagged VLAN NICs each and one with 7 tagged VLAN NICs, all Alma Linux; see below for configuration.
All can ping their respective interfaces' gateways successfully (`ping 192.168.15.1 -I<815_INTERFACE>`), when configuring my router firewall (pfSense) to allow such traffic.
What doesn't work is pinging a public IP from an arbitrary (otherwise functional) interface: `ping 8.8.8.8 -Iens1` doesn't work, eventhough my router firewall allows such traffic.
The two VMs with 2 NICs do succeed in sending outbound traffic (see below).
The VM with 7 NICs does seem to allow any outgoing traffic (see below).

I read somewhere that this behavior is expected and related to the standard linux routing logic of only having one upstream gateway.
But, considering linux' versatility, this must surely be possible, right (how?)?
When hosting a DNS server I need outgoing forwarder / recursive DNS requests (to public servers like 8.8.8.8 or 1.1.1.1) to go out the same NIC that the DNS request came in on, which doesn't sound too hard.


The VM with 7 NICs has the following configuration:

`ip a`:

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host noprefixroute 
       valid_lft forever preferred_lft forever
2: ens18: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether bc:24:11:39:9a:14 brd ff:ff:ff:ff:ff:ff
    altname enp6s18
    altname enxbc2411399a14
    inet 172.16.10.53/24 brd 172.16.10.255 scope global dynamic noprefixroute ens18
       valid_lft 5197sec preferred_lft 5197sec
    inet6 fe80::be24:11ff:fe39:9a14/64 scope link noprefixroute 
       valid_lft forever preferred_lft forever
3: ens19: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether bc:24:11:e4:2d:46 brd ff:ff:ff:ff:ff:ff
    altname enp6s19
    altname enxbc2411e42d46
    inet 192.168.16.53/24 brd 192.168.16.255 scope global dynamic noprefixroute ens19
       valid_lft 5197sec preferred_lft 5197sec
    inet6 fe80::be24:11ff:fee4:2d46/64 scope link noprefixroute 
       valid_lft forever preferred_lft forever
4: ens20: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether bc:24:11:04:fe:1a brd ff:ff:ff:ff:ff:ff
    altname enp6s20
    altname enxbc241104fe1a
    inet 192.168.20.53/24 brd 192.168.20.255 scope global dynamic noprefixroute ens20
       valid_lft 5197sec preferred_lft 5197sec
    inet6 fe80::be24:11ff:fe04:fe1a/64 scope link noprefixroute 
       valid_lft forever preferred_lft forever
5: ens21: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether bc:24:11:0e:73:e7 brd ff:ff:ff:ff:ff:ff
    altname enp6s21
    altname enxbc24110e73e7
    inet 192.168.10.53/24 brd 192.168.10.255 scope global dynamic noprefixroute ens21
       valid_lft 5197sec preferred_lft 5197sec
    inet6 fe80::be24:11ff:fe0e:73e7/64 scope link noprefixroute 
       valid_lft forever preferred_lft forever
6: ens22: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether bc:24:11:2f:6b:d1 brd ff:ff:ff:ff:ff:ff
    altname enp6s22
    altname enxbc24112f6bd1
    inet 192.168.15.53/24 brd 192.168.15.255 scope global dynamic noprefixroute ens22
       valid_lft 5197sec preferred_lft 5197sec
    inet6 fe80::be24:11ff:fe2f:6bd1/64 scope link noprefixroute 
       valid_lft forever preferred_lft forever
7: ens23: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether bc:24:11:16:a1:b9 brd ff:ff:ff:ff:ff:ff
    altname enp6s23
    altname enxbc241116a1b9
    inet 172.16.20.53/24 brd 172.16.20.255 scope global dynamic noprefixroute ens23
       valid_lft 5197sec preferred_lft 5197sec
    inet6 fe80::be24:11ff:fe16:a1b9/64 scope link noprefixroute 
       valid_lft forever preferred_lft forever
8: ens1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether bc:24:11:6f:bb:3e brd ff:ff:ff:ff:ff:ff
    altname enp7s1
    altname enxbc24116fbb3e
    inet 10.35.10.53/24 brd 10.35.10.255 scope global dynamic noprefixroute ens1
       valid_lft 5197sec preferred_lft 5197sec
    inet6 fe80::be24:11ff:fe6f:bb3e/64 scope link noprefixroute 
       valid_lft forever preferred_lft forever

Every NIC interface has gotten the correct DHCP lease.
My router firewall (pfSense) only shows ens1 (10.35.10.53) as up, maybe because of:

`ip route show default`:

default via 10.35.10.1 dev ens1 proto dhcp src 10.35.10.53 metric 100 
default via 172.16.10.1 dev ens18 proto dhcp src 172.16.10.53 metric 101 
default via 192.168.16.1 dev ens19 proto dhcp src 192.168.16.53 metric 102 
default via 192.168.20.1 dev ens20 proto dhcp src 192.168.20.53 metric 103 
default via 192.168.10.1 dev ens21 proto dhcp src 192.168.10.53 metric 104 
default via 192.168.15.1 dev ens22 proto dhcp src 192.168.15.53 metric 105 
default via 172.16.20.1 dev ens23 proto dhcp src 172.16.20.53 metric 106

Trying to access the internet through any interface:

for i in 1 {18..23}; do curl --connect-timeout 3 --verbose --interface ens$i 1.1.1.1; done

*   Trying 1.1.1.1:80...
* socket successfully bound to interface 'ens1'
* Connection timed out after 3002 milliseconds
* closing connection #0
curl: (28) Connection timed out after 3002 milliseconds
*   Trying 1.1.1.1:80...
* socket successfully bound to interface 'ens18'
* Connection timed out after 3002 milliseconds
* closing connection #0
curl: (28) Connection timed out after 3002 milliseconds
*   Trying 1.1.1.1:80...
* socket successfully bound to interface 'ens19'
* Connection timed out after 3002 milliseconds
* closing connection #0
curl: (28) Connection timed out after 3002 milliseconds
*   Trying 1.1.1.1:80...
* socket successfully bound to interface 'ens20'
* Connection timed out after 3002 milliseconds
* closing connection #0
curl: (28) Connection timed out after 3002 milliseconds
*   Trying 1.1.1.1:80...
* socket successfully bound to interface 'ens21'
* Connection timed out after 3002 milliseconds
* closing connection #0
curl: (28) Connection timed out after 3002 milliseconds
*   Trying 1.1.1.1:80...
* socket successfully bound to interface 'ens22'
* Connection timed out after 3002 milliseconds
* closing connection #0
curl: (28) Connection timed out after 3002 milliseconds
*   Trying 1.1.1.1:80...
* socket successfully bound to interface 'ens23'
* Connection timed out after 3002 milliseconds
* closing connection #0
curl: (28) Connection timed out after 3002 milliseconds

My router firewall does allow outgoing traffic from eg. 10.35.10.53.

VM config (`qm config 106` on proxmox host):

bios: ovmf
boot: order=scsi0;net0
cores: 1
cpu: host
efidisk0: local-zfs:vm-106-disk-0,efitype=4m,size=1M
machine: q35
memory: 2048
meta: creation-qemu=10.1.2,ctime=1772731561
name: DeezNatS
net0: virtio=bc:24:11:39:9a:14,bridge=vmbr0,firewall=1,tag=1610
net1: virtio=bc:24:11:e4:2d:46,bridge=vmbr0,firewall=1,tag=816
net2: virtio=bc:24:11:04:fe:1a,bridge=vmbr0,firewall=1,tag=820
net3: virtio=bc:24:11:0e:73:e7,bridge=vmbr0,firewall=1,tag=810
net4: virtio=bc:24:11:2f:6b:d1,bridge=vmbr0,firewall=1,tag=815
net5: virtio=bc:24:11:16:a1:b9,bridge=vmbr0,firewall=1,tag=1620
net6: virtio=bc:24:11:6f:bb:3e,bridge=vmbr0,firewall=1,tag=3510
numa: 1
ostype: l26
scsi0: local-zfs:vm-106-disk-1,iothread=1,size=10G
scsihw: virtio-scsi-single
smbios1: uuid=9ed50fd1-8f33-45fd-a196-9a84d2c56ef6
sockets: 1
vmgenid: 69e95e9b-81d7-41eb-bc30-0527edbf277c

One of the VMs with 2 NICs has the following configuration:

`ip a`:

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host noprefixroute 
       valid_lft forever preferred_lft forever
2: ens18: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether bc:24:11:cb:50:00 brd ff:ff:ff:ff:ff:ff
    altname enp6s18
    altname enxbc2411cb5000
    inet 192.168.15.107/24 brd 192.168.15.255 scope global dynamic noprefixroute ens18
       valid_lft 6079sec preferred_lft 6079sec
    inet6 fe80::be24:11ff:fecb:5000/64 scope link noprefixroute 
       valid_lft forever preferred_lft forever
3: ens19: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether bc:24:11:0e:73:32 brd ff:ff:ff:ff:ff:ff
    altname enp6s19
    altname enxbc24110e7332
    inet 192.168.16.101/24 brd 192.168.16.255 scope global dynamic noprefixroute ens19
       valid_lft 6079sec preferred_lft 6079sec
    inet6 fe80::be24:11ff:fe0e:7332/64 scope link noprefixroute 
       valid_lft forever preferred_lft forever

On this VM, my psSense also only shows ens18 (192.168.15.107) as up.

Note that the 7-NIC and the 2-NIC VMs both have an interface on 192.168.15.0/24, which is allowed all outbound traffic on my pfSense.
So in theory - if one can communicate, the other should, too, right?

`ip route show default`:

default via 192.168.15.1 dev ens18 proto dhcp src 192.168.15.107 metric 100
default via 192.168.16.1 dev ens19 proto dhcp src 192.168.16.101 metric 101

Trying to access the internet through any interface:

for i in ens18 ens19; do curl --connect-timeout 3 --verbose --interface $i 1.1.1.1; done

*   Trying 1.1.1.1:80...
* socket successfully bound to interface 'ens18'
* Connected to 1.1.1.1 (1.1.1.1) port 80
* using HTTP/1.x
> GET / HTTP/1.1
> Host: 1.1.1.1
> User-Agent: curl/8.12.1
> Accept: */*
> 
* Request completely sent off
< HTTP/1.1 301 Moved Permanently
< Server: cloudflare
< Date: Thu, 05 Mar 2026 18:38:28 GMT
< Content-Type: text/html
< Content-Length: 167
< Connection: keep-alive
< Location: https://1.1.1.1/
< CF-RAY: 9d7b3ce22a2e2657-FRA
< 
<html>
<head><title>301 Moved Permanently</title></head>
<body>
<center><h1>301 Moved Permanently</h1></center>
<hr><center>cloudflare</center>
</body>
</html>
* Connection #0 to host 1.1.1.1 left intact
*   Trying 1.1.1.1:80...
* socket successfully bound to interface 'ens19'
* Connection timed out after 3002 milliseconds
* closing connection #0
curl: (28) Connection timed out after 3002 milliseconds

VM config (`qm config 105` on proxmox host):

bios: ovmf
boot: order=scsi0;net0
cores: 1
cpu: host
efidisk0: local-zfs:vm-105-disk-0,efitype=4m,size=1M
machine: q35
memory: 2048
meta: creation-qemu=10.1.2,ctime=1772730688
name: Test2
net0: virtio=BC:24:11:CB:50:00,bridge=vmbr0,firewall=1,tag=815
net1: virtio=BC:24:11:0E:73:32,bridge=vmbr0,firewall=1,tag=816
numa: 1
ostype: l26
scsi0: local-zfs:vm-105-disk-1,iothread=1,size=10G
scsihw: virtio-scsi-single
smbios1: uuid=3a3817fa-2bca-4a52-a17e-e3ecce219b84
sockets: 1
vmgenid: d435e9cb-1321-4cd7-8377-9670b64d52d2

Explanation of subnets and their VLAN IDs:

3510: 10.35.10.0/24
1610: 172.16.10.0/24
1620: 172.16.20.0/24
810: 192.168.10.0/24
815: 192.168.15.0/24
816: 192.168.16.0/24
820: 192.168.20.0/24


Sorry for the lengthy response, I wanted to make sure to thoroughly include all relevant information.

The next best thing that would work for me (after gathering all data above) would be to set up one DNS server VM for every subnet I want to host one on, but this would incur major effort which shouldn't be required, I assume.
Also, this would of course cause chaos in the sense that every server would require similar configurations - configuration changes to DNS would then be needed to be applied accross multiple (at least seven - maybe eleven) VMs.

 

[whydoentitwork]

hi, just tried alma VM with 2 tagged nics - worked like it should alma -> pve host -> hp switch with tagged vlans -> another pve host -> vm

View attachment 96302

[root@localhost ~]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host noprefixroute
       valid_lft forever preferred_lft forever
2: ens18: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether bc:24:11:65:08:72 brd ff:ff:ff:ff:ff:ff
    altname enp0s18
    altname enxbc2411650872
    inet 192.168.131.102/24 brd 192.168.131.255 scope global dynamic noprefixroute ens18
       valid_lft 6831sec preferred_lft 6831sec
    inet6 fe80::be24:11ff:fe65:872/64 scope link noprefixroute
       valid_lft forever preferred_lft forever
3: ens19: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether bc:24:11:b1:c8:04 brd ff:ff:ff:ff:ff:ff
    altname enp0s19
    altname enxbc2411b1c804
    inet 9.13.14.195/29 scope global ens19
       valid_lft forever preferred_lft forever

vlan 3:
[root@localhost ~]# ping 192.168.131.1
PING 192.168.131.1 (192.168.131.1) 56(84) Bytes an Daten.
64 Bytes von 192.168.131.1: icmp_seq=1 ttl=64 Zeit=0.389 ms
64 Bytes von 192.168.131.1: icmp_seq=2 ttl=64 Zeit=0.553 ms
^C
--- 192.168.131.1 Ping-Statistiken ---
2 Pakete übertragen, 2 empfangen, 0% packet loss, time 1025ms
rtt min/avg/max/mdev = 0.389/0.471/0.553/0.082 ms

vlan 50
[root@localhost ~]# ping 9.13.14.193
PING 9.13.14.193 (9.13.14.193) 56(84) Bytes an Daten.
64 Bytes von 9.13.14.193: icmp_seq=1 ttl=64 Zeit=0.340 ms
64 Bytes von 9.13.14.193: icmp_seq=2 ttl=64 Zeit=0.363 ms
64 Bytes von 9.13.14.193: icmp_seq=3 ttl=64 Zeit=0.331 ms
^C
--- 9.13.14.193 Ping-Statistiken ---
3 Pakete übertragen, 3 empfangen, 0% packet loss, time 2065ms
rtt min/avg/max/mdev = 0.331/0.344/0.363/0.013 ms

it's a pve 9.1.5 and alma 10.1 VM - quick setup - worked out of the box - maybe you've got a switch problem ;-) or maybe a loop situation?

you can try without switch if you make 2 VMs on vmbr0 (vlan aware) with 2 nics on each machine and 2 vlans tagged you are NOT using at your switch, so that the vlans only exist on pve. if that works, your problem is outside PVE ;-)

I am not entirely sure what you mean, but here's my setup:

I set up two VMs with each one NIC with VLAN ID 11 and one NIC with VLAN ID 22.
VLAN IDs 11 and 22 are not used anywhere else in my network and thus should also not be allowed by the unifi switch trunk port.
Upon creation, both did not have any internet connection (as no gateway / DHCP server / DNS server was configured).

I then configured static IPs on both VMs:

VM1: 11 VLAN NIC: 192.168.11.100/24
VM1: 22 VLAN NIC: 192.168.22.100/24

VM2: 11 VLAN NIC: 192.168.11.101/24
VM2: 22 VLAN NIC: 192.168.22.101/24

I did this with the corresponding `sudo nmcli connection modify <INTERFACE> ipv4.addresses <CIDR_ADDR>`.

After that I could ping both VMs' IP addresses from the other VM (eg. VM1 could ping 192.168.11.101 and 192.168.22.101 and VM2 could ping 192.168.11.100 and 192.168.22.100).
No outbound traffic possible.

This is all expected and correct behavior, right?

 

[ce3rd]

This is all expected and correct behavior, right?

yeah - looks good...

the other stuff:
if you have pvid 1610 you don#t need vmbr0.1610 - because the default vlan of vmbr0 is then 1610 untagged...

default routes:
normally you only have one default route - other default routes are maybe for backup purposes.

so if you have: default via 10.35.10.1 dev ens1 proto dhcp src 10.35.10.53 metric 100 -> this is your primary default route used - others with metric 101,102,103... are "backup" and not used as long the one with metric 100 is present.

so if you can't go out to internet on 10.35.10.1 - curl will fail
---
in your second example: default via 192.168.15.1 dev ens18 proto dhcp src 192.168.15.107 metric 100
is the primary and you said there is internet on 192.168.15. net - so the 1st curl is sucessfull:

* Trying 1.1.1.1:80...
* socket successfully bound to interface 'ens18'
* Connected to 1.1.1.1 (1.1.1.1) port 80
* using HTTP/1.x
> GET / HTTP/1.1
> Host: 1.1.1.1
> User-Agent: curl/8.12.1
> Accept: */*
>
* Request completely sent off
< HTTP/1.1 301 Moved Permanently
< Server: cloudflare

 

[whydoentitwork]

if you have pvid 1610 you don#t need vmbr0.1610 - because the default vlan of vmbr0 is then 1610 untagged...

If I understand correctly, you suggest the following, right?

auto lo
iface lo inet loopback

iface nic0 inet manual

auto vmbr0
iface vmbr0 inet static
    address 172.16.10.99/24
    gateway 172.16.10.1
    bridge-ports nic0
    bridge-stp off
    bridge-fd 0
    bridge-vlan-aware yes
    bridge-vids 2-4094
    bridge-pvid 1610

source /etc/network/interfaces.d/*

 

[whydoentitwork]

in your second example: default via 192.168.15.1 dev ens18 proto dhcp src 192.168.15.107 metric 100
is the primary and you said there is internet on 192.168.15. net - so the 1st curl is sucessfull:

Thanks for clearing that up, wan't sure about this.

But then - when using `curl --interface <INTERFACE> 1.1.1.1`, does the command still use the default route, causing connectivity issues when the interface specified is non-default?
My understanding was that if I had an interface on the 815 (192.168.15.0/24) network, `curl --interface <INTERFACE> 1.1.1.1` would be able to go through that interface, "ignoring" the default route:

Note that the 7-NIC and the 2-NIC VMs both have an interface on 192.168.15.0/24, which is allowed all outbound traffic on my pfSense.

Is this understanding incorrect?
If so, maybe the proxmox forum is indeed not the right place to ask and I need to ask a linux networking guy, sorry XD.
But my question remains - how can I force any specific traffic (like `curl` or bind9 recursive DNS requests) to go out a specific non-default interface on demand (eg. without changing the default route configuration)?

 

[ce3rd]

I would say if you use curl with --interface you define the source interface and so the source IP. the default route remains the same and if the source IP is not in the same subnet, it can not reach the gateway and fails.

if you want logic like:
if source is a.a.a.x then use default route via a.a.a.1
you need policy-based routing ;-)

 

[ce3rd]

1 more -> normally 1 default route is enough
it is used to reach all "unknown" subnets (Internet for example ;-)) - all subnets where the machine has interfaces in are reached directly without using default route, so maybe for your usecase 1 default route is enough... and because the machine is as server, it is also a good idea to use static IPs...

 

[whydoentitwork]

I would say if you use curl with --interface you define the source interface and so the source IP. the default route remains the same and if the source IP is not in the same subnet, it can not reach the gateway and fails.

if you want logic like:
if source is a.a.a.x then use default route via a.a.a.1
you need policy-based routing ;-)

That cleared up a lot for me, I'll look into that.

 

[whydoentitwork]

1 more -> normally 1 default route is enough
it is used to reach all "unknown" subnets (Internet for example ;-)) - all subnets where the machine has interfaces in are reached directly without using default route, so maybe for your usecase 1 default route is enough... and because the machine is as server, it is also a good idea to use static IPs...

My setup is the following:

I have different subnets that are routed to the public internet via different privacy VPN locations, depending on the subnet.
If I set up a local DNS server and just use one default route for all DNS requests from all clients from all subnets, all recursive requests will be made through the same default route.
The default route potentially has a different VPN location than the client coming from a specific subnet.
This causes the client DNS request to leak through a different VPN location, making the traffic look very weird.

Also, if I just used one default route, I would have to pick my poison:
Either clients from subnets that have no VPN send their DNS through VPN or clients from subnets that do have a VPN send their traffic without VPN.

It's all not great - except if I sent the recursive DNS requests through the same interface that the DNS request came in on.

That's just to clarify the use case, thanks for the help.

 

[whydoentitwork]

I would say if you use curl with --interface you define the source interface and so the source IP. the default route remains the same and if the source IP is not in the same subnet, it can not reach the gateway and fails.

if you want logic like:
if source is a.a.a.x then use default route via a.a.a.1
you need policy-based routing ;-)

Ok, after researching network configurations for the last three hours, I configured the server using nmcli and ip.
It actually works as expected now.
The issue was (as to be expected in hindsight) neither proxmox not alma linux - it was me and my missing knowledge about linux networking.
I learned a lot though, thanks for your patience!

 

[whydoentitwork]

If I understand correctly, you suggest the following, right?

Just to come back to this suggestion, this is not really related to the previous issue (hence the thread being marked as solved):

I did the following on my proxmox host:

( sleep 180; ifdown -a; ifup -a -i /etc/network/interfaces.bak; ) &

Then I backed up my interfaces file to the above location and then edited the original to:

auto lo
iface lo inet loopback

iface nic0 inet manual

auto vmbr0
iface vmbr0 inet static
    address 172.16.10.99/24
    gateway 172.16.10.1
    bridge-ports nic0
    bridge-stp off
    bridge-fd 0
    bridge-vlan-aware yes
    bridge-vids 2-4094
    bridge-pvid 1610

source /etc/network/interfaces.d/*

Running `ifreload -a` after editing the file broke the ssh session and luckily I had run the restore command before.
After restoration, my VMs were confused and I needed to restart them to make everything work again.

I then proceeded to restore /etc/network/interfaces from the .bak file.

Apparently the above configuration did not result in my proxmox host being reachable via the specified IP.

What happened?


I did this because, while almost everything worked as expected, I couldn't reach my proxmox host from another subnet and I thought that this might cause the issue.