[SOLVED] Proxmox and VLANs on different subnets

L

lucjuszmalfoj

New Member
Apr 14, 2024
6
0
1
Hi,

I've been struggling with this for the past few days and I would really appreciate your help here. I believe there might be something very obvious I'm missing here.

I have the following setup:
  1. Opnsense router.
  2. Aruba managed switch.
  3. 2 PVE instances, although currently I'm trying to just get one of them work.
What I want to do is I want to allow my PVE to host containers on different VLANs and different subnets.

OpnSense config:
  • LAN (VLAN 1) set up for 192.168.1.1/24
  • VLAN10 set up for 192.168.8.1/24.
    • Firewall rules are set to disallow contact with other private networks and allow DNS calls. Already tested this setup with a different VLAN I set up, albeit by connecting a device physically to the switch.
Aruba switch config has VLAN 10 configured as follows:
  1. Port 1: tagged (opnsense)
  2. Port 2: tagged (wap)
  3. Port 5: tagged (pve 1)
  4. Port 6: tagged (pve 2)
Proxmox:
  • I want it to be able to serve instances both on VLAN 1 and VLAN 10.
  • I set up my vmbr0 to 192.168.1.xx/16, Gateway 192.168.1.1, VLAN aware, Bridge ports enp1s0 (my eth interface), VLAN IDs 2-4094.
  • For the container, I set up eth0 as bridged via vmbr0, VLAN Tag 10, no firewall, 192.168.8.4/24, Gateway 192.168.8.1 (same as in opnsense).

I can see the container in opnsense's ARP table, I can even ping it from opnsense with no packets lost (though not Aruba – however, Aruba does not allow me to ping over VLAN10, only VLAN1). But I cannot reach any server from the container: neither network, nor the gateway.

I believe I'm missing something extremely obvious here – or possibly not? Would really appreciate your help.

I also tried creating VLAN interfaces separately, to no avail. However, from what I understand, bridge-aware route should just work.
 
I set up my vmbr0 to 192.168.1.xx/16, Gateway 192.168.1.1, VLAN aware, Bridge ports enp1s0 (my eth interface), VLAN IDs 2-4094.
Click to expand...
Check your subnet masks, for example the vmbr0 does not match your LAN mask on OpenSense.
With a /16 mask your servers will see 192.168.8.4 as a "local" ip address and just ARP for its MAC, rather than forwarding it to gateway 192.168.1.1
 
Thank you for the pointer.

My vmbr0 is now set to 192.168.1.32/24 and gateway 192.168.1.1, with eth0 set to vmbr0, VLAN 10, 192.168.8.4/24 and 192.168.8.1.

However, I do not believe anything changed: I can ping the ct from opnsense, but I cannot ping the gateway (should be 192.168.8.1, but of course 192.168.1.1 either), nor can I access the internet.
 
I tried running tcpdump while pinging wp.pl:

Code: 
2025-02-10 13:36:22.328604 IP 192.168.8.4.35329 > 192.168.1.1.53: 50932+ A? wp.pl. (23
)
2025-02-10 13:36:22.328632 IP 192.168.8.4.35329 > 192.168.1.1.53: 23039+ AAAA? wp.pl.
(23)
2025-02-10 13:36:27.558046 ARP, Request who-has 192.168.8.1 tell 192.168.8.4, length 2
8
2025-02-10 13:36:27.558234 ARP, Reply 192.168.8.1 is-at ac:16:2d:95:df:49, length 42

It tries going to port 53 on 192.168.1.1, which is where opnsense runs unbound DNS – is this correct? Shouldn't it go via 192.168.8.1?

The MAC is correct, though – it belongs to the router, so at least this is resolving correctly?
 
Last edited:
Seems to be correct:
1739192939460.png

First one is my ct – MAC address checks out.
Second is the opnsense MAC address, which I can verify by comparing with how it's mapped on VLAN1:
1739193012962.png

My Firewall policy in opnsense for this VLAN is as follows:
1739193129828.png

It's the same as my LAN, which is working fine:
1739193166806.png
 
I think it's the gateway configuration issue. I tried debugging DNS and this is what I got:


Code: 
root@unifi:~# dig NS wp.pl
;; communications error to 192.168.1.1#53: timed out
;; communications error to 192.168.1.1#53: timed out
;; communications error to 192.168.1.1#53: timed out

; <<>> DiG 9.18.28-1~deb12u2-Debian <<>> NS wp.pl
;; global options: +cmd
;; no servers could be reached

root@unifi:~# dig +short @192.168.8.1 NS wp.pl
ns1.wp.pl.
ns2.wp.pl.
ns1.task.gda.pl.

Based off this I believe that because the gateway set on vmbr0 is to 192.168.1.1, my ct tries to use that instead of 192.168.8.1.

EDIT
Which is absolutely correct. My ct can communicate with the outside world:
Code: 
root@unifi:~# ping 8.8.8.8 
PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.
64 bytes from 8.8.8.8: icmp_seq=1 ttl=120 time=7.29 ms
64 bytes from 8.8.8.8: icmp_seq=2 ttl=120 time=7.09 ms
64 bytes from 8.8.8.8: icmp_seq=3 ttl=120 time=7.39 ms
^C
--- 8.8.8.8 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2003ms
rtt min/avg/max/mdev = 7.093/7.257/7.388/0.122 ms
 
Last edited:
Resolved!

Thank you, deakus, you were right all along, it was a firewall issue.

It seems that OpnSense (at least in my config) has DNS server only serving on LAN (VLAN1). I had port 53 open between net and addresses in my network, but not towards LAN net, which is what broke the whole thing. Allowing traffic between these networks fixes the problem.

I believe I might've misapplied settings from the original opnsense VLAN setup guide I used, it does make sense to create pass rule that's specifically above the PrivateNetworks block that opens just this one port. I'll recheck, but seems like my bad.

Again, thank you for the help!
 
You must log in or register to reply here.
Top Bottom