[SOLVED] VMs cannot resolve domains while connected to virtualized OPNsense

[Soper]

Hello all,

I virtualize my firewall via OPNsense, and previously had it set up using Hyper-V core. I recently began the move to Proxmox, but have been struggling with a couple issues. Note, I am not necessarily the most experienced in networking, as most of my knowledge is self-taught.

For context:

I have a four port NIC, as well as two ports on the motherboard (for a total of six physical ethernet ports).

Three of the ports on the NIC are dedicated to my OPNsense VM via PCIe passthrough while the fourth is configured on a bridge (vmbr1). One of the ports on the motherboard is connected to a physical switch, which allows me to access Proxmox (10.75.0.50). The second port is currently unoccupied, but is supposed to be dedicated to a single VM (once I get to that point).

Here are screenshots of my network settings:


OPNsense hardware settings:


My VLANs:
1: 192.168.1.0/24; GW 192.168.1.1
5: 10.75.0.0/24; GW 10.75.0.254
10: 10.75.10.0/24; GW 10.75.10.254
100: 172.16.100.0/24; GW 172.16.100.254

I have a Raspberry Pi running pi-hole on VLAN 5, which serves as a DHCP server for VLAN 5 and a DNS server for my entire home network via IP 10.75.0.100.

Now here are where my issues start.

I created a new VM, and originally attached it to vmbr2, which is a bridge dedicated to just VLAN 10, which is configured within OPNsense. When starting up the VM, the VM is able to pull its IP from the DHCP server on OPNsense, but was unable to resolve any domains. My firewall rules are configured properly within OPNsense, as I am able to ping my pi-hole and other DNS servers such as 1.1.1.1 and 8.8.8.8. Here is its hardware settings for reference:

To make sure this is not just some weird configuration issue I overlooked, I instead connected the VM's NIC to vmbr1, which acts as a trunk for VLANs 100, 101 and 102. This is also connected to my physical switch. The VM, connected to VLAN 100, pulls an IP, but is still unable to resolve any DNS queries. Using a live Debian iso, I was able to confirm that this VM could connect to the OPNsense web UI, but cannot connect to anything else.

I confirmed that devices on VLAN 100 are able to connect out by connecting a physical computer to my physical switch. This computer pulls an IP from DHCP like the VM, but is able to resolve DNS, and access the general internet as normal.

I then connected the VM's NIC to vmbr0, which is supposed to be a port dedicated to Proxmox management connected to my physical switch, on the same subnet as my Raspberry Pi. On this bridge, the VM was able to pull an IP and make DNS queries from the local Raspberry Pi, and from 8.8.8.8, whereas it was previously unable to on vmbr1 and vmbr2.

This leads me to believe the issue lies somewhere within my network config for Proxmox and OPNsense, or am I wrong? I don't know how else to approach diagnosing and troubleshooting this issue, and would greatly appreciate any assistance.

To add, I noticed that Pi-Hole seems to not get any DNS requests from the VM. I was able to confirm that the packets are reaching the firewall, but it doesn't seem to be reaching the Pi, despite OPNsense seemingly passing them.

 

[Hannes Laimer]

Hey,

sounds like the PiHole ignores DNS queries. AFAIK by default the PiHole only responds to requests coming from a specific subnet/interface, this can be changed in the WebUI somewhere in the menu. If this is not the problem could you check if the queries reach the RPI with tcpdump port 53? You can use nslookup www.google.com 10.75.0.100 to manually send DNS queries for testing.

Edit: Settings > DNS > Interface listening behavior, for the config on the PiHole

 

[Soper]

Hey,

sounds like the PiHole ignores DNS queries. AFAIK by default the PiHole only responds to requests coming from a specific subnet/interface, this can be changed in the WebUI somewhere in the menu. If this is not the problem could you check if the queries reach the RPI with tcpdump port 53? You can use nslookup www.google.com 10.75.0.100 to manually send DNS queries for testing.

Edit: Settings > DNS > Interface listening behavior, for the config on the PiHole

Hi! Thanks and sorry about the late response.

My Pi-Hole is already configured:


Here is what I got using tcpdump on the rpi:

pi@raspberrypi:~ $ sudo tcpdump port 53
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), snapshot length 262144 bytes
20:25:32.592013 IP 172.16.100.108.50518 > 10.75.0.100.domain: 17608+ A? google.com. (28)
20:25:32.701691 IP 10.75.0.100.33050 > one.one.one.one.domain: 29138+ PTR? 108.100.16.172.in-addr.arpa. (45)
20:25:32.718618 IP one.one.one.one.domain > 10.75.0.100.33050: 29138 NXDomain 0/0/0 (45)
20:25:32.719279 IP 10.75.0.100.34280 > one.one.one.one.domain: 26513+ PTR? 100.0.75.10.in-addr.arpa. (42)
20:25:32.734133 IP one.one.one.one.domain > 10.75.0.100.34280: 26513 NXDomain 0/0/0 (42)
20:25:32.809504 IP 10.75.0.100.52669 > one.one.one.one.domain: 5639+ PTR? 1.1.1.1.in-addr.arpa. (38)
20:25:32.844436 IP one.one.one.one.domain > 10.75.0.100.52669: 5639 1/0/0 PTR one.one.one.one. (67)
20:25:37.595872 IP 172.16.100.108.50518 > 10.75.0.100.domain: 17608+ A? google.com. (28)
20:25:42.599946 IP 172.16.100.108.50518 > 10.75.0.100.domain: 17608+ A? google.com. (28)

I assume this means that the queries are reaching the rpi from the VM, but somehow aren't going back to the VM? I even used 1.1.1.1 as another DNS server to test:


To further test, I tested using another live Debian instance on a physical computer connected to the physical switch mentioned earlier. Like previously stated, this PC is able to pull an IP like its VM counterpart, but unlike the VM is able to resolve domains like normal:

user@debian:~$ ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: enx00051b94ed9d: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 00:05:1b:94:ed:9d brd ff:ff:ff:ff:ff:ff
    inet 172.16.100.101/24 brd 172.16.100.255 scope global dynamic noprefixroute enx00051b94ed9d
       valid_lft 83118sec preferred_lft 83118sec
    inet6 fe80::baf3:690e:183d:ee53/64 scope link noprefixroute
       valid_lft forever preferred_lft forever
user@debian:~$ nslookup google.com 10.75.0.100
Server:         10.75.0.100
Address:        10.75.0.100#53

Non-authoritative answer:
Name:   google.com
Address: 172.217.2.110
Name:   google.com
Address: 2607:f8b0:4004:83e::200e

tcpdump on the rpi during this request:

pi@raspberrypi:~ $ sudo tcpdump port 53
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), snapshot length 262144 bytes
20:42:48.821701 IP 172.16.100.101.36238 > 10.75.0.100.domain: 58260+ A? google.com. (28)
20:42:48.822711 IP 10.75.0.100.34072 > 1.0.0.1.domain: 51400+ A? google.com. (28)
20:42:48.846816 IP 1.0.0.1.domain > 10.75.0.100.34072: 51400 1/0/0 A 172.217.2.110 (44)
20:42:48.847489 IP 10.75.0.100.domain > 172.16.100.101.36238: 58260 1/0/0 A 172.217.2.110 (44)
20:42:48.849162 IP 172.16.100.101.34292 > 10.75.0.100.domain: 38518+ AAAA? google.com. (28)
20:42:48.849923 IP 10.75.0.100.41813 > 1.0.0.1.domain: 59256+ AAAA? google.com. (28)
20:42:48.871740 IP 10.75.0.100.49360 > one.one.one.one.domain: 4370+ PTR? 101.100.16.172.in-addr.arpa. (45)
20:42:48.873319 IP 1.0.0.1.domain > 10.75.0.100.41813: 59256 1/0/0 AAAA 2607:f8b0:4004:83e::200e (56)
20:42:48.874044 IP 10.75.0.100.domain > 172.16.100.101.34292: 38518 1/0/0 AAAA 2607:f8b0:4004:83e::200e (56)
20:42:48.894809 IP one.one.one.one.domain > 10.75.0.100.49360: 4370 NXDomain 0/0/0 (45)
20:42:48.895347 IP 10.75.0.100.36140 > one.one.one.one.domain: 32996+ PTR? 100.0.75.10.in-addr.arpa. (42)
20:42:48.918854 IP one.one.one.one.domain > 10.75.0.100.36140: 32996 NXDomain 0/0/0 (42)
20:42:48.919514 IP 10.75.0.100.40321 > one.one.one.one.domain: 60819+ PTR? 1.0.0.1.in-addr.arpa. (38)
20:42:53.924683 IP 10.75.0.100.40321 > one.one.one.one.domain: 60819+ PTR? 1.0.0.1.in-addr.arpa. (38)
20:42:58.930653 IP 10.75.0.100.59983 > one.one.one.one.domain: 52681+ PTR? 1.1.1.1.in-addr.arpa. (38)
20:42:58.954730 IP one.one.one.one.domain > 10.75.0.100.59983: 52681 1/0/0 PTR one.one.one.one. (67)
^C
16 packets captured
16 packets received by filter
0 packets dropped by kernel

The VM and the physical computer are able to see each other, as I am able to ssh into either one via the other.

 

[Soper]

A (rather late) update to this issue:

It seems like it had something to do with the way OPNsense was tagging VLAN traffic and the way Proxmox was interpreting it.

I ended up instead creating virtual NICs on the OPNsense VM, removing the VLAN interfaces on OPNsense, and tagging those on Proxmox instead, as so:


What confuses me however, is why my physical network was able to use the VLAN traffic while my Proxmox VMs were not able to.

Regardless, this issue was solved through a workaround.