Difficulty with resolving DNS on Proxmox and OpnSense

[oddRick]

Hello everyone,

I've been battling with this setup for quite some time now, and I really need some guidance. Here’s my current setup:

• ISP Router: In Bridge mode, linked to my Proxmox machine on the enp4s0 interface, marked as WAN in OPNsense.
• Proxmox: Hosting OPNsense as a VM.
• OPNsense: Managing the LAN network on the enp2s0 interface.

What I've Achieved So Far:

• I've successfully set up a DHCP server and a subnet for my LAN (on enp2s0) with the LAN interface set to 10.0.0.1/12.
• All my locally connected devices are recognized perfectly, which is great.

The Issue:

However, I can’t seem to get any internet access on my LAN network. Here’s what I’ve tried:

1. Unbound DNS configuration:
   • I’ve experimented with different Unbound configurations. At one point, I managed to get it working, but the response times were incredibly slow, and I had to refresh websites multiple times to get them to load.
   • I then tweaked some settings, and everything worked fine until I rebooted. After the restart, I lost all internet access again.
2. Current problems:
   • I’m unsure what DNS settings I should be using in Proxmox itself.
   • I’m unclear on how to properly configure DNS in OPNsense, especially since I plan to add Pi-hole/AdGuard later on.
   • Most critically, I’m completely stuck on how to restore internet access to my LAN network. I can't even ping 8.8.8.8 from my WAN interface anymore.

If anyone could provide some insights, particularly on the correct DNS settings for both Proxmox and OPNsense, and advice on how to regain internet access for my LAN, I’d greatly appreciate it.

root@gatekeeper:~# ip a

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000

    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00

    inet 127.0.0.1/8 scope host lo

       valid_lft forever preferred_lft forever

    inet6 ::1/128 scope host noprefixroute

       valid_lft forever preferred_lft forever

2: enp1s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master vmbr4 state UP group default qlen 1000

    link/ether 64:62:66:22:9f:7b brd ff:ff:ff:ff:ff:ff

3: enp2s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master vmbr1 state UP group default qlen 1000

    link/ether 64:62:66:22:9f:7c brd ff:ff:ff:ff:ff:ff

4: enp3s0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq master vmbr2 state DOWN group default qlen 1000

    link/ether 64:62:66:22:9f:7d brd ff:ff:ff:ff:ff:ff

5: enp4s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master vmbr0 state UP group default qlen 1000

    link/ether 64:62:66:22:9f:7e brd ff:ff:ff:ff:ff:ff

6: vmbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000

    link/ether 64:62:66:22:9f:7e brd ff:ff:ff:ff:ff:ff

    inet6 fdc2:7ec4:504d:6700:6662:66ff:fe22:9f7e/64 scope global dynamic mngtmpaddr

       valid_lft 7198sec preferred_lft 3598sec

    inet6 fe80::6662:66ff:fe22:9f7e/64 scope link

       valid_lft forever preferred_lft forever

7: vmbr2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000

    link/ether 64:62:66:22:9f:7d brd ff:ff:ff:ff:ff:ff

    inet6 fe80::6662:66ff:fe22:9f7d/64 scope link

       valid_lft forever preferred_lft forever

8: vmbr1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000

    link/ether 64:62:66:22:9f:7c brd ff:ff:ff:ff:ff:ff

    inet6 fdc1:45d:d77:2c1d:6662:66ff:fe22:9f7c/64 scope global dynamic mngtmpaddr

       valid_lft 1756sec preferred_lft 1756sec

    inet6 fe80::6662:66ff:fe22:9f7c/64 scope link

       valid_lft forever preferred_lft forever

9: vmbr4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000

    link/ether 64:62:66:22:9f:7b brd ff:ff:ff:ff:ff:ff

    inet 192.168.100.1/24 scope global vmbr4

       valid_lft forever preferred_lft forever

    inet6 fe80::6662:66ff:fe22:9f7b/64 scope link

       valid_lft forever preferred_lft forever











root@gatekeeper:~# ping -I enp4s0 8.8.8.8

ping: Warning: source address might be selected on device other than: enp4s0

PING 8.8.8.8 (8.8.8.8) from 192.168.100.1 enp4s0: 56(84) bytes of data.

^C

--- 8.8.8.8 ping statistics ---

5 packets transmitted, 0 received, 100% packet loss, time 4115ms

 

[sw-omit]

Since your proxmox does not have it's own IP on the WAN-side, it is "borrowing" the LAN-IP, which will just get dropped on the first point outside your network, as LAN-IP's should not be routed over WAN (and no-one would know where to send any packages "back" to.)

For proxmox to reach the internet, it needs to go through the OPNSense router which translates things from a LAN to a WAN IP, and when a packet comes back, back from WAN to LAN (somewhat simplified explanation)

Let's start with a few basics:
on proxmox, show the results of the following:

cat /etc/network/interfaces
qm config 99999 #replace the 99999 with the ID of your OPNSense router
ip r

Then, on your OPNSense, in the top-right you should have a searchbox, type in ping, and go to the ping-tool, try starting a ping to 8.8.8.8 from there, see if that at least works.

 

[oddRick]

thanks @sw-omit !
going with your guide I did set `iface vmbr0 inet dhcp` which assigned IP on my enp4s0 so I have ISP assigned IP on my WAN port. Still don't have internet though.

EDIT: i had to rervert to manula as WAN suppose to be exclusive for opnsense...

root@gatekeeper:~# cat /etc/network/interfaces
# network interface settings; autogenerated
# Please do NOT modify this file directly, unless you know what
# you're doing.
#
# If you want to manage parts of the network configuration manually,
# please utilize the 'source' or 'source-directory' directives to do
# so.
# PVE will preserve these directives, but will NOT read its network
# configuration from sourced files, so do not attempt to move any of
# the PVE managed interfaces into external files!

auto lo
iface lo inet loopback

iface enp1s0 inet manual

iface enp2s0 inet manual

iface enp3s0 inet manual

iface enp4s0 inet manual

auto vmbr0
iface vmbr0 inet dhcp
        bridge-ports enp4s0
        bridge-stp off
        bridge-fd 0
#OpnSense WAN

auto vmbr2
iface vmbr2 inet manual
        bridge-ports enp3s0
        bridge-stp off
        bridge-fd 0
#OpnSense LAN #2

auto vmbr1
iface vmbr1 inet manual
        bridge-ports enp2s0
        bridge-stp off
        bridge-fd 0
        bridge-vlan-aware yes
        bridge-vids 2-4094
#OpnSense LAN #1

auto vmbr4
iface vmbr4 inet static
        address 192.168.100.1/24
        bridge-ports enp1s0
        bridge-stp off
        bridge-fd 0
#MGMT

source /etc/network/interfaces.d/*

and

root@gatekeeper:~# qm config 100
agent: 1
balloon: 0
boot: order=scsi0;ide2
cores: 4
cpu: x86-64-v2-AES,flags=+aes
ide2: none,media=cdrom
memory: 12288
meta: creation-qemu=8.1.5,ctime=1715101831
name: OpnSense
net0: virtio=BC:24:11:5C:08:8F,bridge=vmbr0,firewall=1,queues=8
net1: virtio=BC:24:11:BB:97:9C,bridge=vmbr1,firewall=1,queues=8
net2: virtio=BC:24:11:5D:17:2B,bridge=vmbr2,firewall=1,queues=8
numa: 0
onboot: 1
ostype: other
scsi0: local-lvm:vm-100-disk-0,cache=writeback,discard=on,iothread=1,size=32G
scsihw: virtio-scsi-single
smbios1: uuid=a1699f8e-d7f7-47a5-8189-9753c35cd8fa
sockets: 1
vmgenid: 8b54538d-ee0f-4870-83e7-4af5f0bdcfd8

UPDATE: Few hours later...

I managed to get internet working on my LAN. I noticed that i had wrong vmbr assigned to WAN and LAN. That took me sooo long to fix.
Anyway, I have internet now, but still wondering about two items:

1. What DNS should I set in Proxmox settings?
2. How can I access my proxmox with opnsense from LAN?
3. Should I use AdGuard with Unbound or is there some sort of new cool config?

 

[oddRick]

for all the future explorers: i did follow https://windgate.net/setup-adguard-home-opnsense-adblocker/ and it worked like a charm

 

[sw-omit]

1. For DNS set either your router, 8.8.8.8 and/or 1.1.1.1
2. If you have your "lan" side connected to an internal switch and have an IP set up on it, you just need to go to that IP, port 8006
3. Sorry, no idea