OPNsense with VLANs on Proxmox

[silke]

I have a setup that should be quite common:
- Legacy net with unmanaged switch behind a Fritzbox router
- a new managed switch with 3 Ports used for the beginning:
1: Uplink to Fritzbox, VLAN 1 (should be untagged),
2: Port to enp1s0 of my Proxmox host (WAN) also VLAN 1,
3. Trunk- Port to enp2s0 of the Proxmox host (LAN) with VLANs 10, 20, 30, 99
- Proxmox with two bridges defined, one for WAN, one for LAN - LAN is vlan-aware
- OPNsense as VM with two network cards pointing to the two bridges
- Config of OPNsense which should be correct for WAN and the VLAN-Interfaces
- setup DHCP for VLAN99 in OPNsense
- Test-VM with NIC on the VLAN-aware bridge and Tag 99

The Test-VM gets an IP from OPNsense and can ping every interface of the OPNsense VM but and here comes the problem: it has no internet. Ping to 8.8.8.8 (Google-DNS) or 192.168.178.1 (Fritzbox) do not come through. I tried with automatic NAT and with hybrid NAT and an explicit rule to allow anything. Even with deactivated firewall no internet.

Any ideas what might be going wrong here? Or is there a tutorial I can follow? I found lots of stuff how to install OPNsense on Proxmox and also lots on VLANs but nothing with the combination or only simple configs without real VLAN devices in OPNsense.

Here is my /etc/network/interfaces:

auto lo
iface lo inet loopback

iface enp1s0 inet manual

# WAN and Proxmox management:
auto vmbr0
iface vmbr0 inet static
    address 192.168.178.205/24
    gateway 192.168.178.1
    bridge-ports enp1s0
    bridge-stp off
    bridge-fd 0

iface enp2s0 inet manual
# LAN with VLANs
auto vmbr1
iface vmbr1 inet manual
    bridge-ports enp2s0
    bridge-stp off
    bridge-fd 0
    bridge-vlan-aware yes
    bridge-vids 10 20 30 99

and here the definition of the OPNsense NICs:

qm set 100 -net0 "virtio=BC:24:11:DB:85:73,bridge=vmbr1,queues=8,trunks=10;20;30;99"
qm set 100 -net1 "virtio=BC:24:11:30:E9:E9,bridge=vmbr0,queues=8"

I also tried it without the trunks parameter

Any ideas what I could try?

 

[Johannes S]

@meyergru published a tutorial in the OPNsense forum: https://forum.opnsense.org/index.php?topic=44159.0
You might also want to read his piece on running OPNsense behind a Fritzbox: https://forum.opnsense.org/index.php?topic=39556
The latter is in German so you might need to use something like deepl to translate it. The gist is that to have an actual benefit of OPNsense you most likely will need to buy a lot of other hardware (at least a WIFI access point, a VLAN-aware switch and (depending on your Fritzbox model/configuration DSL/fibrechannel modem) if you don't want to loose functionality which the Fritzbox already has.
May I ask why you want to virtualize OPNsense? In the worst case if your ProxmoxVE host is down everything behind the OPNsense will be down too. Not a big deal for VMs but if you want to run something like e.g. adguard or pi-hole on ProxmoxVE you might run into trouble with your flat mates/family members ("Did you mess with the Internet again").

 

[silke]

Thanks for the quick reply, I will read the tutorials, no problem with German, it is my native language, I posted here because there may be more readers able to help than in the German forum.
I want to virtualize OPNsense exactly to avoid problems with downtimes. A VM is very easy to backup and move to another host and I have spare hardware I can switch on, get the last backup and (hopefully) am back in business within very short time.
As for new hardware: I already got new hardware and want to replace my old VMware Workstation VMs with new VMs within a brand new Proxmox cluster behind a managed switch. OPNsense is on a separate machine and should manage the new hardware and VMs and separate the security domains I have identified. But for this I need VLANs.

 

[silke]

The mentioned posts in the OPNsense forum have no real advice on VLANs, they are just mentioned but no real HOWTO. But I will try to post my problem there, too.

 

[Johannes S]

I want to virtualize OPNsense exactly to avoid problems with downtimes. A VM is very easy to backup and move to another host and I have spare hardware I can switch on, get the last backup and (hopefully) am back in business within very short time.

For that usecase running another OPNsense instance on the spare Hardware in a carp HA configuration might be the better fit.

 

[meyergru]

When you read the Fritzbox guide thoroughly, it contains a "traps" section. Read it closely and you will most likely find what you have missed:

In a router-behind-router scenario like yours (which I always advise against, e.g. here, point 4), you must do one of two things to make the subnet behind the second router reachable:

a. You must create a route on the first router (in your case, Fritzbox) to the network(s) behind the second router (in your case, OpnSense). This is sometimes infeasible - on a Fritzbox, you can do this.
b. You can instead use outbound NAT on the second router.

Why is this neccessary? As explained in the Fritzbox guide, you need a back route for packets to reach the clients behind the second router. This must either be set in the first router or you must mask all IPs behind the OpnSense WAN IP, which the first router knows.

I do not give step-by-step instructions, but rather show what problems arise and the principles behind it. Matter-of-fact, especially with a firewall, a step-by-step guide would do more harm than good, because there is no such thing as a "standard" setup and if you do not understand what you are doing, you will likely not "make your network somehow more secure", but actually less so.

 

[silke]

Well, thanks I appreciate your guide, there is a lot to learn but in this case I found the problem and it was not a NAT issue. I did everything right, well almost
The issue was the firewall pass rule for the VLAN interface to the outside world. I had such a rule but with "WAN net" as destination what seemed reasonable but in fact it needed "any" plus of course a block rule to separate it from the other VLANs. Here a step by step instruction how to set up a VLAN interface in OPNsense had really helped. In every scenario where the VLAN should have internet access this step is necessary.

 

[meyergru]

That step is already covered even earlier, in the READ ME FIRST.... point 3. As you already found, there is lots to learn here. And you absolutely need to have the building blocks internalized before going to more complex scenarios.