OPNsense - weird behaviour

[snolli]

Hello!

I noticed a weird behaviour regarding firewall rules with my OPNsense VM, where i don't know what is causing it and wheter it's a problem or not.

I'm trying to setup a DMZ (virtually, with a bridge) in addition to LAN and WAN interfaces (both physical; 3 in total but one for the proxmox host itself), and followed tutorials found online.

At first i tried it with OPNsense installed bare metal (on a different computer with 4 physical NICs). After setting up the interface and DHCP in OPNsense, the only thing left to do ist setting a firewall rule allowing all traffic coming from the DMZ-Net with destination "any BUT LAN-Net", according to tutorials. It worked perfectly well and i had internet connectivity and could not access LAN or OPNsense itself.

But then with Proxmox it is somehow not working like that. I have to set an additional rule, allowing traffic from the DMZ-Net to the LAN-Address respectively "This Firewall".
I tried it with bridging two NICs into the OPNsense-VM and a i tried it with PCI passthrough, but same result. And i also tried to turn of the firewall-settings in proxmox for the OPNsense-VM and a Ubuntu-CT, which i bridged with the DMZ for testing. So somehow it needs a connection to 192.168.1.1 (OPNsense/LAN), which it doesn't need bare-metal.

I attached a picture summarizing my problem; (1) only working on bare-metal, but not on proxmox, (2) and (3) working on proxmox.

I hope somebody can clarify things for me, if i'm doing something wrong or this is normal?

Thanks and BR

 

[Ramalama]

That has somehow something todo with how the bridge works. It's in promiscous mode etc...
I don't know the reason exactly, but i have similar weird things with pf/opn/freebsd in general on proxmox.

You can try to route multicast through opnsense on proxmox, it simply doesn't work. Not even reflectors are working correctly.
While on bare metal this is all easy.

But this isn't only opensense for me, i have same issues with vyos on proxmox (vyos is linux based). However, opn/pfsense gots anyway buggy as hell in the last time. Trying to get rid of that crap. But this weird issues with routing have at least nothing with pf/opnsense todo...
It's simply somehow the weird behavior of the linux bridge...

Cheers

 

[toxic]

Not sure it applies to your config because I don't have a DMZ but I have something similar that is working fine for me : passing several NIC using oci passthrough to my opnSense VL as well as a virtual network card attached to vmbr0 like all my other VMs.
Then in OpnSense I create a bridge between vtnet0 and igb1-5 and it's working like all my VMs are on my LAN properly.
I even enabled pfilter within a bridge and can add FW rules between my LAN clients and my VMs. All VMs have internet access and access to my LAN working.
Maybe you have to add an outbound NAT rule for your LAN interface from your DMZ interface ?

 

[Ramalama]

I have multiple vlans, communicating with each other, outbound nat, is basically only needed for wan or for some networks which doesn't have routes for my network behind the gw.
Like if they get an source ip of 192.168.27.222 and don't know that network, so they reply to their default gw, which isn't the gw the packets comes from. So with nat, they see the source ip of the gw the packets comes from and reply to that gw...

However, that's not my problem, my problem is simply multicast, multicast isn't unicast, where you actually can make any permit or deny rules. Because it doesn't goes to the gw at all...
Basically services listens to 239.255.255.250 or 224.0... (with an udp port) and decides if they ignore or take that packets. (Multicast is layer 3 only)
To make this "routable" through a gw, you need something that listens to those addresses and mirrors it to the other network of your wish.
Basically this is pretty simple in theory.

However, multicast isn't a communication between 2 peers directly and isn't broadcast either (broadcast is only mac based layer 2), that's why there is nothing to route or firewall.
There are ways to route it with iptables etc, but that works more like mirroring.

Back to the problem, while opnsense/pfsense sees this kind of traffic behind the linux bridge and can even mirror it, it isn't working. Whyever, I don't know, trying to figure this out.
While on bare metal this crap works.
Looked into the packets with wireshark, they are correct. But the problem is, that there are some missing packets (looks for me very random) that the pf/opnsense doesn't even get behind the bridge.
And i can't find out why those packets doesn't reach the pfsense.

And well, if you pass through a complete nic, it works. But if you use sr-iov (virtualize physical nic into virtual functions) and pass those through, i get the exact same effect as with the linux bridge.
So this tells me, that either intels sr-iov and the linux bridge are both buggy, or they both work correct as they should.
I think the later is the case, but i can't find out why or with what this has todo.

Normally i would say, okay fuck it, i pass physical nics through and don't use sr-iov or virtio....
But this doesn't works, because i have to much separated networks. (That aren't all routable only with vlans).

Cheers

 

[snolli]

Thank you very much for your answers

I have decided not to virtualize OPNsense, as i had other problems with it too. And since it works like a charm on bare metal i will stick with it.