OPNsense VM Config

[spetrillo]

Hello all,

I have built an OPNsense VM on my Proxmox 8.2.7 server. The VM has a LAN and WAN side. I have also built a simple Ubuntu desktop VM, so I could configure the OPNsense VM via its GUI. I cannot ping the OPNsense VM from the Ubuntu desktop and vice versa. Both VMs are on the same vmbr/vlan on the Proxmox server. Both have manual IP configs. I even took it one more step and made the gateway of the Ubuntu desktop the IP of the OPNsense VM LAN side.

I don't know Proxmox as well as I do VMware. Does Proxmox have a virtual switch similar to VMware? Do I need to configure anything, so that both VMs can ping each other? Has anyone been able to get an OPNsense VM to work?

Thanks,
Steve

 

[itNGO]

Post VM-Config please...

 

[spetrillo]

Here you go...

A few questions:

1) In order to configure this vm via its GUI does my PC need to be directly connected to the Proxmox NIC for vlan 1(vmbr0) or can it be connected to a switch port configured for vlan 1, as Proxmox?
2) Why can I not ping the LAN side of the OPNsense VM? I see the IP listed in the ARP table of my current live OPNsense firewall, so I would think I could ping it, and thus connect to the LAN side IP of the OPNsense VM.

 

[guruevi]

A vmbr (bridge) is equivalent to a virtual switch. Is the bridge set up to be vlan-aware, by default it isn't and will drop tagged packets. Basically we need the network configuration on both sides and the Proxmox one too. Typically two machines on the same bridge should see each other, if you are tagging VLAN, both need to be tagged the same VLAN on the same bridge.

If you are changing the Proxmox network bridge configuration, you do need to restart the VMs to pick up the new config (as it is the 'virtual hardware' that is tagged with the VLAN) as well as the network configuration in Proxmox.

If you're coming from "outside" and connecting a physical network to the bridge, then your physical network also needs to support VLAN tagging (tags will get passed to your switch).

 

[spetrillo]

I have been using vlans for years, so yes I do understand what you are saying.

VMBR0/VMBR1/VMBR2 are vlan aware. VLANs 1, 20, 30 are on VMBR0, vlans 10, 12 are on VMBR1, and vlans 20, 25, 30 are on VMBR2. I had a virtual desktop PC on the same bridge/vlan as the LAN side of the OPNsense VM and I could not ping each other. I do not understand this.

 

[guruevi]

So you have the same VLAN tagged on 2 bridges? If a packet arrives for VLAN 20 or 30, which bridge would it go to?

 

[spetrillo]

I would expect the routing and ARP in the firewall to determine where packets go. VMBR0 is a dual role. It provides vlan 1 access for the PVE server itself, as well as the OPNsense vm. The only vlans defined are just for normal vlans, like other vms or containers.

This still does not answer why I cannot access the GUI of the OPNsense vm from another device on the same vlan/vmbr.

 

[guruevi]

I would expect the routing and ARP in the firewall to determine where packets go. VMBR0 is a dual role. It provides vlan 1 access for the PVE server itself, as well as the OPNsense vm. The only vlans defined are just for normal vlans, like other vms or containers.

This still does not answer why I cannot access the GUI of the OPNsense vm from another device on the same vlan/vmbr.

VLAN tagging happens before ARP/IP routing, each ethernet frame gets a tag, regardless of whether the payload of the frame contains ARP or IP or some other messages, you can't have traffic "jumping" bridges based on an ARP table, the ARP table is for each interface, not for the entire host. You would have to connect the VMBR to each other somehow, which then gets into a whole other monster of (R)STP, routing, gateways etc. By default there is no routing table between bridges (eg. iptables can do that)

As far as why OPNSense isn't responding, have you turned off the firewall, by default OPNsense will not set up random interfaces and will not respond on them, only the one designated LAN does, again, we need a bit more information on all the IP ranges, you're passing 7 or so interfaces to OPNSense, one of them is untagged and one uses the 'default VLAN', which interface is your WAN, which interface is your LAN, does the MAC address match that assumption on the OPNsense side, same for the guest which you've not shown yet, same for the VMBR configuration.

My suggestion: start simpler. Set up a "loose" VMBR, set up a simple 2-interface OPNsense, make sure you match the LAN and WAN side, set up a client, don't use any VLAN on the bridge, have OPNsense hand out IPs and see if it works. OPNsense is capable of handling VLAN by itself as well if that makes more sense.

 

[spetrillo]

OK here is more information...

VMBR0 is associated with vlan 1, which I use as my mgmt vlan. It is also the subnet for the LAN side of this OPNsense VM(192.168.1.2). I also created a simple Ubuntu desktop on the same VMBR0, with an IP of 192.168.1.100. In this use case I would expect to be able to ping the LAN side of the OPNsense VM from this Ubuntu desktop but I cannot.

At this point, and to your point to simplify this, I have two VMs on the same bridge, on the same PVE server. I cannot ping so not sure where I go with this.

 

[guruevi]

So in OPNSense, that MAC address is assigned to your LAN and OPNSense has been setup to allow management traffic. Can you see if any traffic is exiting from either host (use arp, brctl show and use tcpdump to see the traffic on the vmbr0.1)

 

[spetrillo]

Hold on...vmbr0.1 is ONLY for the mgmt IP of the PVE server. Here is my network config of the PVE server:

auto lo
iface lo inet loopback

iface eno1 inet manual

auto vmbr0
iface vmbr0 inet manual
bridge-ports eno1
bridge-stp off
bridge-fd 0
bridge-vlan-aware yes
bridge-vids 1 20 30

auto vmbr0.1
iface vmbr0.1 inet dhcp

iface enp1s0f0 inet manual

auto vmbr1
iface vmbr1 inet manual
bridge-ports enp1s0f0
bridge-stp off
bridge-fd 0
bridge-vlan-aware yes
bridge-vids 10 12

iface enp1s0f1 inet manual

auto vmbr2
iface vmbr2 inet manual
bridge-ports enp1s0f1
bridge-stp off
bridge-fd 0
bridge-vlan-aware yes
bridge-vids 20 25 30

iface enp1s0f2 inet manual

auto vmbr3
iface vmbr3 inet manual
bridge-ports enp1s0f2
bridge-stp off
bridge-fd 0

iface enp1s0f3 inet manual

auto vmbr4
iface vmbr4 inet manual
bridge-ports enp1s0f3
bridge-stp off
bridge-fd 0

 

[guruevi]

No, VMBR0 VLAN 1 (VMBR0.1) is also what you assigned to your VM(s). So when you listen to traffic for those 2 VMs, the traffic will be on VMBR0.1

Note that VLAN 1 is ALSO the default VLAN, the default VLAN is the VLAN that a port belongs to IF you don't assign it to a port (so by definition VMBR3 and 4, although that is not relevant to the functioning of eno1 or vmbr0)

 

[spetrillo]

VMBR3 goes to my ISP router, as a static IP in OPNsense. VMBR4 will be used in the future if/when I can get the OPNsense VM to work. I plan on converting the current physical OPNsense firewall to another PVE node, and then running HA firewall VMs. VMBR4 will connect the nodes together.

Now back to VMBR0.1...so are you saying that any VM that has VMBR0 with a vlan flag of 1 will automatically want to use DHCP in my setup? I thought this was only used for the mgmt IP of PVE.

 

[guruevi]

VMBR3 goes to my ISP router, as a static IP in OPNsense. VMBR4 will be used in the future if/when I can get the OPNsense VM to work. I plan on converting the current physical OPNsense firewall to another PVE node, and then running HA firewall VMs. VMBR4 will connect the nodes together.

Now back to VMBR0.1...so are you saying that any VM that has VMBR0 with a vlan flag of 1 will automatically want to use DHCP in my setup? I thought this was only used for the mgmt IP of PVE.

No, the vmbr0.1 configuration is for the interface on Proxmox (host-side). I'm just warning you, you may be (unknowingly) exposing traffic from the VMs on that interface and the traffic to VLAN 1 (the default VLAN) will get forwarded to the physical interface and whatever else is connected there, your Proxmox interface may get an IP from the OPNsense system, as will other systems on that (V)LAN.

Right now, if your config is the way you say it is, it "should" work, but again, I don't know what else is on the LAN, whether that is throwing interference somehow (conflicting IP, are you allowed to have multiple MAC on that switch port, do you have a smart switch that blocks DHCP "servers" like OPNSense etc ...) hence why I said, start simple, single bridge, no interfaces, does it work?

 

[spetrillo]

OK going back to basics...let me configure and I will show you what I have.

 

[spetrillo]

Ok I have reconfigured the networking. Here is the new config:

auto lo
iface lo inet loopback

iface eno1 inet manual

auto vmbr0
iface vmbr0 inet manual
bridge-ports eno1
bridge-stp off
bridge-fd 0
bridge-vlan-aware yes
bridge-vids 1

auto vmbr0.1
iface vmbr0.1 inet dhcp

iface enp1s0f0 inet manual

auto vmbr1
iface vmbr1 inet manual
bridge-ports enp1s0f0
bridge-stp off
bridge-fd 0
bridge-vlan-aware yes
bridge-vids 1 10 12

iface enp1s0f1 inet manual

auto vmbr2
iface vmbr2 inet manual
bridge-ports enp1s0f1
bridge-stp off
bridge-fd 0
bridge-vlan-aware yes
bridge-vids 20 25 30

iface enp1s0f2 inet manual

auto vmbr3
iface vmbr3 inet manual
bridge-ports enp1s0f2
bridge-stp off
bridge-fd 0

iface enp1s0f3 inet manual

auto vmbr4
iface vmbr4 inet manual
bridge-ports enp1s0f3
bridge-stp off
bridge-fd 0

I have left vmbr0 for vlan 1, so the PVE gets an IP. This is the onboard NIC of my device. VMBR1/VMBR2/VMBR3 will be used for the OPNsense VM only, with VMBR3 being the WAN side. A quick question...I have other VMs that need vlans 20 or 30. Since I want to leave VMBR2 dedicated to OPNsense can I just add them as vlans to VMBR0. Is this ok as a config? Would you make any additional changes to the config above?

If you are cool with this then I am going to build an OPNsense VM, with just a LAN and a WAN side. I still have my Ubuntu desktop vm. Should this vm go on VMBR1, since the OPNsense LAN side will be there?

 

[guruevi]

So I'm assuming VMBR3 "WAN" side means 'untrusted' (cable modem etc) - sure I would leave it in its own bridge as well (or if you're paranoid, simply pass through the interface completely to OPNSense without a bridge, that way Proxmox can't accidentally get an interface there). Note that some cable modems do not allow 'switches' (aka bridges), as each bridge has 2 MAC addresses (really 2 devices) - the one from the VM and the one from your physical interface. At least mine doesn't work if I do that.

For the VLAN question, it doesn't really matter where you put them, as long as every interface you want to talk to each other are also on the same bridge. Be aware that your switch and whatever else on those 4 "LAN" also will get copies of the tagged VLAN packets, if that is not desirable (or you have a simple non-VLAN-aware switch), it is not necessary to connect a bridge to a physical interface if it is purely to handle VM traffic - eg. in my case, I have a bridge to OPNsense that is purely virtual for 'untrusted' servers, you can then use the physical interfaces 'better' (eg. in a bond).

A bridge is just a switch, a VLAN-aware bridge is just a VLAN-aware switch. Think of the way the traffic would flow if you were to do it with an external switch. A VLAN just segments the traffic from each other, the bridge still sees "all" the traffic.

 

[spetrillo]

VMBR3 is connected to one of the 4 ports that is embedded in the ISP router. It is a static IP, so I will assign the right IP thru the OPNsense config. Yes it might make alot more sense to pass thru the port, so its only used by OPNsense. In fact since all 4 ports of the network card(VMBR1-4) I have in my PC are dedicated for OPNsense, would it make more sense to pass thru all of them to the OPNsense VM? I dont want to do SR-IOV, just simple passthru.

 

[guruevi]

Yes, you probably should pass them through directly then.

 

[spetrillo]

I tried to pass thru the 4 ports of the PCIe card but the PVE server hung up and died. I am going to try to pass thru the entire card and see if that changes anything.

Yes I checked that IOMMU was enabled and I blacklisted the card driver. Not sure why I cannot get simple pass thru to work.