Proxmox security with OPNsense and Pfsense

[proxi234]

Hi,

I am a newbie with proxmox and I want to set up my homelab. Initially I wanted to just run OPNsense on a machine I built myself. It has enough hardware capacity. My question is if I should go for OPNsense blank or with Proxmox VE and if it makes sense to run PFsense at the same time and how much security is affected by that or how do I setup it correctly, basically the hardware setup is very simple. I just want a hardware firewall before my workstation. I wanted to use OPNsense + Zenarmor + Proofpoint, but now I think how hard it would be to configure this elevated setup with Proxmox and additionally Pfsense and maybe even the Proxmox Mailserver on top as VM inside Proxmox VE. Is there any comprehensible guide for doing so securely? What do I have to consider? Any help appreciated.

Also which one do you judge better OPNsense or PFsense? I could also go for a subscription with PFsense Plus for example.

Thanks a lot!

 

[Deleted member 205422]

EDIT: Alright I got ahead of myself here, sorry, I had the VyOS (mentioned in the other thread) in my mind and thought you were considering installing both the PVE and routing on the same bare metal. The rest however remains relevant, as far as I am concerned, maybe with my emphasizing I would use *at least* one hardware router in my setup, so that when I need to do anything with that host, I do not have e.g. difficulty even downloading an ISO. You mentioned you want a workstation behind that OPNSense, I would normally have my VMs behind it, but of course you can always have a VLAN out of there and assigned an IP to workstation on that subnet.

Out of my own experience so far with PVE, I would avoid running anything alongside with it, keep it basic and simple so that I can reinstall the node itself. And that's me saying who actually installs is on top of Debian manually (for reasons I'd avoid going at lengths here).

The other part of your question, you might be interested in this post here we were having with someone who was going full circle on the PVE OPNSense setup.

https://forum.proxmox.com/threads/virtualising-opnsense-on-a-proxmox-cluster-–-chicken-vs-egg.135988/#post-602176

But I would just suggest to use one hardware router to most people, whichever they prefer. The MX, etc. I think you should be all fine.

 

[wbk]

I have been running OPNSenae in a BSD VM to full satisfaxtion. I run the ethernet ports bridged; passthrough (at least for WAN) would be an option in my 2-NIC microserver, but I have not found thebenefit of that.

Over at the lowendspirit.com forum someone just did an extensive write-up

As far as I read, there's not yet a "better" between OPNSense and PFSense; the machine I used initially for bare metal install did not support AES-NI (I think it was), which then became a mandatoy requirement for PFSense.

I went with OPNSense and never switched (only routed )

edit : Welcome to the forums, by the way!

 

[Deleted member 205422]

I run the ethernet ports bridged; passthrough (at least for WAN) would be an option in my 2-NIC microserver, but I have not found thebenefit of that.

Do you mind posting - if you followed the referenced guide - what # iptables -L ends up looking like? This is without clustering, correct?

 

[wbk]

Hi Esiy,

Trying not to hijack/derail the thread,

if you followed the referenced guide

no, sorry, I didn't follow the guide myself. I was pleasantly surprised to see Proxmox (and OPNSense) sticky'd there and the guy is usually thorough enough in his descriptions to merit sharing the guide.

My own OPNSense runs in a 'single instance' VM on a Proxmox node that is part of a 3-node cluster without HA. I didn't enable the firewall at Proxmox-level; if it is of any interest, I can share the iptables listing.

 

[Deleted member 205422]

Hi Esiy,

Trying not to hijack/derail the thread,

no, sorry, I didn't follow the guide myself. I was pleasantly surprised to see Proxmox (and OPNSense) sticky'd there and the guy is usually thorough enough in his descriptions to merit sharing the guide.

My own OPNSense runs in a 'single instance' VM on a Proxmox node that is part of a 3-node cluster without HA. I didn't enable the firewall at Proxmox-level; if it is of any interest, I can share the iptables listing.

Hey! Thanks. I would start a new thread if need be, but I think firewall settings in such scenario is relevant for the OP too (if I am told to go off I will though). I just wondered myself how would I run this safely or comfortably, when I have WAN traffic hitting the bridge where PVE binds some of its services too. I suppose you do not have 8006 pveproxy of then node accessible from the WAN.

 

[wbk]

I suppose you do not have 8006 pveproxy of then node accessible from the WAN.

No, the thought of exposing the management interface publicly has not crossed my mind

Wireguard runs on OPNSense's WAN interface though, through which the management interface might be reached 'on the road'

 

[Deleted member 205422]

No, the thought of exposing the management interface publicly has not crossed my mind

No but seriously, can you explain the network topology in your case? You just use Debian's iptables directly then? No hardware routers between WAN and the Microserver? BTW This is HP microserver gen8/10/10+ if I may ask?

 

[wbk]

the network topology in your case?

The fiber is connected to a media converter, the (copper) ethernet cable goes to port 2 (eno2) of the gen8 HP microserver (it does have some upgrade from the original config, though).

Proxmox (at host level) has no IP on eno2, and only IPv6 local link on bridge vmbr1; vmbr1 is connected to em1 on OPNSense, which is the 'physical' interface under the WAN interface.

The LAN side (via em0) of OPNSense is connected to vmbr0, which runs on top of eno1 of the HP machine. I connect to Proxmox via vmbr0 at host level.

I could try to draft a diagram, if it helps visualizing

hardware routers between WAN and the Microserver?

No, does such a thing as a hardware router still exist? I mean, it's all software nowadays, even if the firmware is not always as easily accessible as you'd wish.

 

[Deleted member 205422]

The fiber is connected to a media converter, the (copper) ethernet cable goes to port 2 (eno2) of the gen8 HP microserver (it does have some upgrade from the original config, though).

Proxmox (at host level) has no IP on eno2, and only IPv6 local link on bridge vmbr1; vmbr1 is connected to em1 on OPNSense, which is the 'physical' interface under the WAN interface.

The LAN side (via em0) of OPNSense is connected to vmbr0, which runs on top of eno1 of the HP machine. I connect to Proxmox via vmbr0 at host level.

I could try to draft a diagram, if it helps visualizing

I think I get it: WAN to eno2, PVE host binds on eno1 only, the OPNSense gets its WAN on em1 which is bridged with eno2 on vmbr1; you then have the OPNSense serve your LAN on the em0 bridged on vmbr0 with other VMs alongside with the PVE host. Correct?

The eno1 then continues to a switch to serve rest of LAN as well? No VLANs?

You just set the pveproxy bind via config, set up extra iptables on the host or ... this part you keep a mystery. I would be wondering after every APT upgrade whether my /etc/network/interfaces did not get messed up.

No, does such a thing as a hardware router still exist? I mean, it's all software nowadays, even if the firmware is not always as easily accessible as you'd wish.

No no, don't get me wrong, I am not one of those people who thinks firmware is special, I should have said "dedicated" (sometimes with HW offload too). What's the throughput of the micro gen8 on this setup in pure routing? CPU usage?

 

[wbk]

Correct?

Exactly

No VLANs?

Only on the WAN-side (my ISP requires a specific VLAN for internet access); I've been hesitant to use VLANs ever since I experimented with it for a while and I (and more importantly, the rest of the family) could not reach anything anymore until I managed to reset the switches

There is no additional/manual config involved, as far as I recall. It survived the upgrade from PVE7 to PVE8 (and other apt-actions) just fine.

throughput of the micro gen8 on this setup in pure routing? CPU usage?

I'm trying to stress it running multiple HD streams, mixed with speedtest; maybe torrents had been a better idea. My connection is officially 50/50, though it maxes out at 60 Mbit down and 100 Mbit up in speedtest.net , see attached screenshots (OPNSense on top , the corresponding node below). The graphs end at about the same moment, but the Proxmox graphs span a longer period. I think I redacted sensitive information, let me know if there seems something that should not be on a public forum

 

[Deleted member 205422]

Exactly


Only on the WAN-side (my ISP requires a specific VLAN for internet access); I've been hesitant to use VLANs ever since I experimented with it for a while and I (and more importantly, the rest of the family) could not reach anything anymore until I managed to reset the switches

There is no additional/manual config involved, as far as I recall. It survived the upgrade from PVE7 to PVE8 (and other apt-actions) just fine.

I'm trying to stress it running multiple HD streams, mixed with speedtest; maybe torrents had been a better idea. My connection is officially 50/50, though it maxes out at 60 Mbit down and 100 Mbit up in speedtest.net , see attached screenshots (OPNSense on top , the corresponding node below). The graphs end at about the same moment, but the Proxmox graphs span a longer period. I think I redacted sensitive information, let me know if there seems something that should not be on a public forum

Oh thanks for the extra reply! I have to go now, will have better look a bit later, but I am rather paranoid, would blank out the versions, SSL, MAC addr and anything resolvable at the least.

The thing is, when I check default iptables on a fresh PVE install, there's nothing. I have yet to explore how dependable their firewall solution is and I understand you have no IPs (other than link-local6) on that eno2, but you know never know what update might mess it up or dhclient gets buggy. I would put up at least a few extra rules there to avoid having to deal with anything assigned to eno2. Also what does the route table look like then on the micro... but this is just now flying through my head. Other than that I also forgot there were XEONs, I only have it as G16xxT and 8G RAM, so that makes a difference. The NICs are Gigabit, would be interesting to test it with iperf between eno2 and eno1 what that OPNSense routes on that!