pfSense and proxmox (cluster)

[ott]

Maybe these are simple questions. But I have not been able to find the answers. I am thinking of replacing my bare-metal pfSense with a virtual one. Seems easy enough: https://docs.netgate.com/pfsense/en/latest/recipes/virtualize-proxmox-ve.html . This way I can run a few lightweight apps on the same HW.

(New) server has 2 NICs, one is shared with mgmt, which I'll use for 'LAN' and the other for 'WAN'

- Will the proxmox GUI be available on the WAN NIC? If yes, how do I prevent this?
- What happens if the pfSense VM is down (DHCP, DNS, etc.)? Anything I should do to keep things running as smooth as possible while the pfSense VM is down?


Cluster and extra votes: I have another proxmox server running. While It may be best to keep the two nodes separate (?) If I add the new node to this one and make a 2-node cluster. Should I give new-node an extra vote? I assume if anything is down I need this one to get up and running first and not wait for the other node?

new-server has a E3-1270 v5 3.6GHz 2133MHz 4C/8T and 32GB ram

 

[SimonMcNair]

I am running pfsense on Proxmox. I do this to provide routing and firewall between my LAN and my lab. I would not advise on replacing your main house internet ingress with a virtualised pfsense. I would buy a hardware device which is running pfsense personally. Lots of rationale including:

Increased security footprint, vulnerabilities for a non hardened linux install, vm escape.
increased downtime. Most routers boot straight in to the OS rather than waiting for the host to boot
if your VM breaks your internet goes down. If your host breaks, your internet goes down.
You'd still need a back up in case your PC broke.

Just my 2p though

 

[ott]

I would not advise on replacing your main house internet ingress with a virtualised pfsense.

This is what want to do, or what I am 'investigating' and hope to do... If that makes sense. I run bare metal now. But am considering to run as a VM, since I happen to have new HW available.

I understand the concern regarding "increased security footprint". However; I may be fine with that. ref.:
https://serverfault.com/questions/338666/is-there-danger-to-virtualizing-a-router
The top answer is from Chris Buechler, who co-founded pfsense.

The arguments people generally have against that are security of the hypervisor itself, which history has pretty much proven isn't much of a concern. That could always change, but there haven't yet been any really significant recurring hypervisor security issues. Some people just refuse to trust it, for no good reason. It's not about attacking other hosts if someone owns the firewall, in that case it doesn't matter where it's running, and of all the things that are likely to get compromised, the firewall is WAY down the list unless you do something stupid like open its management to the entire Internet with the default password set. Those people have some irrational fear that there's going to be some magic "root ESX" packet sent in from the Internet through one of its bridged interfaces that's somehow going to do something to the hypervisor. That's extraordinarily unlikely, there are millions of more likely ways your network is going to get compromised.

Numerous production datacenters run pfSense in ESX, I've setup probably in excess of 100 myself alone. Our firewalls run in ESX. From all those experiences, the only couple slight drawbacks to virtualizing your firewalls are: 1) if your virtualization infrastructure goes down, you're not going to be able to get to it to troubleshoot if you aren't physically at that location (mostly applicable to colo datacenters). This should be very rare, especially if you have CARP deployed with one firewall per physical host. I do see scenarios on occasion where this happens though, and someone has to physically go to the location to see what's wrong with their hypervisor as their virtual firewall and only path in is down too. 2) More prone to configuration mistakes that could pose security issues. When you have a vswitch of unfiltered Internet traffic, and one or multiple of private network traffic, there are a few possibilities for getting unfiltered Internet traffic dropped into your private networks (potential impact of which would vary from one environment to another). They're very unlikely scenarios, but far more likely than making the same kind of screw up in an environment where the completely untrusted traffic is not connected in any fashion to internal hosts.

Neither of those should keep you from doing it - just be careful to avoid scenario 1 outages especially if this is sitting in a datacenter where you don't have ready physical access if you lose the firewall.

increased downtime. Most routers boot straight in to the OS rather than waiting for the host to boot
if your VM breaks your internet goes down. If your host breaks, your internet goes down.
You'd still need a back up in case your PC broke.

My current "bare-metal" pfsense takes a long time booting. If my bare-metal is broken then internet is down anyway and I have to bypass using the ISP router. Although, I am more likely to use phone as hot-spot if needed.

New HW should be much faster booting, including proxmox and pfSense-VM. But I want to make sure it boots.

I will take my chances, but I want to avoid configuration mistakes.

Can anyone confirm this reddit-statement?

Proxmox is only accessible on interfaces on which you assign it an IP. As long as you don't assign it an IP on your WAN bridge, you should be good.

-----
fotnote: If I get comfortable with this solution I want to (try) set up HA using a managed switch to create a WAN VLAN and have pfSense with CARP, or HA in proxmox. If time permits...

 

[shrdlicka]

Can anyone confirm this reddit-statement?
Proxmox is only accessible on interfaces on which you assign it an IP. As long as you don't assign it an IP on your WAN bridge, you should be good.

Per default Proxmox listens on all interfaces:
https://pve.proxmox.com/pve-docs/pveproxy.8.html

If you don't give an interface an IP it can't listen on it

 

[SimonMcNair]

If you use



Then it should not allow any traffic to get to your lan interface.

The best way to figure it out is to try it in isolation tbh. If you're not happy to try it then you probably, maybe, shouldn't do it ? You can always do a curl to your public ip and see if it's visible. Restrict it to https in the meantime, and harden your password while you test.

 

[SimonMcNair]

TBH I am so impressed with pfsense that I'm considering buying a baremetal NUC box for it to replace my ERX and modem. I would never use a pc for the reasons I gave as above, but also the rationale that PFsense won't really benefit from the extra power of the PC unless you're pushing over 100Mbps through it imo (bear in mind most embedded systems are optimised with hardware offload etc). I would always run DNS/DHCP/FIREWALL on a dedicated and stand alone box.

I wish you well though

 

[ott]

Per default Proxmox listens on all interfaces:
https://pve.proxmox.com/pve-docs/pveproxy.8.html

If you don't give an interface an IP it can't listen on it

Thanks! I am further toward understanding. I think.

So, as long as I use the "LAN" NIC while setting up proxmox, then that gets and IP, and is the only one which it listens?

What happens if I physically move the network cable to other NIC, wont it just use that and listen?

This is where I get confused, since NICs dont have IPs, only hosts(?). When pfSense WAN (which is connected to vmbrX) gets an IP from the ISP. How does that not translate to proxmox...

TBH I am so impressed with pfsense that I'm considering buying a baremetal NUC box for it to replace my ERX and modem. I would never use a pc for the reasons I gave as above, but also the rationale that PFsense won't really benefit from the extra power of the PC unless you're pushing over 100Mbps through it imo (bear in mind most embedded systems are optimised with hardware offload etc). I would always run DNS/DHCP/FIREWALL on a dedicated and stand alone box.

I wish you well though

Same. I have been using pfSense (bare-metal) for 5-6 years now on an old HP server (~2011 hardware). I have fiber to the wall, so 1 Gbit internet and the new server was free. So it's just a drop in replacement for what I already have.

 

[shrdlicka]

Thanks! I am further toward understanding. I think.

So, as long as I use the "LAN" NIC while setting up proxmox, then that gets and IP, and is the only one which it listens?

What happens if I physically move the network cable to other NIC, wont it just use that and listen?

This is where I get confused, since NICs dont have IPs, only hosts(?). When pfSense WAN (which is connected to vmbrX) gets an IP from the ISP. How does that not translate to proxmox...

Take a look at https://pve.proxmox.com/wiki/Network_Configuration
One solution would be to give the WAN NIC directly to the pfSense VM and set up a Bridge for the LAN NIC. There you can give Proxmox an IP as well as pfSense. The bridge will act like a network switch.

 

[StreetPiet]

Just my two cents...

I run Proxmox and OPNsense together with pihole on a Zotac NUC for several Years now in my home network and im absolutely happy with it.
There some are pros and cons in my opinion:

• pro: proxmox backup, restore and snapshots saves you a ton of time if the update of a vm goes wrong.
• pro: i have the essential internet services (firewall and adblocker) on one physical machine. i can build my flexible dream do-it-all-router
• pro: if the hardware dies, you can quickly install proxmox on a new machine, restore the vm's from backup, an you're online again
• con: forget CPU-intesive tasks as IDS on a small NUC. (but: who stops you from using stronger hardware? ;-)
• con: the whole setup is a bit more complicated than a bare metal OPNsense.

i want to introduce OPNsense to my company and i wonder if anyone in the forum has some thoughts or opinions about

• OPNsense on two dedicated bare-metal hosts as HA-Cluster versus
• two virtual OPNsense as HA-Cluster on a two-host Proxmox Cluster versus
• a single OPNsense on a two-host Proxmox Cluster