Proxmox as host for main pfSense router

[toxic]

Hello everyone,
I'm looking to get a new firewall and I'm looking for advice on how to deal with certain aspects. Attached is a global view of what I want to do.

My current concern is how to get good throughput and still good isolation for the VMs I start on proxmox like (C) and (D) or others not in the graph, and the clients like (F) or (G).

Meaning F would get full 1gb/s to C all the while G gets 1Gb/s to D or another one similar. But C should not be able to talk to D without the traffic going through pfsense so I insure my fw rules apply.

I assume I will need openvswitch to handle all the virtual NICs in this picture, I have drawn a dedicated virtual NIC in the pfsense for each VM, but that would also mean a dedicated vswitch for each VM am I right?
Since I believe the virtual NIC has 10gb/s potential throughput, I could use a single Vswitch for all VMs and a single vNIC on the pfsense VM, but in that case can I prevent a VM to talk to other VMs on the vswitch without before having passed through pfsense? (ie. prevent direct C to D and forcing C to B to D)
Keeping in mind that the VMs(C, D, ...) and clients (F,G,...) should be on the same subnet, I see no solution with a single vswitch... Any downsides to having many vswitches ? ( I fear all openvswitch config is config file based and not GUI... I can handle it but would prefer a nice GUI as always )

I'm also a bit weary of running non critical VMs on the hardware that will host my main router and switch, mainly since I fear I will have to reboot the host proxmox to deal with an issue or config of some low-importance VM, putting the whole networking down when I do this.
Any reassurance on the main usual reasons to reboot proxmox host are welcome

All other advice on this setup are also welcome.

Just in case, the CPU of the host will be core i5 8th gen with 16GB RAM with 6 Intel 1Gb/s physical NICs, so I expect actually good performance and hopefully over 1Gb/s of routing performance. I actually think it is overkill for a router so that's why I'm thinking of hosting other services on this host.

Also, I'm thinking of not dedicating any physical NIC to proxmox and instead pass all 6 physical NICs to pfsense, only keeping a virtual NIC between pfsense and proxmox for proxmox management. Any advice on this? I fear if pfsense is stuck in a boot loop I would actually loose any management of proxmox besides plugging a screen and usb keyboard in the host. But even with that, I would only get console access and no nice http GUI for proxmox, right? Any way to auto-detect that a VM like pfsense is acting up and pull the phisical NIC back to proxmox automatically to ensure management?

Thanks you in advance for your kind help!
Regards,
Toxic.

 

[mbosma]

I've had pretty good experiences with pfsense running as a vm in proxmox, have a few dozen running at the moment.

The only thing I've noticed (on pve5 though) is openvpn doesn't perform as wel as bare metal (even with cpu: host and aes-ni enabled in pfsense).
Recent testing showed on pve6 with the aes-ni passthrough showed a significant improvement with wireguard on opnsense, haven't had the time to research this for ovpn.
(this is only important if you do big numbers over ovpn like 200+mpbs).

Running some "non-critical" vm's probably won't hurt as long as you don't mess with the host.
Usually you won't have to reboot the host just from running some extra vm's as long as you're not doing some weird passthrough stuff or installing anything on the host itself.
I run a few test vm's on the same host as my production vm's as long as everything stays inside the vm, when I'm doing storage/pci-e passthrough or expect to use a lot of resources I switch over to another host which is normally powered down to save on electricity bills.

Passing through a physical nic will help because you can use hardware offloading.
You will have to create two nics in pfsense and bridge them, one (or more) for the physical nic and one for the virtual nic so your vm's can get access to your network as wel as your physical devices.
I strongly advise against having your pve management behing the firewall, this will create a deadlock someday.
Are you able to passthrough each nic individually? If not, maybe plug in an extra pci-e nic and dedicate that one to pve.

 

[toxic]

Thanks a lot for the very quick response!
I'll only receive the hardware in a few weeks but I expect to be able to pass each of the 6 physical NIC individually. Following your advice I will leave one NIC for pve, but I'll probably keep it unplugged, just in case of issue, and would still use pve through the firewall and virtual NIC for usual operation when everything is running well
With all 5 NICs already in there there is no more room for pcie expansion, but no matter, 5 NIC for the FW is plenty enough for my use.
Upstream connection is sadly way slower than 100mb/s so oVPN performance should not be an issue but thanks for the heads up!

Any hints on the VM always going through pfsense for inter-vm traffic and the underlying question of how many vswitches?
I ask that since some VM in there might get exposed on the internet and therefore I'd like to closely control what that VM is able to do on my network... Having it on the same network is very useful but keeping it from infecting others is also nice

It's going to be my first powerful firewall so I'll be trying out an IDS for the first time as well, my guess is that any traffic it doesn't see will not be analyzed or blocked, that will be fine for physical hardware, I can avoid using a physical switch for those who are sensitive, but for the VMs... I need to make sure pfsense sees and control their traffic

 

[chrisdc]

I'm sorry for hijacking the topic but @mbosma why is it not advised to put the pve management behind the firewall ?
(as I was planning on doing this as well in the future)

 

[toxic]

A wild guess : on startup pve could not start the VMs while waiting for an IP being assigned by a DHCP on the virtual interface, but cannot get it since it's the virtual pfsense that's my DHCP and it's not yet started... Snake eating it's own tail...deadlock.

 

[mbosma]

@chrisdc Apart from the dhcp problem which shouldn't be a problem if you're using a static ip on proxmox you won't be able to reach the proxmox webinterface when the pfsense vm doesn't route because of whatever issue might pop up.
I've had a few pfsense updates fail which required me to log into the console and give it a push that way.
Unles you're really familiar with the proxmox cli/api or setup the pfsense using a serial console you will never be able to see the console on the pfsense vm when you can't reach the proxmox webgui.

@toxic You could create a seperate linux or ovs bridge which doesn't have a physical nic and bind a virtual nic to your pfsense vm.
Downside to this setup is that hw offloading won't be possible because the
A way that will support hw offloading is hooking up the physical nic you would usually use for your proxmox management with a looping cable to a nic passed through to your pfsense.
This way you'll be able to unplug that cable and hook it up to your workstation or laptop when something goes haywire.
Create a bridge attached to that nic and give it an IP-address like you normally would on proxmox so you'll be able to use that bridge as lan or dmz network.

 

[toxic]

Thanks @mbosma. I hear your advice, I will keep a NIC dedicated to pve, probably won't loop it back to save on space on those "only" 6 NICs, and also to see how performance is with the virtual switch. I hope I'll almost never use pve after the initial setup, and I'll be a single user consuming only the web GUI of pve an probably no other services... So I think performance without HW offloading should still be fine, but I'll keep your advice if this way really struggles.

Back to my first question though, for the other VMs, to have them go through the pfsense, can you confirm I would need one vswitch/bridge for each VM ? I understand then that the same story about HW offloading happens again, but passing a physical NIC to each VM and looping it back into another NIC passed to pfsense will use up my NICs quite fast.
I could put all VMs on a vswitch and put 1 physical NIC on that switch, loop this one NIc to pfsense with a 10cm cable... But again, this way, VMs would be able to talk to each other without going through pfsense I fear...

Any other idea is welcome if none exist, I'll have to live with 1 vswitch and 2 vNIC to be setup for each VM. That'll motivate me to keep the number of VMs low

 

[mbosma]

I missed your requirement that vm's shouldn't be able to talk to eachother, I thought you just wanted them behind pfsense as firewall/gateway.
You could solve this by creating a new vswitch for each vm or using vlans.
I suggest only using vlans in the proxmox gui and not proxmox gui + pfsense gui as this combination won't work.
Create a new virtual nic for each vm you want to connect to pfsense with a tag matched with the tag on the vnic of the vm.
This method would also work if you loopback a cable from your pfsense to a vswitch, in this case you will have to use vlans from pfsense + proxmox.

 

[guletz]

I fear if pfsense is stuck in a boot loop I would actually loose any management of proxmox besides plugging a screen and usb keyboard in the host

Hi,

This is your weak point in your setup, because you put "all the egg in the same pfsense basket". If I woul be in your shoes, I would add a smart & router like Mikrotik(at least 5 ethernet ports + wifi) where I connect would connect:

1. all the wifi smart-Aphones+ SmartHome
2. Isp1+Isp2(ethernet) -> load balancing and/or fail-over
3. Laptop
4. 2x ethernet for PMX node(bonding)

Or maybe better with 1 Mikrotik with wifi(only for wifi) + other ethernet Mikrotik router with SFP/2.5 Gbits ethernet(for direct connect to PMX host)???

Note: I will try to add more ideas/details next day, I am now out of time ...
Nice setup and a very nice draw(with what appl?)!

 

[toxic]

Thanks @mbosma and @guletz .

Graph is made with draw.io website, free and nice to use I strongly advise to use it for your own needs

I get the idea of splitting things up in several hardware, and I'd actually say the same myself for a business, but it's my home setup, need to keep in mind the wife acceptance factor

This box is gonna replace my main router anyway, so I'm still trying to see if I can get it to do more using pve and not let all that power go to waste.

Given what I have heard here, I'm gonna give it a try like that, maybe even pass all NICs to pfsense and keep some usb NIC around for pve in case of troubles updating pfsense... It has HDMI output anyway and close to an existing monitor so I'll have options to fix it.
If that doesn't run as smoothly as I want, this box will most probably turn into a native pfsense box without pve or anything else running on it, because it's cheaper than a set of microtick router and switches anyway

I think I have enough info now to try it out, just need the box to arrive, hope it is not a dramless SSD in there, setup pve and hope for the best

Thanks again for your kind help!

Will post back here in about 3 weeks when I'm finished with it.

 

[mbosma]

Given what I have heard here, I'm gonna give it a try like that, maybe even pass all NICs to pfsense and keep some usb NIC around for pve in case of troubles updating pfsense... It has HDMI output anyway and close to an existing monitor so I'll have options to fix it.

If you know your way around the linux commandline you'll be able to set an ip-address on an interface, bring up that interface and reach proxmox that way.