Best way to virtualize pfSense on Proxmox

[bearhntr]

I am seeking some guidance - after 20+ YOUTUBESs and unknown other Google searches, I am lost. Do not get me wrong, I can get it to work - just not the way I want it to (or my vision of it to work).
Proxmox VE 8.1.3
Hardware: HP z240 SFF Workstation (Core i5-6500/64GB RAM/250GB SSD/1TB SSD/4-port Intel i350-T4V2 card - trying to decide if I am gonna put a WiFi card in)

This is a brand new setup - moving the pfSense off an HP T620+ with added 2-port card. Never used the on-board NIC on this setup. 2-port NIC card (Port 0 = WAN & Port 1 = LAN). From there into a Netgear ORBI (in AP mode only to give WiFi).

In old configuration everything worked - I made a backup of the configuration, installed pfSense on Promox and restored the configuration. Once I set the WAN and LAN to the new ports, it works just fine. Cable from LAN into ORBI, and everything as it was.

Here is my quandary...The on-board NIC was used during the installation of the Proxmox, and it was given a Static IP as per the installation guide. If I unplug that cable, I can no longer get to the dashboard, no matter what I do. As soon as I plug that back in - dashboard is accessible via Static address assigned at install.

I only have 1 network segment in my home (no VLANs). 10.9.28.0/24 pfSense is .254, ORBI AP is .1 and ORBI Satellite is .2 - All statically set....everything else is DHCP (many with reservations). Proxmox is .240 (I tried setting the on-board NIC to .200 - it took the change, and survived reboot...but that address does not ping or yield dashboard).

How do I set this up so that I do not need to keep the cable plugged into the on-board NIC, yet still have access to the Proxmox dashboard?

As I understand it - in order for any other VMs (currently only pfSense on here - until I figure this out) need to use the same vmbrXX that pfSense is using for LAN in order to get an address? Is this accurate? Here is my current Network Config in Proxmox (see image):



I am just stumped and seeking some guidance.

My old pfSense HP T620+ is unchanged, simply unplugged - I can put that back in place if I have to in order to reload stuff.

 

[Dunuin]

Don't use an IP twice and no IPs of the same subnet on different interfaces...this will screw up routing.
So remove the IP from eno1 and restart your network or reboot.

 

[DragonL80]

Don't use an IP twice and no IPs of the same subnet on different interfaces...this will screw up routing.
So remove the IP from eno1 and restart your network or reboot.

The Ip isnt on eno1 its on vmbr0. which is set not active. eno1 is also set not active.

What I would do is try removing stuff you dont need, why do you have a bridge thats deactive? Also Vmbr0 is not the onboard host, thats eno1, which is a port memeber of vmbr0.

As I understand it - in order for any other VMs (currently only pfSense on here - until I figure this out) need to use the same vmbrXX that pfSense is using for LAN in order to get an address? Is this accurate? Here is my current Network Config in Proxmox (see image):

Yes, you should have the nic's of the lan vm's on the vmbr41 bridge.

I only have 1 network segment in my home (no VLANs). 10.9.28.0/24 pfSense is .254, ORBI AP is .1 and ORBI Satellite is .2 - All statically set....everything else is DHCP (many with reservations). Proxmox is .240 (I tried setting the on-board NIC to .200 - it took the change, and survived reboot...but that address does not ping or yield dashboard).

It wont, because its actually on a completely different segmented network, the deactive bridge on vmbr0

if you want to change the IP to 200, delete the vmbr0 bridge and change the vmbr41 bridge to 200.

if you want to use the onboard port for the lan connection, add eno1 to the vmbr41 bridge.

Edit: Misread first bridge, its vmbr0 not 1, thanks Dunuin

 

[Dunuin]

The Ip isnt on eno1 its on vmbr1. which is set not active. eno1 is also set not active.

Sorry. Yes, IP should be removed from vmbr0 which is attached to eno1.

 

[bearhntr]

The Ip isnt on eno1 its on vmbr0. which is set not active. eno1 is also set not active.

What I would do is try removing stuff you dont need, why do you have a bridge thats deactive? Also Vmbr0 is not the onboard host, thats eno1, which is a port memeber of vmbr0.

Yes, you should have the nic's of the lan vm's on the vmbr41 bridge.


It wont, because its actually on a completely different segmented network, the deactive bridge on vmbr0

if you want to change the IP to 200, delete the vmbr0 bridge and change the vmbr41 bridge to 200.

if you want to use the onboard port for the lan connection, add eno1 to the vmbr41 bridge.

Edit: Misread first bridge, its vmbr0 not 1, thanks Dunuin

Those addresses are there - because I was trying to get it to work.

I basically DO NOT want to use the on-board NIC -- as it is Broadcom, and I have heard horror stories of those cards and Linux

I want to use Port 0 on the Intel i350-T4v2 card as WAN (from ISP Modem), and Port 1 as LAN (to the ORBI (AP) which gives me WiFi and Wired stuff. I kinda have it that way now. My issue is that if I unplug the cable (which comes out of one of the other ports (there is a WAN port on the ORBI and there is a cable from Port 1 to there) back into the on-board NIC on the host) I will not be able to get to the host dashboard - this machine will eventually be headless (no monitor, no keyboard/mouse).

I am willing to reload the whole thing, as I have learned how to make backups of VMs .... lol (I was an ESXi man for years). pfSense is the only thing on there now anyway.

 

[DragonL80]

Okay, well as you have it configured you have port 1 as the lan. you have that in a seperate bridge. vmbr41. You have the wan side as vmbr40.
This is a snapshot of my config, I have two vm's on it underneath the pfsense on the lan side that I then use pfsense to port forward from virtual ip's that I get in the subnet from my host. They have no direct ipv4 exposure as they use private ip subnets for an internal lan.


(Caveat about this picture. there is nothing actually plugged in to enp9s0 on my box. Its got one connection. To the internet. On ens9. i dont really need enp9s0 to be a port member. Nothing will ever be plugged in physically here. its only accessible from the lan side of pfsense. )
Your proxmox does not have an ip on the wan side of your pfsense. You have one on the lan side, meaning you'd have to connect to a port thats in the vmbr41 bridge. I dont imagine you'd have both the lan and the wan of your pfsense router in the same subnet..

If thats what you're doing, thats not gonna work..

Im not even sure why you would want pfsense in this case as your entire network has no internet exposure. unless its just a home project?
You could just set them to DHCP assigned on the wan bridge and be able to access them directly. You would need a seperate subnet for the lan side. controlled by your pfsense. so you could us 172.16.32.2/24 and assign your pfsense lan 172.16.32.1/24

So, the short answer is.
Assign your ip to vmbr40 for the proxmox server, make sure the ip and gateway are deleted from 41. move that info 40. Remove the ip and cable from the onboard port. configure a seperate subnet for 41.

At that point if you need to get to pfsense, you'll either need to reverse ssh tunnel and use something like vncserver and then start a virtual X session where you could use a browser on the subnet. No, you'd still need an IP on the lan side on the proxmox interface for that to work..... (or the text console on the pfsense from proxmox if you can make the changes to the firewall from the commandline.. I know there is a really easy guide on netgate's forums on what the cli commands are to allow pfsense admin access from the wan. you'll need to make sure that the admin login is available on wan as well. that would still leave your vm's needing to be in a seperate subnet. Otherwise whats the point of pfsense?

Edit: re-reading your last post I missed something else, if your ISP/Modem is giving you a nat'd network, which it may be if its in "Gateway" mode. You would need to be assigned multiple real world ip's. And still need a private lan subnet. If you receive a private nework from your isp, you can still use it locally by putting the address you get from your modem, into the wan of the pfsense (usually by dhcp). and then using a broadcast domain (and physical domain) seperated private network and address space. like the 172.16 I suggested. Because you have to keep in mind, if the way I think you're setting it up is how you're actually trying to set it up... the network that you want to use the vm's from, would have to be accessible only on the lan side of pfsense. making the pfsense wan, your actual lan, in terms of lan and wan from your ISP. If you go to icanhazip.com and its not in the same subnet as your computer's ip according to network properties, then you're on the lan side of an upstream gateway device. Thats what you're trying to create with pfsense being the gateway to your vm's. If your computer's ip is in the same subnet as what you're trying to access proxmox from, then that needs to be the wan side of pfsense, so it can have a seperate lan to talk to the vm's on. So be careful with which context you're considering lan and wan from. when it comes to proxmox and pfsense, you should be thinking about it as you're the provider (WAN) instead of the customer(LAN) this time.

 

[DragonL80]

Remove the ip and cable from the onboard port.

I guess for a way if that didnt work for maybe potentially other device or medium related issues, you could leave another different ip from the ip on 40, but in the same subnet, on the onboard port, and unplug the cable, you should then be able to plug in the onboard cable if your access to the proxmox dashboard quits.. because the only reason this config wont work, is possibly wrong sysctl tunables, or other upstream devices. (other than driver issues, and the like but unrelated software failure points)

 

[bearhntr]

Okay, well as you have it configured you have port 1 as the lan. you have that in a seperate bridge. vmbr41. You have the wan side as vmbr40.
This is a snapshot of my config, I have two vm's on it underneath the pfsense on the lan side that I then use pfsense to port forward from virtual ip's that I get in the subnet from my host. They have no direct ipv4 exposure as they use private ip subnets for an internal lan.

View attachment 59045
(Caveat about this picture. there is nothing actually plugged in to enp9s0 on my box. Its got one connection. To the internet. On ens9. i dont really need enp9s0 to be a port member. Nothing will ever be plugged in physically here. its only accessible from the lan side of pfsense. )
Your proxmox does not have an ip on the wan side of your pfsense. You have one on the lan side, meaning you'd have to connect to a port thats in the vmbr41 bridge. I dont imagine you'd have both the lan and the wan of your pfsense router in the same subnet..

If thats what you're doing, thats not gonna work..

Im not even sure why you would want pfsense in this case as your entire network has no internet exposure. unless its just a home project?
You could just set them to DHCP assigned on the wan bridge and be able to access them directly. You would need a seperate subnet for the lan side. controlled by your pfsense. so you could us 172.16.32.2/24 and assign your pfsense lan 172.16.32.1/24

So, the short answer is.
Assign your ip to vmbr40 for the proxmox server, make sure the ip and gateway are deleted from 41. move that info 40. Remove the ip and cable from the onboard port. configure a seperate subnet for 41.

At that point if you need to get to pfsense, you'll either need to reverse ssh tunnel and use something like vncserver and then start a virtual X session where you could use a browser on the subnet. No, you'd still need an IP on the lan side on the proxmox interface for that to work..... (or the text console on the pfsense from proxmox if you can make the changes to the firewall from the commandline.. I know there is a really easy guide on netgate's forums on what the cli commands are to allow pfsense admin access from the wan. you'll need to make sure that the admin login is available on wan as well. that would still leave your vm's needing to be in a seperate subnet. Otherwise whats the point of pfsense?

Edit: re-reading your last post I missed something else, if your ISP/Modem is giving you a nat'd network, which it may be if its in "Gateway" mode. You would need to be assigned multiple real world ip's. And still need a private lan subnet. If you receive a private nework from your isp, you can still use it locally by putting the address you get from your modem, into the wan of the pfsense (usually by dhcp). and then using a broadcast domain (and physical domain) seperated private network and address space. like the 172.16 I suggested. Because you have to keep in mind, if the way I think you're setting it up is how you're actually trying to set it up... the network that you want to use the vm's from, would have to be accessible only on the lan side of pfsense. making the pfsense wan, your actual lan, in terms of lan and wan from your ISP. If you go to icanhazip.com and its not in the same subnet as your computer's ip according to network properties, then you're on the lan side of an upstream gateway device. Thats what you're trying to create with pfsense being the gateway to your vm's. If your computer's ip is in the same subnet as what you're trying to access proxmox from, then that needs to be the wan side of pfsense, so it can have a seperate lan to talk to the vm's on. So be careful with which context you're considering lan and wan from. when it comes to proxmox and pfsense, you should be thinking about it as you're the provider (WAN) instead of the customer(LAN) this time.

This has me even more confused...

I had pfSense running on an HP T620+ ThinClient box with an added 2-port Intel card (on that box the onboard NIC was disabled in the BIOS - pfSense would not even see it).

I made a BACKUP of that configured environment and installed pfSense into Proxmox - after reviewing many videos and notes as how to best setup the VM (hardware wise) - as I could not find a script to do that one - there are lots of scripts out there to do other VMs/Containers, etc. Not for pfSense.

The installation said to create 2 bridges (I did that vmbr40 and 41 -- I gave them those names as a indicator port-0 and port-1 on the 4-port card). The onboard NIC already had a cable in it - from the ORBI which was getting its Internet access from the pfSense ThinClient (the way it has worked for 2+ years). When I installed Proxmox - I had the cable in the onboard NIC - maybe this was the issue. I dunno.

When pfSense was on the ThinClient what I am trying to replicate in Proxmox it was as follows:

HP T620+ (dual-port NIC installed, onboard NIC disabled)
Port 0 - attached to MODEM from ISP (I own my own modem)
Port 1 - attached to ORBI (WAN port - see image) I actually have a yellow cable in there:

Since pfSense is my Router/Firewall now - the ORBI is in AP mode to give me WiFi in the house (and there is a satellite in my office at the other end of my house which gives me 4 Ethernet ports and repeated WiFi in my office). pfSense WAN is set for DHCP/DHCP6 and it gets both addresses. LAN is set to Static 10.9.28.254 (and Track Interface (WAN) for IPv6...but I have not enabled the DHCPv6 Server and RA - yet) So it gets an IP (/64 from my ISP), but nothing is handed out to any of my devices for now.

This all worked. Now I want to make the pfSense virtualized....as I plan to put other VMs/Containers (HomeAssistant, MariaDB, Zigbee2mqtt, MQTT - and others). They each have their own web or IP methods of access (they will for now be all in the same 10.9.28.xxx/24 segment). But I will at some point create a separate IoT segment) I also need to be able to use the 10.9.28.xxx/24 for my personal computers and my work laptops (which have their own VPNs for my customers).

Almost all of this I understand beyond the setting up of Proxmox to not need a cable back into it from the ORBI to be able to get the dashboard...and to give the VMs (with pfSense as DHCP/DHCPv6/DNS/FW to the other VMs/containers I put on there).

IP Setup as it is now:

Proxmox dashboard - 10.9.28.240/24
pfSense dashboard - 10.9.28.254/24
ORBI AP dashboard - 10.9.28.1/24 Satellite is 10.9.28.2/24
SmartThings v3 HUB - 10.9.28.3/24
HomeAssistant (on another T620 ThinClient - to be virtualized at some point along with others from above) - 10.9.28.4/24 (it also has CloudFlared on it and working to give me access when not on my home network).

So basically I need to now how to get (even if I have to reload it all) - Proxmox box to only need 2 cables (WAN and LAN) and be able to access everything within it and provide the IP segment to my home. Where the box is sitting, I have a Windows 2019 Server box which I can use to do setup from start if I need to using a 4-port hub.

 

[bearhntr]

I also have just discovered (no idea when this started) - that when I try and PING any computer on my network from a Windows machine that is my primary computer -- I get error: "Ping request could not find host pfsense.bearcave-home.com. Please check the name and try again."

I also tried to do "ping -a 10.9.28.254" (my pfSense box) -- and I get this:

Pinging 10.9.28.254 with 32 bytes of data:
Reply from 10.9.28.254: bytes=32 time=2ms TTL=64
Reply from 10.9.28.254: bytes=32 time=3ms TTL=64
Reply from 10.9.28.254: bytes=32 time=2ms TTL=64
Reply from 10.9.28.254: bytes=32 time=4ms TTL=64

Ping statistics for 10.9.28.254:
Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),

It pings, but no name resolution either way. From pfSense - I can do a DNS lookup and all of them are found. This is too weird.

 

[bearhntr]

@DragonL80 and @Dunuin,

By Jeeves - I think we (you) got it. I think it is working -- not 100% tested yet, as I have yet to put another VM on there, and use vmbr41 as the NETWORK for it - gonna try that here in a bit.

I edited the vmbr0 - and set it as so:



and rebooted things. Then I simply DELETED and APPLIED Configuration (I have yet to go into the BIOS and disable the on-board, for now there is a plug in the port with a cut off red-cable) -- hence things look like this now.




I am able to access the Host Dashboard from my Office PC with is wired to the ORBI Satellite (which is wireless to the ORBI AP in the other end of the house - where the Proxmox box is located and the modem). The pfSense is working fine with the 10.9.28.254 address I set on the LAN setup (Static). I think that my brain was not grasping the 'bridge' thing. It was like...how can you set an address on VMBR41 and then turn around and give another address to a VM which is 'front-end' (DNS/DHCP/FW) of your network.

Much thanks for sticking with me. Now to figure out why DNS forward and backward lookup is not working on many machines. nslookup is not pulling things back properly. Oh, well - more fun to ensue.