Proxmox gets IP from pfsense VM installed on proxmox host possible with 2 NICS?

[copetogo]

Hi Guys
So i want to set up a pbs server or backup using nfs of proxmox instance on my nas .My proxmox host has 2 NICS. Here is my network route

ISP router (192.168.5.1)> Proxmox Host (192.168.5.150-vmbr0)>PFsense VM (Wan interface -192.168.5.3 Lan Interface - 192.168.1.1-vmbr01)>NAS (192.168.1.69)
As you can see since nas is on different subnet to proxmox host i cannot do backups. now my question is how can i bring it to same subnet to NAS. if i manually change IP of Proxmox host to PFsense subnet it would break my Pfsense(i haven't tried it but just thinking it loud it cannot be done).

 

[ComputerWarlock]

You should be able to just set the IP and NIC over to the PFSense LAN side as well as point the gateway of proxmox to the LAN IP of PFSense.
Might need a reboot of proxmox host to take.

 

[copetogo]

You should be able to just set the IP and NIC over to the PFSense LAN side as well as point the gateway of proxmox to the LAN IP of PFSense.
Might need a reboot of proxmox host to take.

if i set the IP & gateway of proxmox host to pfsense would it not break IP resolution coming from the ISP to proxmox host, how will internet directed from router to proxmox to pfsense if i do that. i know i can try but scared i may be locked out of my proxmox host access

 

[ComputerWarlock]

When messing with IP settings it is ALWAYS a possibility to lock yourself out. You can however plug a keyboard and monitor into the host and have access that way. It's all command line but it is nice to get to know how to fix it when you have to.
The IP address you have in Proxmox is just for the GUI. What you can do is set the other NIC for an IP inside the network 192.168.1.x and have no gateway (You can only have one gateway for proxmox host) apply the settings then see if you can ping that IP from your PC on the same subnet 192.168.1.x if you can then try to browse to the IP and see if the GUI comes up. In the past I have had to edit some settings in the configuration with keyboard and monitor on host.

Is your goal to have PFsense be your firewall and the rest of your network behind it?

So you have ISP -> vmbr0 -> PFsense WAN -> PFsense LAN -> vmbr1 -> REST OF NETWORK

 

[copetogo]

When messing with IP settings it is ALWAYS a possibility to lock yourself out. You can however plug a keyboard and monitor into the host and have access that way. It's all command line but it is nice to get to know how to fix it when you have to.
The IP address you have in Proxmox is just for the GUI. What you can do is set the other NIC for an IP inside the network 192.168.1.x and have no gateway (You can only have one gateway for proxmox host) apply the settings then see if you can ping that IP from your PC on the same subnet 192.168.1.x if you can then try to browse to the IP and see if the GUI comes up. In the past I have had to edit some settings in the configuration with keyboard and monitor on host.

Is your goal to have PFsense be your firewall and the rest of your network behind it?

So you have ISP -> vmbr0 -> PFsense WAN -> PFsense LAN -> vmbr1 -> REST OF NETWORK

you are bang on with what network route you suggested. So let me understand what you mean under vmbr 0 which acts as the wan bridge for pfsense (also lan for proxmox host ) i only input the IP and keep the gateway entry blank. What do you mean by other NIC (vmbr 0 or vmbr1) because vmbr 1 has no gateway anyways .attaching screenshot of what i understood.

 

[rtorres]

you are bang on with what network route you suggested. So let me understand what you mean under vmbr 0 which acts as the wan bridge for pfsense (also lan for proxmox host ) i only input the IP and keep the gateway entry blank. What do you mean by other NIC (vmbr 0 or vmbr1) because vmbr 1 has no gateway anyways .attaching screenshot of what i understood.

This is how I have mine set up from installation. The vmbr0 (enp4s0) is the Proxmox Management port and LAN port for pfSense. 172.22.1.10:8006 is what I use to get into Proxmox and 172.22.1.11 would be for gateway (pfSense).

After Proxmox is installed, I add my interfaces and name them accordingly:

vmbr0 - enp4s0 - LAN for Proxmox and VMs.
vmbr1 - enp3s0 - WAN for pfSense only
vmbr2 - eno1 - LAN (Empty/Not used)



At this point I still don't have internet, so I install pfSense and set vmbr0 as LAN and vmbr1 as WAN. Now internet is up and running I can start building my VMs and getting latest updates for Proxmox (apt update && apt upgrade).

All my VMs will get access to only vmbr0 - enp4s0 since I want to route everything from Proxmox through pfSense out to WAN (ISP) like a virtual switch.

So essentially -

Proxmox:
vmbr0 - enp4s0 - LAN

pfSense VM:
vmbr0 - enp4s0 - LAN
vmbr1 - enp3s0 - WAN


UniFi Console (Debian) VM:
vmbr0 - enp4s0 - LAN


TrueNAS Scale VM:
vmbr0 - enp4s0 - LAN


So Gateway I set to 172.22.1.11 and DNS I set to 172.22.1.11 in Proxmox because that's where my pfSense resides in - 172.22.1.11.

 

[copetogo]

This is how I have mine set up from installation. The vmbr0 (enp4s0) is the Proxmox Management port and LAN port for pfSense. 172.22.1.10:8006 is what I use to get into Proxmox and 172.22.1.11 would be for gateway (pfSense).

After Proxmox is installed, I add my interfaces and name them accordingly:

vmbr0 - enp4s0 - LAN for Proxmox and VMs.
vmbr1 - enp3s0 - WAN for pfSense only
vmbr2 - eno1 - LAN (Empty/Not used)

View attachment 70801

At this point I still don't have internet, so I install pfSense and set vmbr0 as LAN and vmbr1 as WAN. Now internet is up and running I can start building my VMs and getting latest updates for Proxmox (apt update && apt upgrade).

All my VMs will get access to only vmbr0 - enp4s0 since I want to route everything from Proxmox through pfSense out to WAN (ISP) like a virtual switch.

So essentially -

Proxmox:
vmbr0 - enp4s0 - LAN

pfSense VM:
vmbr0 - enp4s0 - LAN
vmbr1 - enp3s0 - WAN
View attachment 70799

UniFi Console (Debian) VM:
vmbr0 - enp4s0 - LAN
View attachment 70798

TrueNAS Scale VM:
vmbr0 - enp4s0 - LAN
View attachment 70800

So Gateway I set to 172.22.1.11 and DNS I set to 172.22.1.11 in Proxmox because that's where my pfSense resides in - 172.22.1.11.

I got your route clearly but from what i understand from the forum, pfsense is not desirable as VM and using it to give IP in proxmox and doing what you suggested for one reason may not be desirable . Say your pfsense VM goes down (can be due to anything) ,what happens then . one thing to mention is since you are routing all your other vms via proxmox , will they be accessible say if your pfsense vm goes down. Have you faced any issues with the setup you have . also curious to know why is your CIDR 16 bit .any specific reason?

 

[rtorres]

I got your route clearly but from what i understand from the forum, pfsense is not desirable as VM and using it to give IP in proxmox and doing what you suggested for one reason may not be desirable . Say your pfsense VM goes down (can be due to anything) ,what happens then . one thing to mention is since you are routing all your other vms via proxmox , will they be accessible say if your pfsense vm goes down. Have you faced any issues with the setup you have . also curious to know why is your CIDR 16 bit .any specific reason?

I haven't had any issues running pfSense as a VM.. The only time it's down is when I either mess around with configurations and literally bring it down due to some stupid reason or my ISP (xFinity) goes down.

Even Netgate has a pfSense ISO for VMs:


This is a homelab, if pfSense goes down it's not a big deal. I connect to the WiFi ( UniFi U6 Mesh Access Point connected in the LAN port that pfSense manages), can access Proxmox and VMs and log into pfSense to see what is going on.

So virtually I have Proxmox handling LAN via vmbr0 - enp4s0 and physically I have an access point connected to the enp4s0 port.

I can always add the additional 3rd NIC I have empty but I haven't had the need to. I use only vmbr0 - enp4s0 - LAN and vmbr1 - enp3s0 because those are 2.5GbE NICs. vmbr2 - eno1 is only a 1GbE NIC.



/16 I kept it since messing around with Proxmox and pfSense. It's what I've put since and have left it Do I need that many IPs? No but I have too much OCD to change it to another CIDR

 

[vesalius]

Will second @rtorres. I ran pfsense for a couple of years then switched to opnsense for the last 3-4 years all running as a proxmox VM, in a config similar to his description. No issues with either going down on their own. Have come to have as much faith in this setup as any separate hardware with a 1g fiber connection. Some of the things that I have found most helpful are while learning/tinkering with *sense (if I make a difficult-to-recover mistake) I can within a few seconds revert to working saved proxmox VM backup or snapshot. For major *sense version upgrades or RC candidates I could either duplicate my working *sense vm and see if an upgrade in place worked there or install the Upgrade into a new vm and reconfigure or import a config from my current stable production.

The biggest neg I have seen is not that the *sense vm will go down, but if proxmox is hobby time and you tinker significantly resulting in host reboot (or anytime you have to reboot proxmox for a kernel upgrade) then the network will also be down for you (and worse still with the neg and often instant feedback from fam) until proxmox reboots and the *sense vm restarts. Not a problem if time proxmox reboot to family network downtime. Many network admins for businesses run firewalls/routers as vms for internal and edge deployments.

 

[copetogo]

This is how I have mine set up from installation. The vmbr0 (enp4s0) is the Proxmox Management port and LAN port for pfSense. 172.22.1.10:8006 is what I use to get into Proxmox and 172.22.1.11 would be for gateway (pfSense).

After Proxmox is installed, I add my interfaces and name them accordingly:

vmbr0 - enp4s0 - LAN for Proxmox and VMs.
vmbr1 - enp3s0 - WAN for pfSense only
vmbr2 - eno1 - LAN (Empty/Not used)

View attachment 70801

At this point I still don't have internet, so I install pfSense and set vmbr0 as LAN and vmbr1 as WAN. Now internet is up and running I can start building my VMs and getting latest updates for Proxmox (apt update && apt upgrade).

All my VMs will get access to only vmbr0 - enp4s0 since I want to route everything from Proxmox through pfSense out to WAN (ISP) like a virtual switch.

So essentially -

Proxmox:
vmbr0 - enp4s0 - LAN

pfSense VM:
vmbr0 - enp4s0 - LAN
vmbr1 - enp3s0 - WAN
View attachment 70799

UniFi Console (Debian) VM:
vmbr0 - enp4s0 - LAN
View attachment 70798

TrueNAS Scale VM:
vmbr0 - enp4s0 - LAN
View attachment 70800

So Gateway I set to 172.22.1.11 and DNS I set to 172.22.1.11 in Proxmox because that's where my pfSense resides in - 172.22.1.11.

So this is what i did and then exactly happened what i predicted locked myself out of proxmox gui . had to physically login to interface to proxmox and reset network interface Just retracing the steps i did may be some one can jump into what i missed

1) removed static reservation of proxmox on ISP router (192.168.5.150)
2) in VMBR0 of promox host i changed ip to 192.168.1.150/24 (subnet of pfsense)
3) in VMBR0 of proxmox host default gateway set to 192.168.1.1 (pfsense router WAN IP)
4) changed proxmox dns IP to pfsense ip 192.168.1.1

 

[vesalius]

What doess the pfsense VM side look like under hardware? is pfsense pulling an IP from WAN (assume that is vmbr1). Whats the IP of the computer you are using to access the proxmox webgui? is it pulling an ip address from pfsense DHCP or at least manually set to be in the new pfsense LAN subnet?

 

[copetogo]

What doess the pfsense VM side look like under hardware? is pfsense pulling an IP from WAN (assume that is vmbr1). Whats the IP of the computer you are using to access the proxmox webgui? is it pulling an ip address from pfsense DHCP or at least manually set to be in the new pfsense LAN subnet?

vtnet o is wan side for pfsense vtnet 1 is the lan side

 

[vesalius]

To start you may want to consider editing your /etc/network/interfaces with the changes below:

auto vmbr0
    iface vmbr0 inet static
    address 192.168.5.150/24
    bridge-ports enp2s0
    bridge-stp off bridge-fd 0
    bridge-vlan-aware yes
    bridge-vids 2-4094

auto vmbr1
    iface vmbr1 inet static
    address 192.168.1.150/24
    gateway 192.168.1.1
    bridge-ports enps0
    bridge-stp off bridge-fd 0
    bridge-vlan-aware yes
    bridge-vids 2-4094

This interim step will allow you still access proxmox at 192.168.5.150 that worked under the pre-existing network setup, and will make proxmox accessible from 192.168.1.150. then you can figure out next steps. Goal state would be to erase the vmbr0 - address 192.168.5.150/24 line and still have everything working as expected.

How about the other 2 questions:
1. Is pfsense pulling an IP from WAN over vmbr0/vnet0?
2. Whats the IP of the computer you are using to access the proxmox webgui?

Other questions,
what is vmbr0 (enp2s0) plugged into? Assume this is your ISP router?
What is vmbr1 (enp1s0) plugged into? should not be your ISP router, a separate cheap multipart switch will do.

Right now proxmox is attempting to connect to pfsense at 192.168.1.1 through vmbr0, which is (vnet0/wan) to pfsense and is appropriately denied.

 

[copetogo]

To start you may want to consider editing your /etc/network/interfaces with the changes below:

auto vmbr0
    iface vmbr0 inet static
    address 192.168.5.150/24
    bridge-ports enp2s0
    bridge-stp off bridge-fd 0
    bridge-vlan-aware yes
    bridge-vids 2-4094

auto vmbr1
    iface vmbr1 inet static
    address 192.168.1.150/24
    gateway 192.168.1.1
    bridge-ports enps0
    bridge-stp off bridge-fd 0
    bridge-vlan-aware yes
    bridge-vids 2-4094

This interim step will allow you still access proxmox at 192.168.5.150 that worked under the pre-existing network setup, and will make proxmox accessible from 192.168.1.150. then you can figure out next steps. Goal state would be to erase the vmbr0 - address 192.168.5.150/24 line and still have everything working as expected.

How about the other 2 questions:
1. Is pfsense pulling an IP from WAN over vmbr0/vnet0?
2. Whats the IP of the computer you are using to access the proxmox webgui?

Other questions,
what is vmbr0 (enp2s0) plugged into? Assume this is your ISP router?
What is vmbr1 (enp1s0) plugged into? should not be your ISP router, a separate cheap multipart switch will do.

Right now proxmox is attempting to connect to pfsense at 192.168.1.1 through vmbr0, which is (vnet0/wan) to pfsense and is appropriately denied.

ok firstly i think a slight error in the code bridge port for vmbr 1 should be enp1s0 (must be a typo error)
to answer your questions

1) PF sense is accessible from both ips (meaning 192.168.5.3 -from ISP side and 192.168.1.1 from the lan side). so to answer your question PFsense IP from wan side is ISP router provided one as mentioned above
2) IP of the computer i am trying to access proxmomx web gui is one provided by PFsense subnet (from the 192.168.1.0/24)
3) You are correct vmbr0 is from ISP to proxmox host and no vmbr 1 is not into router, my proxmox host has two nics, vmbr 1 goes from proxmox host to switch then to a mesh system which then gives me wifi network (LAN side of pfsense)

i will try setting wan side ip as you mentioned and then see how it goes from there

 

[rtorres]

If it helps, this is my /etc/network/interfaces. I don't do vlans because I don't understand vlans too well and don't want to cause any more problems than I already have.

Any more time I spend with this will lead to divorce! Haha! Only joking of course..... I hope..............!

auto lo
iface lo inet loopback

iface enp4s0 inet manual

iface enp3s0 inet manual

iface eno1 inet manual

auto vmbr0
iface vmbr0 inet static
        address 172.22.1.10/16
        gateway 172.22.1.11
        bridge-ports enp4s0
        bridge-stp off
        bridge-fd 0
#LAN-FlexIO

auto vmbr1
iface vmbr1 inet manual
        bridge-ports enp3s0
        bridge-stp off
        bridge-fd 0
#WAN-M2

auto vmbr2
iface vmbr2 inet manual
        bridge-ports eno1
        bridge-stp off
        bridge-fd 0
#LAN-HPGbE

Does your ISP give you a static IP?

When I install Proxmox from USB, I select my LAN interface (vmbr0 - enp4s0) and set a static IP of 172.22.1.10 /16 for Proxmox, Gateway 172.22.1.11 and DNS 172.22.1.11.

Then to get into Proxmox, I hook into the enp4s0 LAN port with a ethernet cable, connect it into either my laptop or desktop and get into Proxmox to set up pfSense and all other VMs. WAN port is connected as well but I don't have internet until I set up pfSense - sometimes I have to reset my modem for pfSense to pick up my ISP IPs (xFinity, DHCP IPs (IPv4 and IPv6) are automatic from ISP)

The only Static IPs I set are during Proxmox USB install, all other IPs are handled/configured in pfSense.