pfsense WAN - only 1 public ip - how to bridge

[pjprops]

Hello,

We use proxmox as host with pfsense as firewall and dhcp-server virtualised on it.

We received last week from our new ISP our public ip address. This means we have only 1 ip to assign to our pfsense server but while it's a virtual one I have no clue how I can bridge though proxmox to our pfsense server.

Here is our situation

received from the ISP
1xx.24x.1xx.24/30 – network address (useless)
1xx.24x.1xx.25/30 – router IP (gateway address)
1xx.24x.1xx.26/30 – our firewall IP (pfsense)
1xx.24x.1xx.27/30 – broadcast address (useless)

proxmox /etc/network/interfaces

network interface settings
auto lo
iface lo inet loopback

iface eth0 inet manual
iface eth1 inet manual (lan network)
iface eth3 inet manual
iface eth4 inet manual
iface eth6 inet manual

auto vmbr1 (lan network)
iface vmbr1 inet static
        address  10.0.0.2
        netmask  255.255.0.0
        bridge_ports eth1
        bridge_stp off
        bridge_fd 0

auto vmbr2 (wan network for pfsense)
iface vmbr2 inet manual
        bridge_ports eth6
        bridge_stp off
        bridge_fd 0

As you can see we didn't define anything on the proxmox server but only bridged the eth6 port to vmbr2.
On the pfsense server I have defined the gateway and ipaddress I received from our isp but this is not working.
I think I have to define more on the proxmox server but I don't know what or how...
I hope someone can help me a little bit.

thanks in advance

 

[Q-wulf]

Are you looking for this kind of setup ? :

Public IP <-> pfSense <-> Proxmox

Then it will work basically like this:

Public IP <-> vmbrA <-> Pfsense <-> vmbrB <-> Proxmox
Public IP <-> vmbrA <-> Pfsense <-> vmbrB <-> VM1
Public IP <-> vmbrA <-> Pfsense <-> vmbrC <-> VM2


Since i am only really familiar with openvswitch based networking on Proxmox (https://pve.proxmox.com/wiki/Open_vSwitch), to the point where i am comfortable giving advice on this, i shall make these examples using said method.

1. Create a OVS_Bridge (vmbr0)
   1. Assign physical nic(s) or bond(s) to the ovs_bridge as ovs_ports.
2. Create a OVS_Bridge (vmbr1) with no ovs_ports.
3. Create a Virtual Machine with pfsense.
   1. Assign it a OVS_IntPort (net0) for your public_IP(P1) and vmbr0 as Bridge.
   2. Assign it a OVS_intPort (net1) with a private_IP(A) and vmbr1 as Bridge.
4. On Proxmox create a OVS_IntPort with a private_IP(B) and vmbr1 as Bridge. Use private_IP(A) as gateway.
5. For VM's you have the following options:
   1. Use vmbr1
      1. assign vNic(s) using the vmbr1 as Bridge and private_IP(A) as Gateway.
   2. Create a new vmbrX (e.g. vmbr3)
      1. Assign your pfSense Instance a new private_IP(C) with vmbr3 as Bridge.
      2. assign vNic(s) using the vmbr3 as Bridge and private_IP(C) as Gateway.

Important to note:
In order to not log yourself out of Proxmox, while you are doing your setups, you need to take at least 1 of the Options presented below

• Be very careful to assign IP's in the right order and without mistakes, or you will not be able to access Proxmox if you made a mistake.
• have a second public_IP(P2) that you have assigned to a OVS_Bridge like e.g. vbmr2 that you can use for proxmox access.(needs ethX as ovs_ports)
  • Once your setup works you can disable it.
• have local (or IPMI or similar) access to your server.
  • so you can make make changes when you have errors in your networking.
• Run your Proxmox nodes OVS_IntPort(N1) of vmbr0 (second public IP)
  • I'd use this method, least painful.
    • When you confirm the setup as working, you then you create a second OVS_IntPort (N2) for Proxmox node using vmbr1.
      • when you have that confirmed as working, you remove the first OVS_IntPort (N1)
        • Your Proxmox-Node now sits behind your pfSense.

Hope that helps to get you started.
The basic model should translate to native linux networking as well, if you do not wanna use openvswitch.

 

[pjprops]

Thanks for your reply,

Our setup is as following:

Internet Modem (gateway address) <--> Proxmox <--> Pfsense (virtual machine) (firewall ip)

I created a vmbr2 on our proxmox which bridges the eth6-port to pfsense, but as you can see below, I didn't specify any ip or range on the proxmox itself and I don't know if it can work this way.

I hope there is a way to directly connect the modem to the vm (pfsense) so I can use only my available firewall-ip-address.

auto vmbr2 (wan network for pfsense)
iface vmbr2 inet manual
        bridge_ports eth6
        bridge_stp off
        bridge_fd 0

 

[ossvirt]

I have a similar setup with Pfsense in a VM. I also have a vmbr configured as above with no IP address or network for Proxmox defined on the bridge, just a physical ethernet port slaved to the bridge - exactly as you have. This allows me to use PPPoE on Pfsense to connect straight through to the fibre modem and it makes a solid connection. I have had this working for over 6 months.

As I understand it - An IP address defined on a vmbr bridge is only for use by the Proxmox interface, it doesn't give anything extra to the bridge itself. Similarly the netmask, gateway, broadcast etc is for the Proxmox webgui's use and doesn't define the network for any other host. DHCP on the Pfsense VM can do that, or just static config on each host connected to the bridge. Can anyone else confirm that?

So I think you define the ISP network config as a static network setup on the Pfsense interface that connects through the bridge to the modem. That works for me with PPPoE. I don't know if it is the same with a static ethernet connection but I think it could be.

 

[pjprops]

Thanks for your reply,

It's working that way indeed.
I made a stupid mistake on Pfsense itselves (mistake in the ipaddress), causing the problem. The proxmox-setup with a bridge without IP-address on proxmox is working just fine.

thanks for your time

 

[Q-wulf]

As I understand it - An IP address defined on a vmbr bridge is only for use by the Proxmox interface, it doesn't give anything extra to the bridge itself. Similarly the netmask, gateway, broadcast etc is for the Proxmox webgui's use and doesn't define the network for any other host. DHCP on the Pfsense VM can do that, or just static config on each host connected to the bridge. Can anyone else confirm that?

to use analogies, think of a linux native bridge as a virtual switch.
You assign it a IP, the analogy would be that it is the switchs IP (since the switch is actually proxmox, its a Ip used by proxmox).
You assign a VM that bridge, you can use the analogy of putting a virtual cable between your VM and your virtual switch.
You assign that VM a IP via that bridge via its vNic you can then use the analogy of asigning a physical nic an IP in order for the nic on that VM to be able to connect to what ever network you feel like assigning.

Now regarding DHCP:
Proxmox AFAIK needs at least one IP assigned that is static. That IP is used for Cluster communication. You can identify it in /etc/hosts via the "pvelocalhost" flag. That is why during install you need to specify that IP/Subnet/Gateway and they do not give you the DHCP option.
However you can if you feel like it use DHCP in the vPfSense to assign Proxmox additional IP's, so you can e.g. grab the GUI via 10.1.1.x and 192.168.2.x and 10.200.5.x based networks.
You defnitely can use PfSense to asisgn VM's IP's to your hearts desire.