IP Failover / Routing via hosted Pfsense configuration

[Bruno SINOU]

Hello,

I have been using a Proxmox for a while with a very simple network configuration, with only one NIC and one bridge that NAT the traffic to a private VLAN like this: (/etc/network/interfaces of the host)

auto eno1
# internet facing interface
iface eno1 inet static
    address XXX.XXX.XXX.105
    netmask 255.255.255.0
    gateway XXX.XXX.XXX.1   
 
auto vmbr0
# private VLAN
iface vmbr0 inet static
        address 10.10.10.254
        netmask 255.255.255.0
        bridge_ports none
        bridge_stp off
        bridge_fd 0
        post-up echo 1 > /proc/sys/net/ipv4/ip_forward

        # Redirect all trafic for http ports to internal reverse proxy
        post-up iptables -t nat -A PREROUTING -i eno1 -p tcp --dport 80 -j DNAT --to 10.10.10.100:80
        post-up iptables -t nat -A PREROUTING -i eno1 -p tcp --dport 443 -j DNAT --to 10.10.10.100:443
        post-down iptables -t nat -D PREROUTING -i eno1 -p tcp --dport 80 -j DNAT --to 10.10.10.100:80
        post-down iptables -t nat -D PREROUTING -i eno1 -p tcp --dport 443 -j DNAT --to 10.10.10.100:443
       # NAT
        post-up iptables -t nat -A POSTROUTING -s '10.10.10.0/24' -o eno1 -j MASQUERADE
        post-down iptables -t nat -D POSTROUTING -s '10.10.10.0/24' -o eno1 -j MASQUERADE
        # SSH   
       post-up iptables -t nat -A PREROUTING -i eno1 -p tcp --dport 22100 -j DNAT --to 10.10.10.100:22
       post-down iptables -t nat -D PREROUTING -i eno1 -p tcp --dport 22100 -j DNAT --to 10.10.10.100:22
       .... More ssh rules

And I would like to go one step further now:
I have a fresh Proxmox 5 on Debian 9 by online.net with one failover IP, so that:
- main IP is only used to reach the host
- VLAN is accessed via the failover IP that is used on a pfsense VM that makes the routing and so on.

I am not a network specialist and despite many hours of googling and trial & error, I still don't get how I have to tweak the host interfaces file to have the pfsense VM getting the Public Failover IP address.

For the record, here are the few highlights of my setup at online.net:
- there is *only one interface* on the WAN eno1
- failover IP address has a generated MAC
- I have installed a fresh Proxmox pve on an up-to-date debian 9, rather than using the pre-packaged proxmox offered by online.net: that's the only way to go if I don't want to have half of my SSD space spoiled to make RAID 1.
- host must not access the VLAN

I imagine it should be obvious and straightforward because this setup seems very _classic_ to me but I cannot make it work and would be very gratefull for any hint or pointers: all step by step guides I have found always seems to miss THE crucial point that makes things work.

Please don't hesitate to ask if you need more info or if something is not clear.

Many thanks!

 

[Bruno SINOU]

OK, after more research:

- I realized I should have put this in the other Category, sorry for this.
- I found some solution in the German version of the forum
- I finally rather use Opnsensethan pfsense; it's smoother.

... And I solved my problems and if it might help anyone here are the main hints that saved me:

First the interface file:

iface eno1 inet manual

# Virtual switch for VLAN
# (connect your firewall/router KVM instance and private VLAN guests here)
# I use vmbr0 here so that when I create a new VM, correct bridge is already selected
# and I don't forget to select it
auto vmbr0
iface vmbr0 inet manual
    bridge_ports none
      bridge_stp off
      bridge_fd 0

# Bridge with internet facing IP
auto vmbr1
iface vmbr1 inet static
        address  <YOUR MAIN PUBLIC ADDRESS>
        netmask  255.255.255.255
        gateway  <GATEWAY FOR YOUR MAIN PUBLIC ADDRESS>
        pointopoint <GATEWAY FOR YOUR MAIN PUBLIC ADDRESS>
        bridge_ports eno1
        bridge_stp off
        bridge_fd 0
        up ip route add 10.0.0.0/24 via <YOUR FAILOVER IP> dev vmbr1
        up sysctl -w net.ipv4.ip_forward=1
        up sysctl -w net.ipv4.conf.eno1.send_redirects=0

After reboot, via the Proxmox UI :
- create an new VM using the OPNSense ISO, 20GB ssd, 2GB RAM and 1 core should be sufficient.
- Use vmbr1 as internet access. DO NOT START yet.
- add a second interface plugged in at vmbr0.
- set the virtual MAC address given by your ISP for the failover IP on vmbr1

Start. You should be then able to configure opnsense by following setup instruction.
Main tips:

- do not start by configuring LAN when you are asked at the beginning
- add 2 interface one for the WAN on vmbr1 one for the VLAN on vmbr0
- set IP for the WAN:
- No DHCP
- IP is your Failover IP
- Chose 32 as network mask (255.255.255.255)
- Gateway IP is the *same IP as the gateway for your main IP* (for instance if your main IP is 1.2.3.4, gateway is probably 1.2.3.1)
- DNS Name Server is the one provided by your ISP

After install if you chose 8 and can ping google.com, you have won!

Hope this might help someone....

 

[Bruno SINOU]

PS: main trick in the above post are the few lines in the interface file that enable routing of the failover IP:

(...)
pointopoint <GATEWAY FOR YOUR MAIN PUBLIC ADDRESS>
  (...)
      up ip route add 10.0.0.0/24 via <YOUR FAILOVER IP> dev vmbr1
        up sysctl -w net.ipv4.ip_forward=1
        up sysctl -w net.ipv4.conf.eno1.send_redirects=0
(..)

that I do not completely understand to be honnest. So if anyone in the neighbourhood can give a little more info on this, it would be very nice:

1) Why do we have to add the pointopoint directive
2) what does the ip route add exactly do
3) What does the 2 last lines...

Thanks again in advance!