[SOLVED] Hetzner Server with 2 public IP addresses and OPNSense Firewalls

Skyfay · May 29, 2024

Hello everyone

I need some help today regarding networking. Until now I had a working setup but now I have a new server which I want to share with someone.

My configuration for myself as the solo user is as follows:

Code:

auto lo
iface lo inet loopback

iface enp7s0 inet manual

auto vmbr0
iface vmbr0 inet static
    address XXX.XXX.96.93/26
    gateway XXX.XXX.96.65
    bridge-ports enp7s0
    bridge-stp off
    bridge-fd 0
    post-up sysctl -w net.ipv4.ip_forward=1
    post-up iptables -t nat -A PREROUTING -i vmbr0 -p tcp -m multiport ! --dport 22,8006 -j DNAT --to 10.10.10.1
    post-up iptables -t nat -A PREROUTING -i vmbr0 -p udp -m multiport ! --dport 41741 -j DNAT --to 10.10.10.1
# Public IP Configuration

auto vmbr1
iface vmbr1 inet static
    address 10.10.10.0/31
    bridge-ports none
    bridge-stp off
    bridge-fd 0
    post-up iptables -t nat -A POSTROUTING -s '10.10.10.1/31' -o vmbr0 -j MASQUERADE
    post-down iptables -t nat -D POSTROUTING -s '10.10.10.1/31' -o vmbr0 -j MASQUERADE
# Proxmox - OPNSense Bridge

auto vmbr2
iface vmbr2 inet manual
    bridge-ports none
    bridge-stp off
    bridge-fd 0
# VM Main Network Interface Bridge

The network forwards all traffic to my OPNSense firewall. Only port 8006 for Porxmox and 1-2 other ports are not. It works perfectly.

But now another public IP address is added, and another OPNSense firewall. The setup should simply be mirrored, only with the second public IP address.
I'm really getting desperate because I can't manage to configure it like this. I know there are two methods bridged and routed (brouter). I would prefer the first if that is possible. I can also generate a separate MAC address for the second IP address.

I tried the following configuration earlier, but unfortunately it did not work.

Code:

auto lo
iface lo inet loopback

iface enp7s0 inet manual

auto vmbr0
iface vmbr0 inet static
    address XXX.XXX.96.93/26
    gateway XXX.XXX.96.65
    bridge-ports enp7s0
    bridge-stp off
    bridge-fd 0
    post-up sysctl -w net.ipv4.ip_forward=1
    post-up iptables -t nat -A PREROUTING -i vmbr0 -p tcp -m multiport ! --dport 22,8006 -j DNAT --to 10.10.10.1
    post-up iptables -t nat -A PREROUTING -i vmbr0 -p udp -m multiport ! --dport 41741 -j DNAT --to 10.10.10.1
# Public IP Configuration


auto vmbr1
iface vmbr1 inet static
    address 10.10.10.1/31
    bridge-ports none
    bridge-stp off
    bridge-fd 0
    post-up iptables -t nat -A POSTROUTING -s '10.10.10.1/31' -o vmbr0 -j MASQUERADE
    post-down iptables -t nat -D POSTROUTING -s '10.10.10.1/31' -o vmbr0 -j MASQUERADE
# Proxmox - OPNSense Bridge

auto vmbr2
iface vmbr2 inet manual
    bridge-ports none
    bridge-stp off
    bridge-fd 0
# VM Main Network Interface Bridge

auto vmbr3
iface vmbr3 inet static
    address XXX.XXX.96.90/26
    bridge-ports enp7s0.2
    bridge-stp off
    bridge-fd 0
    post-up sysctl -w net.ipv4.ip_forward=1
    post-up ip link add link enp7s0 address 00:50:56:01:1D:21 dev enp7s0.2
    post-up ip link set dev enp7s0.2 up
    post-up iptables -t nat -A PREROUTING -i vmbr3 -p tcp -j DNAT --to 10.10.10.1
    post-up iptables -t nat -A PREROUTING -i vmbr3 -p udp -j DNAT --to 10.10.10.1
    pre-down ip link del enp7s0.2
# Public IP Configuration 2. Public IP

auto vmbr4
iface vmbr4 inet static
    address 10.10.10.1/31
    bridge-ports none
    bridge-stp off
    bridge-fd 0
    post-up iptables -t nat -A POSTROUTING -s '10.10.10.1/31' -o vmbr3 -j MASQUERADE
    post-down iptables -t nat -D POSTROUTING -s '10.10.10.1/31' -o vmbr3 -j MASQUERADE
# Proxmox - OPNSense Bridge 2. Firewall

auto vmbr5
iface vmbr5 inet manual
    bridge-ports none
    bridge-stp off
    bridge-fd 0
# Proxmox - OPNSense Bridge 2. Network

source /etc/network/interfaces.d/*

Any help is welcome. Thanks!

_gabriel · May 30, 2024

easier to set the ip within the guest which bypass your opnsense.

Skyfay · May 30, 2024

What exactly do you mean? So my goal is to have two OPNsense firewalls both with a public IP which I can then use to manage the “LAN” for the VMs, DHCP, DNS etc.
The first step already exists, I just don't understand how I can include the second public IP address and how exactly I have to set this up.

_gabriel · May 30, 2024

oh, I got it.
first vmbr1 and 3 can't have same address.

Skyfay · May 30, 2024

Ah okay, yes, I've already thought about that. But I had also tested it with another one and unfortunately it didn't work either.
But its no problem i can change that IP address.

Do you mean vmbr1 and vmbr4?

_gabriel · May 30, 2024

second, switch to regular /24 subnet.

Skyfay · May 30, 2024

_gabriel said:
first vmbr1 and 3 can't have same address.

Do you mean vmbr1 and vmbr4?

_gabriel said:
second, switch to regular /24 subnet.

What exactly do you mean? Only 2 IP addresses are needed for the connection between the public IP interface and the OPNSense connection. In this case, the OPNSense firewall uses 10.10.10.0 as the WAN interface.

Theoretically, you could omit this step for the second public IP since Proxmox already runs via the first one, right? So that the OPNSense gets the second public IP address directly as WAN.

Skyfay · May 31, 2024

I have now been able to fix the problem. I was thinking a bit too complicated the whole time.
I can pass the second public IP address directly to the OPNSense VM without a second bridge as with the main public IP. I only need this to avoid passing on the ports for Proxmox. Maybe it would even be possible without it.

The following instructions helped me to understand how it works, especially with Hetzner and the virtual MAC address. The virtual MAC address of Hetzner is entered in the VM under the network adapter vmbr0. All traffic is then forwarded directly to the VM via this MAC address.

This is the configuration I am using now. My setup is two public IPs, one for Proxmox itself and the first OPNSense firewall, and the second public IP for the second OPNSense firewall. Both then have a network adapter for the “LAN” to which the VMs are connected.

Code:

auto lo
iface lo inet loopback

iface enp7s0 inet manual

auto vmbr0
iface vmbr0 inet static
        address <Main Public IP/CIDR>
        broadcast <Broadcast>
        gateway <Gateway>
        bridge-ports enp7s0
        bridge-stp off
        bridge-fd 0
        bridge-hw enp7s0
        pointopoint <Gateway>
        post-up sysctl -w net.ipv4.ip_forward=1
        post-up iptables -t nat -A PREROUTING -i vmbr0 -p tcp -m multiport ! --dport 22,8006 -j DNAT --to 10.10.10.1
        post-up iptables -t nat -A PREROUTING -i vmbr0 -p udp -m multiport ! --dport 54937 -j DNAT --to 10.10.10.1
#Public IP-Adresses

auto vmbr1
iface vmbr1 inet static
        address 10.10.10.0/31
        bridge-ports none
        bridge-stp off
        bridge-fd 0
        post-up iptables -t nat -A POSTROUTING -s '10.10.10.1/31' -o vmbr0 -j MASQUERADE
        post-down iptables -t nat -A POSTROUTING -s '10.10.10.1/31' -o vmbr0 -j MASQUERADE
#Proxmox - OPNSense Bridge for Main Public IP

auto vmbr2
iface vmbr2 inet manual
        bridge-ports none
        bridge-stp off
        bridge-fd 0
#VM Lan Network 1

auto vmbr3
iface vmbr3 inet manual
        bridge-ports none
        bridge-stp off
        bridge-fd 0
#VM Lan Network 2

Notes:

enp7s0 must be replaced with your network adapter if you do not have the same one. You have to adjust this over the whole configuration.
you connect the OPNSense firewall for the second public IP with the vmbr0 inteface, enter the MAC address of Hetzner and then configure the WAN interface of OPNSense to the values displayed by Hetzner in the Robot Dashboard from the 2. Public IP-Address.
The first OPNSense firewall is connected to vmbr1. The WAN interface of OPNSense must be configured here to 10.10.10.1/31. All ports that you do not want to pass on to the OPNSense firewall because the Proxmox host needs them can be defined in the ports TCP and UDP. (Currently it is 8006, 22 TCP and 54937 UDP for Tailscale in my case). If you do not want to have the UDP port at Proxmox, the entry looks like this: post-up iptables -t nat -A PREROUTING -i vmbr0 -p udp -j DNAT --to 10.10.10.1

Maybe I can help someone with this post who had the same problems as me.

ElectricCircus · Aug 22, 2024

Skyfay said:
I have now been able to fix the problem. I was thinking a bit too complicated the whole time.
I can pass the second public IP address directly to the OPNSense VM without a second bridge as with the main public IP. I only need this to avoid passing on the ports for Proxmox. Maybe it would even be possible without it.

The following instructions helped me to understand how it works, especially with Hetzner and the virtual MAC address. The virtual MAC address of Hetzner is entered in the VM under the network adapter vmbr0. All traffic is then forwarded directly to the VM via this MAC address.

This is the configuration I am using now. My setup is two public IPs, one for Proxmox itself and the first OPNSense firewall, and the second public IP for the second OPNSense firewall. Both then have a network adapter for the “LAN” to which the VMs are connected.

Code:

auto lo iface lo inet loopback iface enp7s0 inet manual auto vmbr0 iface vmbr0 inet static address <Main Public IP/CIDR> broadcast <Broadcast> gateway <Gateway> bridge-ports enp7s0 bridge-stp off bridge-fd 0 bridge-hw enp7s0 pointopoint <Gateway> post-up sysctl -w net.ipv4.ip_forward=1 post-up iptables -t nat -A PREROUTING -i vmbr0 -p tcp -m multiport ! --dport 22,8006 -j DNAT --to 10.10.10.1 post-up iptables -t nat -A PREROUTING -i vmbr0 -p udp -m multiport ! --dport 54937 -j DNAT --to 10.10.10.1 #Public IP-Adresses auto vmbr1 iface vmbr1 inet static address 10.10.10.0/31 bridge-ports none bridge-stp off bridge-fd 0 post-up iptables -t nat -A POSTROUTING -s '10.10.10.1/31' -o vmbr0 -j MASQUERADE post-down iptables -t nat -A POSTROUTING -s '10.10.10.1/31' -o vmbr0 -j MASQUERADE #Proxmox - OPNSense Bridge for Main Public IP auto vmbr2 iface vmbr2 inet manual bridge-ports none bridge-stp off bridge-fd 0 #VM Lan Network 1 auto vmbr3 iface vmbr3 inet manual bridge-ports none bridge-stp off bridge-fd 0 #VM Lan Network 2

Notes:

enp7s0 must be replaced with your network adapter if you do not have the same one. You have to adjust this over the whole configuration.

you connect the OPNSense firewall for the second public IP with the vmbr0 inteface, enter the MAC address of Hetzner and then configure the WAN interface of OPNSense to the values displayed by Hetzner in the Robot Dashboard from the 2. Public IP-Address.

The first OPNSense firewall is connected to vmbr1. The WAN interface of OPNSense must be configured here to 10.10.10.1/31. All ports that you do not want to pass on to the OPNSense firewall because the Proxmox host needs them can be defined in the ports TCP and UDP. (Currently it is 8006, 22 TCP and 54937 UDP for Tailscale in my case). If you do not want to have the UDP port at Proxmox, the entry looks like this: post-up iptables -t nat -A PREROUTING -i vmbr0 -p udp -j DNAT --to 10.10.10.1

Maybe I can help someone with this post who had the same problems as me.

Thank you: this post helped me a lot!

Skyfay · Aug 22, 2024

ElectricCircus said:
Thank you: this post helped me a lot!

I'm glad I could help

eXRuLeZz · Oct 4, 2024

I managed to get it work with a simpler setup (1st IP for Proxmox, 2nd IP for OPNSense running in a VM on that Proxmox host):

/etc/network/interfaces

Code:

source /etc/network/interfaces.d/*


auto lo
iface lo inet loopback


iface eno1 inet manual
    up sysctl -w net.ipv4.ip_forward=1
    up sysctl -w net.ipv4.conf.eno1.send_redirects=0


auto vmbr0
iface vmbr0 inet static
    address x.x.x.188/32 ## This is the 1st IP on the server for Proxmox
    gateway x.x.x.161
    bridge-ports eno1
    bridge-stp off
    bridge-fd 0
    pointopoint x.x.x.161 ## The same as the gateway
# WAN Bridge


auto vmbr1
iface vmbr1 inet manual
    bridge-ports none
    bridge-stp off
    bridge-fd 0
# LAN Bridge


auto vmbr2
iface vmbr2 inet manual
    bridge-ports none
    bridge-stp off
    bridge-fd 0
# DMZ Bridge

As you can see the second IP is not even set in this interface.

Then you create the VM with OPNSense, simply add the MAC address of the 2nd IP (the additional IP) to that VM, then manually set the 2nd IP during OPNSense installation for the WAN interface. For LAN and DMZ you can use 192.168.1.1/24 and 10.0.0.1/24 for example.

The only other think you need to do is to place the following lines in /etc/sysctl.conf :

Code:

# Prevent ARP Conflicts
net.ipv4.conf.all.arp_ignore = 1
net.ipv4.conf.all.arp_announce = 2
net.ipv4.conf.vmbr0.arp_ignore=1
net.ipv4.conf.vmbr0.arp_announce=2

For me it is working perfectly.

Skyfay · Oct 4, 2024

@eXRuLeZz Thanks for the post. I just think your setup is not the same as mine.

I have a Proxmox host and TWO OPNsense firewalls in use. The Proxmox host and the first OPNSense firewall share a public IP (the Proxmox host gets 2 ports needed to work and access the host) and the rest runs via the OPNSense firewall. The second OPNSense firewall then receives the second public IP. Behind the two OPNSense firewalls are two completely different local networks separated from each other with two different public IPs.

In my setup, it would now also be possible to order a third public IP and then assign it to another virtual firewall or directly to a Linux VM.

Search

Search

[SOLVED] Hetzner Server with 2 public IP addresses and OPNSense Firewalls

Skyfay

Member

_gabriel

Renowned Member

Skyfay

Member

_gabriel

Renowned Member

Skyfay

Member

_gabriel

Renowned Member

Skyfay

Member

Skyfay

Member

ElectricCircus

New Member

Skyfay

Member

eXRuLeZz

New Member

Skyfay

Member