[TUTORIAL] Proxmox @ Hetzner, using a single public IPv4 address (+IPv6/64) while all traffic, including host goes through virtualized Firewall (ex. Pfsense)

effgee

Renowned Member
Jul 29, 2013
44
15
73
I'm a bit short on time and I was hesitant to share this as recently Hetzner started sending out MAC abuse messages which took me a long time to resolve as it was both a Proxmox issue, which was fixed with PVE 7, and also an issue with this configuration which I finally resolved.

I have been using this configuration for quite a few years and it works great.

Proxmox built in firewall is good, but a bit inflexible as well as I like the VPN functionality (and simplifications) of Firewall dedicated vms. I also wanted to access both my port mapped virtual machines as well as the host with a single public ip, protected by my virtualized firewall of choice.

Most people eventually break down and buy extra ipv4 address, ex. 1 ip for host, 1 for virtualized firewall + vms. With this technique, an extra ip is not necessary

You can have your cake and eat it too.

Quick Overview

This is not going to be a step by step, you will need to be able to understand the configuration here and conceptually understand how to apply the configuration.
Oftentimes a mistake will force you to reboot into rescue mode and reconfigure you network.

Also, understand, since all traffic will be forced through a virtual firewall, if it malfunctions, you will not have access to the host. Additionally, during a reboot, until the virtual firewall has booted, your port mappings to the host will not be functional as well.

I keep two copies of my interfaces file,
  • vm-firewall-as-public-interface interfaces file
  • and 1 for the standard-baremetal-config where the host has the public ip (standard setup).


Create your firewall VM, with 2 virtual network interfaces.
  • Virtual FW, eth1 will be the "LAN" connected to vmbr1
  • Virtual FW, eth0, will be the firewall WAN port, connected to vmbr0.
    • Its IP (4/6) settings should match what you normally would set the host physical card to.
    • Because Hetzner switch ports only answer traffic from the registered physical MAC address, we need to push the physical MAC address into a virtual network adapter attached to the Firewall VM.
      • Setup the MAC address of Virtual FW, eth0 to match your physical cards MAC. (Pro-tip: In the VM config, leave this port unplugged under advanced while you configure it, only virtually plug it in while the firewall is turned off before you reboot the PVE host to swap into the vm-firewall-as-public-interface interfaces, otherwise you gonna have a bad time)


/etc/network/interfaces - vm-firewall-as-public-interface interfaces file

source /etc/network/interfaces.d/* auto lo iface lo inet loopback iface lo inet6 loopback # Bring up the physical interface auto enp3s0 # Set the physical interface to manual networking configuration, but we DO NOT assign it any IPs. iface enp3s0 inet manual # We rewrite any bridge traffic leaving through the physical adapter (enp3s0) to the real MAC to avoid generating Hetzner Mac abuse warnings via its switches seeing ethernet frames with "wrong" MAC address.. ex Real MAC address: aa:bb:cc:dd:ee pre-up ebtables -t nat -A POSTROUTING -j snat --to-src aa:bb:cc:dd:ee -o enp3s0 # Change the physical cards MAC address to a generated/fake MAC, so it frees up the real MAC address to be passed through to the virtual Firewall. pre-up ifconfig enp3s0 hw ether 00:11:22:33:44:55 auto vmbr0 iface vmbr0 inet manual bridge-ports enp3s0 bridge-stp off bridge-fd 0 bridge_maxwait 0 auto vmbr1 iface vmbr1 inet static address 192.168.4.3/24 bridge-ports none bridge-stp off bridge-fd 0


And on your firewall, suggest port mapping at the very least 22, and 8006 to your PVE host, via its vmbr1 LAN address.

ex
Public ip is 1.2.3.4
PVE host vmbr1 LAN is: 192.168.4.3 (like above config)

On your virtual firewall, port forward from your public interface, 1.2.3.4 port 22,8006 to 192.168.4.3 port 22,8006

Again, this is not a full step by step, there are many considerations such as fully understanding the network configuration process, understanding when to reboot and activate/swap configurations, but this config works well for me for many years. I really like having all my traffic forced through my virtualized firewall, including host traffic.
 
Just wanted to say: you sir, are a life saver!

With your config I was able to get my virtual pfSense reachable on a Hetzner host. Thanks!
 
  • Like
Reactions: effgee
Thanks so much for this! I'm just getting into using hetzner and this was exactly what I needed!!
 
I'm a bit short on time and I was hesitant to share this as recently Hetzner started sending out MAC abuse messages which took me a long time to resolve as it was both a Proxmox issue, which was fixed with PVE 7, and also an issue with this configuration which I finally resolved.

I have been using this configuration for quite a few years and it works great.

Proxmox built in firewall is good, but a bit inflexible as well as I like the VPN functionality (and simplifications) of Firewall dedicated vms. I also wanted to access both my port mapped virtual machines as well as the host with a single public ip, protected by my virtualized firewall of choice.

Most people eventually break down and buy extra ipv4 address, ex. 1 ip for host, 1 for virtualized firewall + vms. With this technique, an extra ip is not necessary

You can have your cake and eat it too.

Quick Overview

This is not going to be a step by step, you will need to be able to understand the configuration here and conceptually understand how to apply the configuration.
Oftentimes a mistake will force you to reboot into rescue mode and reconfigure you network.

Also, understand, since all traffic will be forced through a virtual firewall, if it malfunctions, you will not have access to the host. Additionally, during a reboot, until the virtual firewall has booted, your port mappings to the host will not be functional as well.

I keep two copies of my interfaces file,
  • vm-firewall-as-public-interface interfaces file
  • and 1 for the standard-baremetal-config where the host has the public ip (standard setup).


Create your firewall VM, with 2 virtual network interfaces.
  • Virtual FW, eth1 will be the "LAN" connected to vmbr1
  • Virtual FW, eth0, will be the firewall WAN port, connected to vmbr0.
    • Its IP (4/6) settings should match what you normally would set the host physical card to.
    • Because Hetzner switch ports only answer traffic from the registered physical MAC address, we need to push the physical MAC address into a virtual network adapter attached to the Firewall VM.
      • Setup the MAC address of Virtual FW, eth0 to match your physical cards MAC. (Pro-tip: In the VM config, leave this port unplugged under advanced while you configure it, only virtually plug it in while the firewall is turned off before you reboot the PVE host to swap into the vm-firewall-as-public-interface interfaces, otherwise you gonna have a bad time)


/etc/network/interfaces - vm-firewall-as-public-interface interfaces file

source /etc/network/interfaces.d/* auto lo iface lo inet loopback iface lo inet6 loopback # Bring up the physical interface auto enp3s0 # Set the physical interface to manual networking configuration, but we DO NOT assign it any IPs. iface enp3s0 inet manual # We rewrite any bridge traffic leaving through the physical adapter (enp3s0) to the real MAC to avoid generating Hetzner Mac abuse warnings via its switches seeing ethernet frames with "wrong" MAC address.. ex Real MAC address: aa:bb:cc:dd:ee pre-up ebtables -t nat -A POSTROUTING -j snat --to-src aa:bb:cc:dd:ee -o enp3s0 # Change the physical cards MAC address to a generated/fake MAC, so it frees up the real MAC address to be passed through to the virtual Firewall. pre-up ifconfig enp3s0 hw ether 00:11:22:33:44:55 auto vmbr0 iface vmbr0 inet manual bridge-ports enp3s0 bridge-stp off bridge-fd 0 bridge_maxwait 0 auto vmbr1 iface vmbr1 inet static address 192.168.4.3/24 bridge-ports none bridge-stp off bridge-fd 0


And on your firewall, suggest port mapping at the very least 22, and 8006 to your PVE host, via its vmbr1 LAN address.

ex
Public ip is 1.2.3.4
PVE host vmbr1 LAN is: 192.168.4.3 (like above config)

On your virtual firewall, port forward from your public interface, 1.2.3.4 port 22,8006 to 192.168.4.3 port 22,8006

Again, this is not a full step by step, there are many considerations such as fully understanding the network configuration process, understanding when to reboot and activate/swap configurations, but this config works well for me for many years. I really like having all my traffic forced through my virtualized firewall, including host traffic.
Good evening sir and thank you for this informative tutorial.

Is there any reason I couldn't use this same setup on my personal ProxMox box (with 6 onboard NICs) at home?

The primary difference between my current setup and most of the other tutorials/scenarios I see around online is that I have 5 of my 6 NICs' hardware passed directly through to pfSense via PCI already (WAN/LAN/OPT1/OPT2/OPT3), thus making it difficult to follow along with the virtual bridge setups.

My 1st NIC is statically mapped to an un-used ip address outside of pfSense's current network scope and is currently serving as my offline management access port to get back to ProxMox's web GUI in the event I screw something up too bad that I can't correct from CLI. I also have HDMI out from my ProxMox box to my main PC monitor with PiP and another keyboard for CLI fixes.
 
Good evening sir and thank you for this informative tutorial.

Is there any reason I couldn't use this same setup on my personal ProxMox box (with 6 onboard NICs) at home?

The primary difference between my current setup and most of the other tutorials/scenarios I see around online is that I have 5 of my 6 NICs' hardware passed directly through to pfSense via PCI already (WAN/LAN/OPT1/OPT2/OPT3), thus making it difficult to follow along with the virtual bridge setups.

My 1st NIC is statically mapped to an un-used ip address outside of pfSense's current network scope and is currently serving as my offline management access port to get back to ProxMox's web GUI in the event I screw something up too bad that I can't correct from CLI. I also have HDMI out from my ProxMox box to my main PC monitor with PiP and another keyboard for CLI fixes.
You shouldn't need to use this setup at home. This is only needed because Hetzner bind the MAC address to the IP they provided.

I'm running proxmox with pfsense at home and I use bridges instead of passing the NICs through.
 
  • Like
Reactions: maleko48
You shouldn't need to use this setup at home. This is only needed because Hetzner bind the MAC address to the IP they provided.

I'm running proxmox with pfsense at home and I use bridges instead of passing the NICs through.
Yes, I understand that much I suppose. Just trying to understand how I can achieve the same end result with my NICs passed through directly to pfSense rather than virtualized. I want my ProxMox host to be able to access the internet for updates, but do not want it at the front of my network, exposed and would prefer everything be behind pfSense and segmented as securely as possible. Trying to get the base setup correctly before building on top of it all with more. ( Sorry for my networking ignorance, it makes my head spin a bit and I have limited Linux experience as well... I recently dove into all this head first, bull in a china shop style. )
 
Last edited:
Yes, I understand that much I suppose. Just trying to understand how I can achieve the same end result with my NICs passed through directly to pfSense rather than virtualized. I want my ProxMox host to be able to access the internet for updates, but do not want it at the front of my network, exposed and would prefer everything be behind pfSense and segmented as securely as possible. Trying to get the base setup correctly before building on top of it all with more. ( Sorry for my networking ignorance, it makes my head spin a bit and I have limited Linux experience as well... I recently dove into all this head first, bull in a china shop style. )
You don't have to pass the NIC's through to get proxmox behind pfsense. That's how I have it setup. You can PM for help and let's not hijack this thread.
 
But there are still a few things missing: I still have to install net-tools and still run ip route default gw (ip from router lan) dev vmbr1 because the pve wants everything via vmbr0 routes what is not possible since no ip available
 
I've read this guide multiple times but I still cannot understand the steps I need to go to make sure my setup is right. Here is my understanding from the post:

1. I need to create the bridge settings.
2. The bridges should not have public ip address assigned to them at the beginning.
3. I need to up a firewall image, assign the main bridge as WAN and the other one as LAN.
4. I need to assign a public IP (and the main MAC) to the main bridge.
5. I need to make sure the outgoing traffic gets rewritten with the valid MAC address.
6. I need to route incoming traffic and outgoing traffic as the main IP address using NAT.

Is this the correct understanding? It would be very helpful if you could write more details into this.

Sorry for the vague post, I'm a total newbie to proxmox configurations.
 
Hey - does anyone know if the latest OPNSense 22.7 breaks this? I updated to that and ... nothing has worked since. :(
 
I'm a bit short on time and I was hesitant to share this as recently Hetzner started sending out MAC abuse messages which took me a long time to resolve as it was both a Proxmox issue, which was fixed with PVE 7, and also an issue with this configuration which I finally resolved.

I have been using this configuration for quite a few years and it works great.

Proxmox built in firewall is good, but a bit inflexible as well as I like the VPN functionality (and simplifications) of Firewall dedicated vms. I also wanted to access both my port mapped virtual machines as well as the host with a single public ip, protected by my virtualized firewall of choice.

Most people eventually break down and buy extra ipv4 address, ex. 1 ip for host, 1 for virtualized firewall + vms. With this technique, an extra ip is not necessary

You can have your cake and eat it too.
Thanks a lot @effgee ! Wonderful! I was looking exactly for that setup.

Btw, i use the vswitches of Hetzner to connect multiple servers together.
And one of the vswitches is connected even to a Hetzner cloud VM network, so I use a cloud VM as jump host to reach my Proxmox server even if there is an issue with the Proxmox Firewall VM :)

And I have the root disk encrypted, which needs to be "unlocked" at boot time, whereas the ZFS encrypted volumes get auto-unlocked with a key on the (encrpyted) root disk.

Merry Christmas, what a great present!
 
Last edited:

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!