opnsense in a vm with primary ip (hetzner)

[NullBy7e]

Has anyone managed to spin up an opnsense VM and assign a primary ip into it from hetzner?
For example consider the primary IP is 100.100.100.50:

You create a bridge like so:

auto enp7s0
iface enp7s0 inet manual

auto vmbr0
iface vmbr0 inet static
        bridge-ports enp7s0
        bridge-stp off
        bridge-fd 0

Then you create a VM and install opnsense on it, whilst configuring it's WAN interface (which is vmbr0) you assign:
ip 100.100.100.50/26 gateway 100.100.100.65.

I personally thought this would work but I ended up with a box I could no longer reach.
And perhaps my knowledge of networking is not good so this scenario might never work at all...

What I also tried is to assign the IP on vmbr0 and create another bridge vmbr1 to act as WAN for the opnsense.
This does work but this means you have to route all your LAN traffic through vmbr1 so opnsense can process it.

I would love to hear some success stories if any ^_^

 

[EenKoekenbakker]

The trick to get this working is to change the MAC of the physical network adapter to something random and than assign the actual physical MAC address to the virtual network adapter of the router VM. I have used the following guide in the past and this works perfectly: https://blog.labx.dev/blog/hetzner/pfSense-in-Proxmox-with-1-IPv4-in-Hetzner-dedi/

 

[NullBy7e]

The trick to get this working is to change the MAC of the physical network adapter to something random and than assign the actual physical MAC address to the virtual network adapter of the router VM. I have used the following guide in the past and this works perfectly: https://blog.labx.dev/blog/hetzner/pfSense-in-Proxmox-with-1-IPv4-in-Hetzner-dedi/

Good idea, I went with a different route though, I created a cloud server with opnsense and I am routing all my traffic through that via a vSwitch. The only problem I have now is that opnsense cannot communicate with my internal networks that are below the vswitch because it's dropping packets for those subnets.

 

[EenKoekenbakker]

Can't think of anything obvious which could cause that, many possibility's however, probably requires more debugging in your specific setup. Although in an afford of not trying to over complicate things, using opnsense on the Proxmox host itself is probably the easiest.

 

[NullBy7e]

Can't think of anything obvious which could cause that, many possibility's however, probably requires more debugging in your specific setup. Although in an afford of not trying to over complicate things, using opnsense on the Proxmox host itself is probably the easiest.

Indeed haha but I'd like to put in some more effort to fix my current situation. I don't know how well versed you are in networking but i will try to explain and perhaps you'll know of something:

Cloud network: 10.10.0.0/16
Gateway: 10.10.0.1
Opnsense: 10.10.0.2

Static route to connect 10.10.0.1 with 10.10.1.1 (this couples cloud with the root server)

vSwitch: 10.10.1.1/24
Root server: 10.10.1.2/24

Internal network on the root server: 10.0.1.0/24 (bridge is 10.0.1.1/24)

Routing table on the root server:

default via 10.10.1.1 dev enp7s0.4000 proto kernel onlink
10.0.1.0/24 dev vmbr1 proto kernel scope link src 10.0.1.1
10.10.1.0/24 dev enp7s0.4000 proto kernel scope link src 10.10.1.2

Alright that's the topology, so there are two scenarios:

The first:
On the root server i apply SNAT to all packets, rewriting the source IP to 10.10.1.2.
the opnsense receives the packets and applies outbound NAT for the WAN interface and after a response is received they are sent back towards 10.10.1.2. This works fine.

The second that is not working:
I configured a static route like so (not sure if needed):
route to: 10.0.1.3
destination: 10.0.1.0
mask: 255.255.255.0
gateway: 10.10.1.2

Works: ping -i vtnet1 10.10.1.2
Does not work: ping -i vtnet1 10.0.1.3 (or any other ip on that subnet)

Tcpdump shows: IP firewall > 10.0.1.3: ICMP echo request, but the root server receives nothing.
Therefore I think that the vSwitch on hetzner is dropping these packets because they are on a different subnet.

It could also be the case that 10.10.0.1 (the gateway of the cloud network) is dropping packets but I can't check that as it's managed by hetzner's infra.

Ideally though if a request to 10.0.1.0/24 is made then I'd need to rewrite the destination IP to 10.10.1.2 but I am not even sure if that's possible.

 

[EenKoekenbakker]

Not much experience in these type of network configurations. As to your theory of the Hetzner vSwitch dropping the packets, maybe their documentation has something to say about this and possible limitations but other than that can't really help with this unfortunately.