PVE with Hetzner and additional subnet

[hoffmn01]

Hello everyone,

I have looked around and did not find the one solution that fits my current setup. Maybe you can help me what needs to be done to enable the additional subnet for my instance.

Here is the general network layout:


I would like to use the additional subnet for several servers in the network. Later I would like to enable ceph-Storage and turn the Proxmox PVE into a clustered setup out of three nodes.

My network config is at the moment (only subnet not working):

auto lo
iface lo inet loopback

auto enp41s0
iface enp41s0 inet static
        address 65.*.*.117/26
        gateway 65.*.*.65
        post-up   echo 1 > /proc/sys/net/ipv4/ip_forward
        post-up iptables -t nat -A PREROUTING -i enp41s0 -p tcp -m multiport ! --dport 4711,8006 -j DNAT --to 10.0.0.2
        post-up iptables -t nat -A PREROUTING -i enp41s0 -p udp -j DNAT --to 10.0.0.2

iface enp41s0.4040 inet manual

auto vmbr0
iface vmbr0 inet manual
        bridge-ports none
        bridge-stp off
        bridge-fd 0

auto vmbr1
iface vmbr1 inet static
        address 10.0.0.1/30
        bridge-ports none
        bridge-stp off
        bridge-fd 0
        post-up   iptables -t nat -A POSTROUTING -s '10.0.0.0/30' -o enp41s0 -j MASQUERADE
        post-down iptables -t nat -D POSTROUTING -s '10.0.0.0/30' -o enp41s0 -j MASQUERADE
        post-up   iptables -t nat -A POSTROUTING -s '10.0.0.0/30' -o vmbr2 -j MASQUERADE
        post-down iptables -t nat -D POSTROUTING -s '10.0.0.0/30' -o vmbr2 -j MASQUERADE

auto vmbr4040
iface vmbr4040 inet static
        address 10.0.10.201/24
        bridge-ports enp41s0.4040
        bridge-stp off
        bridge-fd 0
        bridge-vlan-aware yes
        bridge-vids 2-4094
        mtu 1400

auto vmbr2
iface vmbr2 inet static
  address 95.*.*.153
  netmask 255.255.255.248
  bridge_ports none
  bridge_stp off
  bridge_fd 0

I am not sure, if I need to have all the subnet IPs being mentioned inside the network config. I was hoping only to point one of the network addresses towards the OPNsense firewall, but I may be wrong.

Thank you for any help.

Best regards,
Nico

 

[B.Otto]

That double NAT will cause a lot of headaches...

I suppose in order to allow several subnets it would be best to swap from a single interface enp41s0 to a bridge (with enp41s0 as bridge-port). Something along the lines of:

auto vmbr0
iface vmbr0 inet manual
        bridge-ports enp41s0
        address 65.*.*.117/26
        gateway 65.*.*.65
        bridge-stp off
        bridge-fd 0

(Note: the iptables rules have to be changed so that they apply to vmbr0 then instead of enp41s0)

You then attach the OPNsense firewall to that interface and configure the corresponding IP adresses there - maybe DNAT the mail ports to your internal mail server and a reverse proxy for the web server?


The other way would be to give the proxmox all your new WAN adresses on enp41s0 and configure all the routes (or DNATs) manually, and THEN configure them again on your OPNsense firewall. That sounds like a lot of hassle and is also hard to maintain...

 

[hoffmn01]

Hello @B.Otto,

I followed your thoughts, but I am stuck now:

auto lo
iface lo inet loopback

auto enp41s0
iface enp41s0 inet manual

iface enp41s0.4040 inet manual
#Test

auto vmbr0
iface vmbr0 inet manual
        address 65.*.*.117/26
        gateway 65.*.*.65
        pointopoint 65.*.*.65
        hwaddress a8:a1:59:*:*:*
        bridge-ports enp41s0
        bridge-stp off
        bridge-fd 0
        post-up iptables -t nat -A PREROUTING -i vmbr0 -p tcp -m multiport ! --dport 4711,8006 -j DNAT --to 95.*.*.158
        post-up iptables -t nat -A PREROUTING -i vmbr0 -p udp -j DNAT --to 95.*.*.158

# Subnet
auto vmbr1
iface vmbr1 inet static
        address 95.*.*.153
        netmask 255.255.255.248
        bridge_ports none
        bridge_stp off
        bridge_fd 0

iface vmbr1 inet6 static
  addressi 2a01:*:*::153
  netmask 64
  up ip -6 route add 2a01:*:*::/64 dev vmbr0

The Proxmox host system is available via networking and seems to be fine. But the OPNsense VM is neither willing to reach the internet nor to be pinged from outside. I have the corresponding firewall rules set:

For most parts I followed this instruction set: https://community.hetzner.com/tutor...e-proxmox_ve/de#schritt-2---netzkonfiguration

The configuration of the VM in Proxmox is for the network part as follows:

Maybe there is some bit missing. Does anyone have any idea?

Thank you and best reagards,
Nico

 

[B.Otto]

Hello,

well you seem to have attached your OPNsense WAN interface to vmbr1 and this bridge seems to have no connection so far - or did you configure routes on the PVE?

The Hetzner-network is attached to enp41s0, so to the bridge vmbr0. I think it should be enough if you attach the OPNsense WAN interface to vmbr0.

Traffic from Hetzner with dest 95.*.*.xxx will arrive at vmbr0 and get catched by OPNsense - if it has that IP-Address. On the other way, traffic from your OPNsense to the internet wil be sent to the gateway 95.*.*.153 and that should get catched by Hetzner.

Kind regards,
Benedikt

 

[hoffmn01]

Hello @B.Otto and others listening in,

I did as supposed and attached the OPNsense WAN to the vmbr0 of the Proxmox System:

auto lo
iface lo inet loopback

auto enp41s0
iface enp41s0 inet manual

iface enp41s0.4040 inet manual
#Test

auto vmbr0
iface vmbr0 inet manual
        address 65.*.*.117/26
        gateway 65.*.*.65
        pointopoint 65.*.*.65
        hwaddress a8:a1:59:*:*:*
        bridge-ports enp41s0
        bridge-stp off
        bridge-fd 0
        post-up echo 1 > /proc/sys/net/ipv4/ip_forward

Now the OPNsense is not willing to connect to the internet nor to the gateway from the Hetzner system: 65.*.*.103. I think I am missing some piece now. Also the Subnet IPs are neither working from OPNsense (configured as virtual ip) nor from the outside world.

Do you have any idea?

Best regards,
Nico

 

[rickygm]

Hi , I am in the same situation, and I can't get opnsense to connect to the additional subnet, I need to have three vms behind opnsense and I am configuring a 1:1 nat but I can't get it to work.

did you solve it?