Hetzner Routed Network (3 Subnets + Additional IPs) Config Check

M

MisterNobody

Member
Oct 31, 2020
10
0
6
37
Hey,

what is the best way to route the maximum number of IP addresses to the guests (IPV4)?
I have 3 Subnets (/29, /27, /27) and an additional IP that I wish to fully utilize (8, 32, 32 IPs) without loss of an IP for the host system and broadcast/subnet.

I have theorized this config but haven't had the chance to actually test it yet:

IP Overview:

Code: 
Host IP: 59.12.13.174
Hetzner Gateway: 59.12.13.129
Subnet 1: 134.243.210.24/29 (routed over Host IP)
Subnet 2: 136.113.218.128/27 (routed over Host IP)
Subnet 3: 178.13.236.32/27 (routed over Host IP)
IPv6 Subnet: 2a01:4f8:251:1faf::/64
Additional IP: 49.14.25.161/32 (routed to Host Mac)
vmbr0 for normal VM connections
vmbr1 for non-reachable VMs (only outgoing)




Host Config:

Code: 
# Loopback device:
auto lo
iface lo inet loopback
iface lo inet6 loopback

# device: enp35s0
auto enp35s0
iface enp35s0 inet static
  # Single Host IPv4 address
  address   59.12.13.174
  netmask   255.255.255.192
  # The Hetzner Gateway IP
  gateway   59.12.13.129
  post-up echo 1 > /proc/sys/net/ipv4/ip_forward
  post-up echo 1 > /proc/sys/net/ipv4/conf/enp35s0/proxy_arp

  # route  59.12.13.174/26 via 59.12.13.129
  up route add -net 59.12.13.0 netmask 255.255.255.192 gw 59.12.13.129 dev enp35s0

  offload-rx  off
  offload-tx  off
  offload-sg  off
  offload-tso off

iface enp35s0 inet6 static
  # We gave our Host a single IPv6
  address 2a01:4f8:251:1faf::2
  # /128 because our Bridge has the whole /64 Subnet!
  netmask 128
  # The Hetzner IPv6 Gateway
  gateway fe80::1

# VM-Bridge used by Proxmox Guests
auto vmbr0
iface vmbr0 inet static
  # Host will be the Gateway for all VM's
  address 59.12.13.174
  netmask 255.255.255.192
  # Some basic bridge configuration
  bridge_stp off
  bridge_ports none
  bridge_fd 0
  bridge_maxwait 0

  # Add all single IP's from the /29 subnet
  # Test Subnet address
  up route add -host 134.243.210.24 dev vmbr0

  up route add -host 134.243.210.25 dev vmbr0
  up route add -host 134.243.210.26 dev vmbr0
  up route add -host 134.243.210.27 dev vmbr0
  up route add -host 134.243.210.28 dev vmbr0
  up route add -host 134.243.210.29 dev vmbr0
  up route add -host 134.243.210.30 dev vmbr0

  # Test Broadcast address
  up route add -host 134.243.210.31 dev vmbr0

  # Add all single IP's from the first /27 subnet
  # Test Subnet address
  up route add -host 136.113.218.128 dev vmbr0

  up route add -host 136.113.218.129 dev vmbr0
  up route add -host 136.113.218.130 dev vmbr0
  up route add -host 136.113.218.131 dev vmbr0
  up route add -host 136.113.218.132 dev vmbr0
  up route add -host 136.113.218.133 dev vmbr0
  up route add -host 136.113.218.134 dev vmbr0
  up route add -host 136.113.218.135 dev vmbr0
  up route add -host 136.113.218.136 dev vmbr0
  up route add -host 136.113.218.137 dev vmbr0
  up route add -host 136.113.218.138 dev vmbr0
  up route add -host 136.113.218.139 dev vmbr0
  up route add -host 136.113.218.140 dev vmbr0
  up route add -host 136.113.218.141 dev vmbr0
  up route add -host 136.113.218.142 dev vmbr0
  up route add -host 136.113.218.143 dev vmbr0
  up route add -host 136.113.218.144 dev vmbr0
  up route add -host 136.153.218.145 dev vmbr0
  up route add -host 136.113.218.146 dev vmbr0
  up route add -host 136.113.218.147 dev vmbr0
  up route add -host 136.113.218.148 dev vmbr0
  up route add -host 136.113.218.149 dev vmbr0
  up route add -host 136.113.218.150 dev vmbr0
  up route add -host 136.113.218.151 dev vmbr0
  up route add -host 136.113.218.152 dev vmbr0
  up route add -host 136.113.218.153 dev vmbr0
  up route add -host 136.113.218.154 dev vmbr0
  up route add -host 136.113.218.155 dev vmbr0
  up route add -host 136.113.218.156 dev vmbr0
  up route add -host 136.113.218.157 dev vmbr0
  up route add -host 136.113.218.158 dev vmbr0

  # Test Broadcast address
  up route add -host 136.113.218.159 dev vmbr0

  # Add all single IP's from the second /27 subnet
  # Test Subnet address
  up route add -host 178.13.236.32 dev vmbr0

  up route add -host 178.13.236.33 dev vmbr0
  up route add -host 178.13.236.34 dev vmbr0
  up route add -host 178.13.236.35 dev vmbr0
  up route add -host 178.13.236.36 dev vmbr0
  up route add -host 178.13.236.37 dev vmbr0
  up route add -host 178.13.236.38 dev vmbr0
  up route add -host 178.13.236.39 dev vmbr0
  up route add -host 178.13.236.40 dev vmbr0
  up route add -host 178.13.236.41 dev vmbr0
  up route add -host 178.13.236.42 dev vmbr0
  up route add -host 178.13.236.43 dev vmbr0
  up route add -host 178.13.236.44 dev vmbr0
  up route add -host 178.13.236.45 dev vmbr0
  up route add -host 178.13.236.46 dev vmbr0
  up route add -host 178.13.236.47 dev vmbr0
  up route add -host 178.13.236.48 dev vmbr0
  up route add -host 178.13.236.49 dev vmbr0
  up route add -host 178.13.236.50 dev vmbr0
  up route add -host 178.13.236.51 dev vmbr0
  up route add -host 178.13.236.52 dev vmbr0
  up route add -host 178.13.236.53 dev vmbr0
  up route add -host 178.13.236.54 dev vmbr0
  up route add -host 178.13.236.55 dev vmbr0
  up route add -host 178.13.236.56 dev vmbr0
  up route add -host 178.13.236.57 dev vmbr0
  up route add -host 178.13.236.58 dev vmbr0
  up route add -host 178.13.236.59 dev vmbr0
  up route add -host 178.13.236.60 dev vmbr0
  up route add -host 178.13.236.61 dev vmbr0
  up route add -host 178.13.236.62 dev vmbr0

  # Test Broadcast address
  up route add -host 178.13.236.63 dev vmbr0


  # Additional IP
  up route add -host 49.14.25.161 dev vmbr0


iface vmbr0 inet6 static
  # Host as the Gateway
  address 2a01:4f8:251:1faf::2
  # route the whole /64 subnet
  netmask 64

# Private Network
auto vmbr1
iface vmbr1 inet static
    address 10.20.30.1
    netmask 255.255.255.0
    bridge_ports none
    bridge_stp off
    bridge_fd 0
    post-up iptables -t nat -A POSTROUTING -s '10.20.30.0/24' -o eth0 -j MASQUERADE
    post-down iptables -t nat -D POSTROUTING -s '10.20.30.0/24' -o eth0 -j MASQUERADE

I would also set:
Code: 
net.ipv4.ip_forward=1
net.ipv6.conf.all.forwarding=1
net.ipv4.conf.all.send_redirects=0


Guests (eth0 bridged to vmbr0) would get a config along the lines of:

Code: 
# Loopback device:
auto lo
iface lo inet loopback
iface lo inet6 loopback

allow-hotplug eth0
iface eth0 inet static
  # IPv4 address from Subnet 3
  address   178.13.236.61
  netmask   255.255.255.255
  gateway   59.12.13.174
  pointopoint 59.12.13.174

iface eth0 inet6 static
  # IPv6 from /64 subnet
  address 2a01:4f8:251:1faf::3
  netmask 64
  gateway 2a01:4f8:251:1faf::2

I currently route the subnets through the additional IP to a pfSense instance but would like to go back to plain Linux for routing.

Is the above viable? What is the alternative?

I saw many configs which spawn an own bridge per Subnet (the Hetzner documentation, or using the script by schaal-it), but that way I would lose 3 IPs per subnet (1x Subnet, 1x Host, 1x Broadcast) before even assigning one to the VM, which is currently not the case with pfSense

I would test it before asking but I have to schedule a maintenance window for the changes so I would like to know if I'm on the right track.
 
Last edited:
Hey There,

I would do it a little bit different

Host Config

Code: 
# Loopback device:
auto lo
iface lo inet loopback
iface lo inet6 loopback

# device: enp35s0
auto enp35s0
iface enp35s0 inet static
  # Single Host IPv4 address
  address   59.12.13.174/32
  gateway   59.12.13.129
  pointopoint   59.12.13.129
  post-up echo 1 > /proc/sys/net/ipv4/ip_forward
  post-up echo 1 > /proc/sys/net/ipv4/conf/enp35s0/proxy_arp


  offload-rx  off
  offload-tx  off
  offload-sg  off
  offload-tso off

iface enp35s0 inet6 static
  # We gave our Host a single IPv6
  address 2a01:4f8:251:1faf::2
  # /128 because our Bridge has the whole /64 Subnet!
  netmask 128
  # The Hetzner IPv6 Gateway
  gateway fe80::1

# VM-Bridge used by Proxmox Guests
auto vmbr0
iface vmbr0 inet static
  # Host will be the Gateway for all VM's
  address 192.168.0.1
  netmask 255.255.255.192
  # Some basic bridge configuration
  bridge_stp off
  bridge_ports none
  bridge_fd 0
  bridge_maxwait 0

  up route add -host 134.243.210.24/29 dev vmbr0 scope link
  up route add -host 136.113.218.128/27 dev vmbr0 scope link
  up route add -host 178.13.236.32/27 dev vmbr0 scope link
  up route add -host 49.14.25.161 dev vmbr0 scope link


iface vmbr0 inet6 static
  # Host as the Gateway
  address 2a01:4f8:251:1faf::2
  # route the whole /64 subnet
  netmask 64

# Private Network
auto vmbr1
iface vmbr1 inet static
    address 10.20.30.1
    netmask 255.255.255.0
    bridge_ports none
    bridge_stp off
    bridge_fd 0
    post-up iptables -t nat -A POSTROUTING -s '10.20.30.0/24' -o eth0 -j MASQUERADE
    post-down iptables -t nat -D POSTROUTING -s '10.20.30.0/24' -o eth0 -j MASQUERADE

On the VMs you can use

Code: 
# Loopback device:
auto lo
iface lo inet loopback
iface lo inet6 loopback

allow-hotplug eth0
iface eth0 inet static
  # IPv4 address from Subnet 3
  address   178.13.236.61
  netmask   255.255.255.255
  gateway   192.168.0.1
  pointopoint 192.168.0.1

iface eth0 inet6 static
  # IPv6 from /64 subnet
  address 2a01:4f8:251:1faf::3
  netmask 64
  gateway 2a01:4f8:251:1faf::2

I suggest you to use a RFC1918 address for the PoinToPoint Gateway, so if you are ever change the Hostsystem and the main IP changes you don't have to change the config of all your VMs...
 
  • Like
Reactions: kolaps and MisterNobody
Hey,

that makes sense and is way cleaner. Didn't know you could use CIDR notation :)

I will give it a try and then update this thread
 
Sorry to hijack the thread & excuse my ignorance .. I've tried adding an additional subnet (vmbr4) with a /29 the way described above by @jlebherz
but my server doesn't want me to add 'scope link' to the route. What am I missing?
 
try it like this

Code: 
auto vmbr4
iface vmbr4 inet static
  address 192.168.0.1/31
  bridge_stp off
  bridge_ports none
  bridge_fd 0
  bridge_maxwait 0

  up ip route add 134.243.210.24/29 via 192.168.0.1 dev vmbr4 scope link
 
  • Like
Reactions: iworx
jlebherz said:
well I think there was a mistake on my first post you should use iproute2

Code: 
up ip route add 134.243.210.24/29 via 192.168.0.1 dev vmbr4 scope link
Click to expand...
Thanks for replying so swift! Much appreciated.
It's added now without error. My debian test server on that IF still doesn't want to connect -- but I think that might be was an issue on my side.
This is so much simpler! Thanks so much.
 
Last edited:
on your client it should look like this
Code: 
root@test:~# ip route
default via 192.168.0.1 dev ens18 proto kernel onlink
192.168.0.1 dev ens18 proto kernel scope link src 1.2.3.4

and on your Proxmox Host
Code: 
root@host:~# ip route
192.168.0.0/31 dev vmbr4 proto kernel scope link src 192.168.0.1
1.2.3.0/29 via 192.168.0.1 dev vmbr4 scope link
 
jlebherz said:
and make sure, that ip_forward is enabled!
that must result 1
Code: 
cat /proc/sys/net/ipv4/ip_forward
Click to expand...
On the host OS you mean? That's working because my other hosts are working too.
I got it working for a sec .. rebooted and it's gone again. I'm missing something obvious.
 
iworx said:
On the host OS you mean? That's working because my other hosts are working too.
I got it working for a sec .. rebooted and it's gone again. I'm missing something obvious.
Click to expand...
Got it working... stupid issue in resolv.conf :/
Thanks again for your time and help.
 
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom