Multiple VM's with only 1 public IP on Hetzner

M

memoria-guy

New Member
Oct 4, 2021
2
0
1
29
Hi everyone,

I am a beginner to this space and am pretty lost on how to setup networking for my needs. I have a bare metal server with Hetzner (AX41) and have gotten Proxmox VE running. Because Hetzner has significantly increased their public IP pricing, I am trying to avoid having to buy a public IP for each VM. My ultimate goal is to have 3 AX41 servers in a cluster with at least 3 VM's on each machine (HA K8s so 1 VM for control plane, 1 VM for SQL DB, 1 VM as worker node). That would be 9 IP's right there and would cost like $600 in set up fees.

So, how should I setup networking with just one public IP? I bought a second public IP for a previous project on this server so I actually have two available. I have read the Hetzner blog post on installing Proxmox, some other posts on this forum, and this guide https://razva.ro/how-to-setup-proxmox-with-hetzner-vswitch-and-public-ip/, but they all seem to recommend getting one public IP per VM. For example this blog recommends a /27 public IP subnet which currently costs $600 to setup and then $30/month.

Does bridged or routed allow me to use just 1 IP?

Thanks!
 
I don't known exactly your setup, but you don't need public ip address on every vms. (database,worker,...).
for example, for k8s, you just need to have a public ip for the ingress pod for incoming traffic, and a public ip what maquerade for outgoing traffic from pod.

Another way : use a pair a vms with haproxy as reverse proxy http/tcp + 1 public ip, and use-it a reverse proxy in front of your private ips k8s cluster && vms. (could be use a nat too for outgoing traffic, vpn server or ssh bastion)
 
Hello. Im also a beginner and i think i have setup masquerading. The problem is after i did the setup, i cant access my proxmox host via SSH (port 22), but i can reach and configure the server via the web interface (port 8006)
I have disabled all the firewalls while testing
Im using Hetzner with a single public IP

My ubuntu VM (10.30.30.10) can access the internet, but my port-forwarding is not working (cant SSH into the VM)

EDIT:
It looks like my VM cant access the internet after all, it could just a few minutes ago but not anymore (i have not changed any settings). So the setup does not seem stable

Update:
So it seems like a reboot was needed, i can now SSH into the Ubuntu VM with port 2210 and the VM has internet access, but i cant SSH into the host on port 22 (connection refused). I could connect to the host on port 22 before i setup bridge network

Update 2:
Im a noob... Forgot i had changed away from the default 22 port in sshd_config... Everything works fine now ^^

Ubuntu VM setup:
IP: 10.30.30.10/24
Gateway: 10.30.30.1
nameserver: 8.8.8.8

/etc/network/interfaces (host)
Code: 
### Hetzner Online GmbH installimage

auto lo
iface lo inet loopback


auto enp9s0
iface enp9s0 inet static
  address **.***.**.87
  hwaddress aa:aa:bb:bb:bb:aa (faked for this example)
  netmask 255.255.255.192
  gateway **.***.**.105


auto vmbr0
iface vmbr0 inet static
  address 10.30.30.1/24
  bridge_ports none
  bridge_stp off
  bridge_fd 0

  post-up   echo 1 > /proc/sys/net/ipv4/ip_forward
  post-up iptables -t nat -A POSTROUTING -s '10.30.30.0/24' -o enp9s0 -j MASQUERADE
  post-down iptables -t nat -D POSTROUTING -s '10.30.30.0/24' -o enp9s0 -j MASQUERADE

  post-up iptables -t nat -A PREROUTING -i enp9s0 -p tcp --dport 2210 -j DNAT --to 10.30.30.10:22
  post-down iptables -t nat -D PREROUTING -i enp9s0 -p tcp --dport 2210 -j DNAT --to 10.30.30.10:22

Any ideas? =)
 
Last edited:
Update:
So it seems like a reboot was needed, i can now SSH into the Ubuntu VM with port 2210 and the VM has internet access, but i cant SSH into the host on port 22 (connection refused). I could connect to the host on port 22 before i setup bridge network

Update 2:
Im a noob... Forgot i had changed away from the default 22 port in sshd_config... Everything works fine now ^^
 
Last edited:
spectryx said:
Hello. Im also a beginner and i think i have setup masquerading. The problem is after i did the setup, i cant access my proxmox host via SSH (port 22), but i can reach and configure the server via the web interface (port 8006)
I have disabled all the firewalls while testing
Im using Hetzner with a single public IP

My ubuntu VM (10.30.30.10) can access the internet, but my port-forwarding is not working (cant SSH into the VM)

EDIT:
It looks like my VM cant access the internet after all, it could just a few minutes ago but not anymore (i have not changed any settings). So the setup does not seem stable

Update:
So it seems like a reboot was needed, i can now SSH into the Ubuntu VM with port 2210 and the VM has internet access, but i cant SSH into the host on port 22 (connection refused). I could connect to the host on port 22 before i setup bridge network

Update 2:
Im a noob... Forgot i had changed away from the default 22 port in sshd_config... Everything works fine now ^^

Ubuntu VM setup:
IP: 10.30.30.10/24
Gateway: 10.30.30.1
nameserver: 8.8.8.8

/etc/network/interfaces (host)
Code: 
### Hetzner Online GmbH installimage

auto lo
iface lo inet loopback


auto enp9s0
iface enp9s0 inet static
  address **.***.**.87
  hwaddress aa:aa:bb:bb:bb:aa (faked for this example)
  netmask 255.255.255.192
  gateway **.***.**.105


auto vmbr0
iface vmbr0 inet static
  address 10.30.30.1/24
  bridge_ports none
  bridge_stp off
  bridge_fd 0

  post-up   echo 1 > /proc/sys/net/ipv4/ip_forward
  post-up iptables -t nat -A POSTROUTING -s '10.30.30.0/24' -o enp9s0 -j MASQUERADE
  post-down iptables -t nat -D POSTROUTING -s '10.30.30.0/24' -o enp9s0 -j MASQUERADE

  post-up iptables -t nat -A PREROUTING -i enp9s0 -p tcp --dport 2210 -j DNAT --to 10.30.30.10:22
  post-down iptables -t nat -D PREROUTING -i enp9s0 -p tcp --dport 2210 -j DNAT --to 10.30.30.10:22

Any ideas? =)
Click to expand...
Thanks for posting this!!! This made me realize my mistake when setting up my VM network. I was unable to connect to the internet but I had set my VM IP address to a /32 subnet and not /24, so I think the VM could not "see" the gateway. It is working now with the public IP! I am going to try using a private IP next :)
 
spectryx said:
Hello. Im also a beginner and i think i have setup masquerading. The problem is after i did the setup, i cant access my proxmox host via SSH (port 22), but i can reach and configure the server via the web interface (port 8006)
I have disabled all the firewalls while testing
Im using Hetzner with a single public IP

My ubuntu VM (10.30.30.10) can access the internet, but my port-forwarding is not working (cant SSH into the VM)

EDIT:
It looks like my VM cant access the internet after all, it could just a few minutes ago but not anymore (i have not changed any settings). So the setup does not seem stable

Update:
So it seems like a reboot was needed, i can now SSH into the Ubuntu VM with port 2210 and the VM has internet access, but i cant SSH into the host on port 22 (connection refused). I could connect to the host on port 22 before i setup bridge network

Update 2:
Im a noob... Forgot i had changed away from the default 22 port in sshd_config... Everything works fine now ^^

Ubuntu VM setup:
IP: 10.30.30.10/24
Gateway: 10.30.30.1
nameserver: 8.8.8.8

/etc/network/interfaces (host)
Code: 
### Hetzner Online GmbH installimage

auto lo
iface lo inet loopback


auto enp9s0
iface enp9s0 inet static
  address **.***.**.87
  hwaddress aa:aa:bb:bb:bb:aa (faked for this example)
  netmask 255.255.255.192
  gateway **.***.**.105


auto vmbr0
iface vmbr0 inet static
  address 10.30.30.1/24
  bridge_ports none
  bridge_stp off
  bridge_fd 0

  post-up   echo 1 > /proc/sys/net/ipv4/ip_forward
  post-up iptables -t nat -A POSTROUTING -s '10.30.30.0/24' -o enp9s0 -j MASQUERADE
  post-down iptables -t nat -D POSTROUTING -s '10.30.30.0/24' -o enp9s0 -j MASQUERADE

  post-up iptables -t nat -A PREROUTING -i enp9s0 -p tcp --dport 2210 -j DNAT --to 10.30.30.10:22
  post-down iptables -t nat -D PREROUTING -i enp9s0 -p tcp --dport 2210 -j DNAT --to 10.30.30.10:22

Any ideas? =)
Click to expand...


Hey @spectryx, thanks for sharing. This was very helpful!

I was wondering what this part does:

Code: 
  post-up iptables -t nat -A PREROUTING -i enp9s0 -p tcp --dport 2210 -j DNAT --to 10.30.30.10:22
  post-down iptables -t nat -D PREROUTING -i enp9s0 -p tcp --dport 2210 -j DNAT --to 10.30.30.10:22
That isn't included in any of the other examples I've seen?

Also, the Proxmox docs example includes a /24 after the public IP address. That's not necessary? https://pve.proxmox.com/wiki/Networ...ith_tt_span_class_monospaced_iptables_span_tt
 
interstellar said:
was wondering what this part does:
post-up iptables -t nat -A PREROUTING -i enp9s0 -p tcp --dport 2210 -j DNAT --to 10.30.30.10:22
post-down iptables -t nat -D PREROUTING -i enp9s0 -p tcp --dport 2210 -j DNAT --to 10.30.30.10:22
Click to expand...
that part is for forwarding a single port, in this case the traffic to port 2210 of the PVE machine is forwarded to the port 22 of the VM/CT on 10.30.30.10

interstellar said:
Also, the Proxmox docs example includes a /24 after the public IP address. That's not necessary?
Click to expand...
that is to create the masquerade routing rule, which in essence allows you to do the port forward shown above.
 
Thanks @oguz :)

For some reason, I'm having trouble with outgoing connectivity on the host. I can't ping google.com, 1.1.1.1 etc. This is my current config:


Code: 
source /etc/network/interfaces.d/*

auto lo
iface lo inet loopback

auto enp0s31f6
#Real IP address
iface enp0s31f6 inet static
        address xxx.xxx.xxx.55/24
        hwaddress aa:bb:cc:dd:ee:ff (fake)
        netmask 255.255.255.xxx
        gateway xxx.xxx.xx.29


auto vmbr0
#Private sub network
iface vmbr0 inet static
        address  10.1.1.1/24
        bridge-ports none
        bridge-stp off
        bridge-fd 0

        post-up   echo 1 > /proc/sys/net/ipv4/ip_forward
        post-up   iptables -t nat -A POSTROUTING -s '10.1.1.0/24' -o eno1 -j MASQUERADE
        post-down iptables -t nat -D POSTROUTING -s '10.1.1.0/24' -o eno1 -j MASQUERADE
It's a Hetzner bare metal server. Not sure what I'm doing wrong. I've tried the public IP with and without the /24, and I've tried it with and without the hwaddress entry.

Do you have any suggestions?
 
Last edited:
interstellar said:
Do you have any suggestions?
Click to expand...
* what's the output from ip a and ip r?

* is your interface called eno1 or enp0s31f6 (you have both in your configuration, those need to be the same)

* can you ping the gateway address in your config file?
 
oguz said:
* what's the output from ip a and ip r?

* is your interface called eno1 or enp0s31f6 (you have both in your configuration, those need to be the same)

* can you ping the gateway address in your config file?
Click to expand...
Oops, I missed the eno1. I've changed those to enp0s31f6 but still not getting any host connectivity.
Here's the output from ip -a:

Code: 
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: enp0s31f6: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether aa:bb:cc:dd:ee:ff brd ff:ff:ff:ff:ff:ff
    inet xxx.xxx.xxx.55/26 scope global enp0s31f6
       valid_lft forever preferred_lft forever
    inet6 fe80::921b:eff:fef3:4ca/64 scope link
       valid_lft forever preferred_lft forever
6: vmbr0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default qlen 1000
    link/ether 6a:f0:18:e0:6b:af brd ff:ff:ff:ff:ff:ff
    inet 10.1.1.1/24 scope global vmbr0
       valid_lft forever preferred_lft forever

I noticed the /26 prefix and tried adding that to the public IP as well. No luck.
 
@oguz This is the default interface file and public connectivity works fine:


Code: 
### Hetzner Online GmbH installimage

source /etc/network/interfaces.d/*

auto lo
iface lo inet loopback
iface lo inet6 loopback

auto enp0s31f6
iface enp0s31f6 inet static
  address xxx.xxx.xxx.55
  netmask 255.255.255.xxx
  gateway xxx.xxx.xxx.29
  # route xxx.xxx.xxx.28/26 via xxx.xxx.xxx.29
  up route add -net xxx.xxx.xxx.28 netmask 255.255.255.xxx gw xxx.xxx.xxx.29 dev enp0s31f6

iface enp0s31f6 inet6 static
  address aaaa:bbb:ccc:dddd::2
  netmask 64
  gateway fe80::1
 
try like this:
Code: 
auto lo
iface lo inet loopback
iface lo inet6 loopback

auto enp0s31f6
iface enp0s31f6 inet static
    address x.x.x.55/26
    gateway x.x.x.29
    up route add -net x.x.x.28 netmask 255.255.255.192 gw x.x.x.29 dev enp0s31f6

auto vmbr0
iface vmbr0 inet static
    hwaddress aa:bb:cc:dd:ee:ff
    address 10.1.1.1/24
    bridge-ports none
    bridge-stp off
    bridge-fd 0
    post up echo 1 > /proc/sys/net/ipv4/ip_forward
    post-up   iptables -t nat -A POSTROUTING -s '10.1.1.0/24' -o enp0s31f6 -j MASQUERADE
    post-down iptables -t nat -D POSTROUTING -s '10.1.1.0/24' -o enp0s31f6 -j MASQUERADE
 
You must log in or register to reply here.
Top Bottom