Multiple VM's with only 1 public IP on Hetzner

memoria-guy · Oct 4, 2021

Hi everyone,

I am a beginner to this space and am pretty lost on how to setup networking for my needs. I have a bare metal server with Hetzner (AX41) and have gotten Proxmox VE running. Because Hetzner has significantly increased their public IP pricing, I am trying to avoid having to buy a public IP for each VM. My ultimate goal is to have 3 AX41 servers in a cluster with at least 3 VM's on each machine (HA K8s so 1 VM for control plane, 1 VM for SQL DB, 1 VM as worker node). That would be 9 IP's right there and would cost like $600 in set up fees.

So, how should I setup networking with just one public IP? I bought a second public IP for a previous project on this server so I actually have two available. I have read the Hetzner blog post on installing Proxmox, some other posts on this forum, and this guide https://razva.ro/how-to-setup-proxmox-with-hetzner-vswitch-and-public-ip/, but they all seem to recommend getting one public IP per VM. For example this blog recommends a /27 public IP subnet which currently costs $600 to setup and then $30/month.

Does bridged or routed allow me to use just 1 IP?

Thanks!

oguz · Oct 5, 2021

hi,

memoria-guy said:
So, how should I setup networking with just one public IP?

you need a masquerade NAT setup [0]

in that scenario your PVE host will be forwarding the necessary ports to the internal IP addresses of the guests in the private network you create.

[0]: https://pve.proxmox.com/wiki/Networ...ith_tt_span_class_monospaced_iptables_span_tt

spirit · Oct 6, 2021

I don't known exactly your setup, but you don't need public ip address on every vms. (database,worker,...).
for example, for k8s, you just need to have a public ip for the ingress pod for incoming traffic, and a public ip what maquerade for outgoing traffic from pod.

Another way : use a pair a vms with haproxy as reverse proxy http/tcp + 1 public ip, and use-it a reverse proxy in front of your private ips k8s cluster && vms. (could be use a nat too for outgoing traffic, vpn server or ssh bastion)

spectryx · Oct 9, 2021

Hello. Im also a beginner and i think i have setup masquerading. The problem is after i did the setup, i cant access my proxmox host via SSH (port 22), but i can reach and configure the server via the web interface (port 8006)
I have disabled all the firewalls while testing
Im using Hetzner with a single public IP

My ubuntu VM (10.30.30.10) can access the internet, but my port-forwarding is not working (cant SSH into the VM)

EDIT:
It looks like my VM cant access the internet after all, it could just a few minutes ago but not anymore (i have not changed any settings). So the setup does not seem stable

Update:
So it seems like a reboot was needed, i can now SSH into the Ubuntu VM with port 2210 and the VM has internet access, but i cant SSH into the host on port 22 (connection refused). I could connect to the host on port 22 before i setup bridge network

Update 2:
Im a noob... Forgot i had changed away from the default 22 port in sshd_config... Everything works fine now ^^

Ubuntu VM setup:
IP: 10.30.30.10/24
Gateway: 10.30.30.1
nameserver: 8.8.8.8

/etc/network/interfaces (host)

Code:

### Hetzner Online GmbH installimage

auto lo
iface lo inet loopback


auto enp9s0
iface enp9s0 inet static
  address **.***.**.87
  hwaddress aa:aa:bb:bb:bb:aa (faked for this example)
  netmask 255.255.255.192
  gateway **.***.**.105


auto vmbr0
iface vmbr0 inet static
  address 10.30.30.1/24
  bridge_ports none
  bridge_stp off
  bridge_fd 0

  post-up   echo 1 > /proc/sys/net/ipv4/ip_forward
  post-up iptables -t nat -A POSTROUTING -s '10.30.30.0/24' -o enp9s0 -j MASQUERADE
  post-down iptables -t nat -D POSTROUTING -s '10.30.30.0/24' -o enp9s0 -j MASQUERADE

  post-up iptables -t nat -A PREROUTING -i enp9s0 -p tcp --dport 2210 -j DNAT --to 10.30.30.10:22
  post-down iptables -t nat -D PREROUTING -i enp9s0 -p tcp --dport 2210 -j DNAT --to 10.30.30.10:22

Any ideas? =)

spectryx · Oct 9, 2021

Update:
So it seems like a reboot was needed, i can now SSH into the Ubuntu VM with port 2210 and the VM has internet access, but i cant SSH into the host on port 22 (connection refused). I could connect to the host on port 22 before i setup bridge network

Update 2:
Im a noob... Forgot i had changed away from the default 22 port in sshd_config... Everything works fine now ^^

memoria-guy · Oct 12, 2021

spectryx said:
Hello. Im also a beginner and i think i have setup masquerading. The problem is after i did the setup, i cant access my proxmox host via SSH (port 22), but i can reach and configure the server via the web interface (port 8006)
I have disabled all the firewalls while testing
Im using Hetzner with a single public IP

My ubuntu VM (10.30.30.10) can access the internet, but my port-forwarding is not working (cant SSH into the VM)

EDIT:
It looks like my VM cant access the internet after all, it could just a few minutes ago but not anymore (i have not changed any settings). So the setup does not seem stable

Update:
So it seems like a reboot was needed, i can now SSH into the Ubuntu VM with port 2210 and the VM has internet access, but i cant SSH into the host on port 22 (connection refused). I could connect to the host on port 22 before i setup bridge network

Update 2:
Im a noob... Forgot i had changed away from the default 22 port in sshd_config... Everything works fine now ^^

Ubuntu VM setup:
IP: 10.30.30.10/24
Gateway: 10.30.30.1
nameserver: 8.8.8.8

/etc/network/interfaces (host)

Code:

### Hetzner Online GmbH installimage auto lo iface lo inet loopback auto enp9s0 iface enp9s0 inet static address **.***.**.87 hwaddress aa:aa:bb:bb:bb:aa (faked for this example) netmask 255.255.255.192 gateway **.***.**.105 auto vmbr0 iface vmbr0 inet static address 10.30.30.1/24 bridge_ports none bridge_stp off bridge_fd 0 post-up echo 1 > /proc/sys/net/ipv4/ip_forward post-up iptables -t nat -A POSTROUTING -s '10.30.30.0/24' -o enp9s0 -j MASQUERADE post-down iptables -t nat -D POSTROUTING -s '10.30.30.0/24' -o enp9s0 -j MASQUERADE post-up iptables -t nat -A PREROUTING -i enp9s0 -p tcp --dport 2210 -j DNAT --to 10.30.30.10:22 post-down iptables -t nat -D PREROUTING -i enp9s0 -p tcp --dport 2210 -j DNAT --to 10.30.30.10:22

Any ideas? =)

Thanks for posting this!!! This made me realize my mistake when setting up my VM network. I was unable to connect to the internet but I had set my VM IP address to a /32 subnet and not /24, so I think the VM could not "see" the gateway. It is working now with the public IP! I am going to try using a private IP next

interstellar · Jul 20, 2022

spectryx said:
Hello. Im also a beginner and i think i have setup masquerading. The problem is after i did the setup, i cant access my proxmox host via SSH (port 22), but i can reach and configure the server via the web interface (port 8006)
I have disabled all the firewalls while testing
Im using Hetzner with a single public IP

My ubuntu VM (10.30.30.10) can access the internet, but my port-forwarding is not working (cant SSH into the VM)

EDIT:
It looks like my VM cant access the internet after all, it could just a few minutes ago but not anymore (i have not changed any settings). So the setup does not seem stable

Update:
So it seems like a reboot was needed, i can now SSH into the Ubuntu VM with port 2210 and the VM has internet access, but i cant SSH into the host on port 22 (connection refused). I could connect to the host on port 22 before i setup bridge network

Update 2:
Im a noob... Forgot i had changed away from the default 22 port in sshd_config... Everything works fine now ^^

Ubuntu VM setup:
IP: 10.30.30.10/24
Gateway: 10.30.30.1
nameserver: 8.8.8.8

/etc/network/interfaces (host)

Code:

### Hetzner Online GmbH installimage auto lo iface lo inet loopback auto enp9s0 iface enp9s0 inet static address **.***.**.87 hwaddress aa:aa:bb:bb:bb:aa (faked for this example) netmask 255.255.255.192 gateway **.***.**.105 auto vmbr0 iface vmbr0 inet static address 10.30.30.1/24 bridge_ports none bridge_stp off bridge_fd 0 post-up echo 1 > /proc/sys/net/ipv4/ip_forward post-up iptables -t nat -A POSTROUTING -s '10.30.30.0/24' -o enp9s0 -j MASQUERADE post-down iptables -t nat -D POSTROUTING -s '10.30.30.0/24' -o enp9s0 -j MASQUERADE post-up iptables -t nat -A PREROUTING -i enp9s0 -p tcp --dport 2210 -j DNAT --to 10.30.30.10:22 post-down iptables -t nat -D PREROUTING -i enp9s0 -p tcp --dport 2210 -j DNAT --to 10.30.30.10:22

Any ideas? =)

Hey @spectryx, thanks for sharing. This was very helpful!

I was wondering what this part does:

Code:

  post-up iptables -t nat -A PREROUTING -i enp9s0 -p tcp --dport 2210 -j DNAT --to 10.30.30.10:22
  post-down iptables -t nat -D PREROUTING -i enp9s0 -p tcp --dport 2210 -j DNAT --to 10.30.30.10:22

That isn't included in any of the other examples I've seen?

Also, the Proxmox docs example includes a /24 after the public IP address. That's not necessary? https://pve.proxmox.com/wiki/Networ...ith_tt_span_class_monospaced_iptables_span_tt

oguz · Jul 20, 2022

interstellar said:
was wondering what this part does:
post-up iptables -t nat -A PREROUTING -i enp9s0 -p tcp --dport 2210 -j DNAT --to 10.30.30.10:22
post-down iptables -t nat -D PREROUTING -i enp9s0 -p tcp --dport 2210 -j DNAT --to 10.30.30.10:22

that part is for forwarding a single port, in this case the traffic to port 2210 of the PVE machine is forwarded to the port 22 of the VM/CT on 10.30.30.10

interstellar said:
Also, the Proxmox docs example includes a /24 after the public IP address. That's not necessary?

that is to create the masquerade routing rule, which in essence allows you to do the port forward shown above.

interstellar · Jul 20, 2022

Thanks @oguz

For some reason, I'm having trouble with outgoing connectivity on the host. I can't ping google.com, 1.1.1.1 etc. This is my current config:

Code:

source /etc/network/interfaces.d/*

auto lo
iface lo inet loopback

auto enp0s31f6
#Real IP address
iface enp0s31f6 inet static
        address xxx.xxx.xxx.55/24
        hwaddress aa:bb:cc:dd:ee:ff (fake)
        netmask 255.255.255.xxx
        gateway xxx.xxx.xx.29


auto vmbr0
#Private sub network
iface vmbr0 inet static
        address  10.1.1.1/24
        bridge-ports none
        bridge-stp off
        bridge-fd 0

        post-up   echo 1 > /proc/sys/net/ipv4/ip_forward
        post-up   iptables -t nat -A POSTROUTING -s '10.1.1.0/24' -o eno1 -j MASQUERADE
        post-down iptables -t nat -D POSTROUTING -s '10.1.1.0/24' -o eno1 -j MASQUERADE

It's a Hetzner bare metal server. Not sure what I'm doing wrong. I've tried the public IP with and without the /24, and I've tried it with and without the hwaddress entry.

Do you have any suggestions?

oguz · Jul 20, 2022

interstellar said:
Do you have any suggestions?

* what's the output from ip a and ip r?

* is your interface called eno1 or enp0s31f6 (you have both in your configuration, those need to be the same)

* can you ping the gateway address in your config file?

interstellar · Jul 20, 2022

oguz said:
* what's the output from ip a and ip r?

* is your interface called eno1 or enp0s31f6 (you have both in your configuration, those need to be the same)

* can you ping the gateway address in your config file?

Oops, I missed the eno1. I've changed those to enp0s31f6 but still not getting any host connectivity.
Here's the output from ip -a:

Code:

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: enp0s31f6: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether aa:bb:cc:dd:ee:ff brd ff:ff:ff:ff:ff:ff
    inet xxx.xxx.xxx.55/26 scope global enp0s31f6
       valid_lft forever preferred_lft forever
    inet6 fe80::921b:eff:fef3:4ca/64 scope link
       valid_lft forever preferred_lft forever
6: vmbr0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default qlen 1000
    link/ether 6a:f0:18:e0:6b:af brd ff:ff:ff:ff:ff:ff
    inet 10.1.1.1/24 scope global vmbr0
       valid_lft forever preferred_lft forever

I noticed the /26 prefix and tried adding that to the public IP as well. No luck.

interstellar · Jul 20, 2022

@oguz This is the default interface file and public connectivity works fine:

Code:

### Hetzner Online GmbH installimage

source /etc/network/interfaces.d/*

auto lo
iface lo inet loopback
iface lo inet6 loopback

auto enp0s31f6
iface enp0s31f6 inet static
  address xxx.xxx.xxx.55
  netmask 255.255.255.xxx
  gateway xxx.xxx.xxx.29
  # route xxx.xxx.xxx.28/26 via xxx.xxx.xxx.29
  up route add -net xxx.xxx.xxx.28 netmask 255.255.255.xxx gw xxx.xxx.xxx.29 dev enp0s31f6

iface enp0s31f6 inet6 static
  address aaaa:bbb:ccc:dddd::2
  netmask 64
  gateway fe80::1

oguz · Jul 20, 2022

interstellar said:
This is the default interfaces file and public connectivity works fine:

your gateway is different

interstellar · Jul 20, 2022

@oguz Sorry, that was a typo in the original post. I've fixed the post. The gateway is the same in both

oguz · Jul 20, 2022

try like this:

Code:

auto lo
iface lo inet loopback
iface lo inet6 loopback

auto enp0s31f6
iface enp0s31f6 inet static
    address x.x.x.55/26
    gateway x.x.x.29
    up route add -net x.x.x.28 netmask 255.255.255.192 gw x.x.x.29 dev enp0s31f6

auto vmbr0
iface vmbr0 inet static
    hwaddress aa:bb:cc:dd:ee:ff
    address 10.1.1.1/24
    bridge-ports none
    bridge-stp off
    bridge-fd 0
    post up echo 1 > /proc/sys/net/ipv4/ip_forward
    post-up   iptables -t nat -A POSTROUTING -s '10.1.1.0/24' -o enp0s31f6 -j MASQUERADE
    post-down iptables -t nat -D POSTROUTING -s '10.1.1.0/24' -o enp0s31f6 -j MASQUERADE

interstellar · Jul 20, 2022

@oguz Got it working

Turns out it was a switch problem.

Thanks for your help!

Search

Search

Multiple VM's with only 1 public IP on Hetzner

memoria-guy

New Member

oguz

Proxmox Retired Staff

spirit

Distinguished Member

spectryx

New Member

spectryx

New Member

memoria-guy

New Member

interstellar

Member

oguz

Proxmox Retired Staff

interstellar

Member

oguz

Proxmox Retired Staff

interstellar

Member

interstellar

Member

oguz

Proxmox Retired Staff

interstellar

Member

oguz

Proxmox Retired Staff

interstellar

Member