Hetzner Proxmox routed configuration halfway working

Retuas · Aug 31, 2019

Hello,

i recently bought (or rent) myself my first Hetzner Root Server. I did install Proxmox on it and configured two bridges vmbr0 and vmbr1. With vmbr0 i want to route my second public ip address to my first virtualized host. I have searched the whole internet now and could not find a solution for my problem.

I managed to get a internet connection on my virtual host. I can ssh it with the additional ip address i routet onto vmbr0. But if i try to connect to it via browser and http/https i get somthing like a redirect to my main ip (Or basically the server on the main ip - which is the proxmox server- answers that connection)? So im not able to install a web server there for example. Another strange thing is, that if i try to update the virt machine with dnf (or wget a file from the www) i get a strange error where certificate is not the right one which leads me to the thesis that im talking to the proxmox host which acts like a proxy. Can you please help me here? Did i miss something maybe?

PVEVERION = pve-manager/6.0-6/c71f879f (running kernel: 5.0.21-1-pve)
OS = Debian Buster

### Config of Proxmox Server ###

# /etc/network/interfaces
auto lo
iface lo inet loopback
iface lo inet6 loopback

auto enp2s0
iface enp2s0 inet static
address <MAIN IP>
netmask 255.255.255.255
gateway <GATEWAY OF MAIN IP>
pointopoint <GATEWAY OF MAIN IP>

auto vmbr0
iface vmbr0 inet static
address <MAIN IP>
netmask 255.255.255.255
bridge_ports none
bridge_stp off
bridge_fd 0
up ip route add <ADDITIONAL IP>/32 dev vmbr0

auto vmbr1
iface vmbr1 inet static
address 10.20.30.1
netmask 255.255.255.0
bridge_ports none
bridge_stp off
bridge_fd 0
post-up iptables -t nat -A POSTROUTING -s '10.20.30.0/24' -o eth0 -j MASQUERADE
post-down iptables -t nat -D POSTROUTING -s '10.20.30.0/24' -o eth0 -j MASQUERADE

### Config of Guest (Fedora 30 Server) ###

# /etc/sysconfig/network-scripts/ifcfg-ens18
TYPE=Ethernet
BOOTPROTO=none
NAME=ens18
UUID=0970dc53-3b3b-3035-bdad-f403d398bd23
DEVICE=ens18
ONBOOT=yes
AUTOCONNECT_PRIORITY=-999
IPADDR=<ADDITIONAL IP>
NETMASK=255.255.255.255
GATEWAY=<MAIN IP>
DNS=1.1.1.1

Richard · Sep 5, 2019

Retuas said:
i recently bought (or rent) myself my first Hetzner Root Server. I did install Proxmox on it and configured two bridges vmbr0 and vmbr1. With vmbr0 i want to route my second public ip address to my first virtualized host. I have searched the whole internet now and could not find a solution for my problem.

I managed to get a internet connection on my virtual host. I can ssh it with the additional ip address i routet onto vmbr0. But if i try to connect to it via browser and http/https i get somthing like a redirect to my main ip (Or basically the server on the main ip - which is the proxmox server- answers that connection)? So im not able to install a web server there for example. Another strange thing is, that if i try to update the virt machine with dnf (or wget a file from the www) i get a strange error where certificate is not the right one which leads me to the thesis that im talking to the proxmox host which acts like a proxy. Can you please help me here? Did i miss something maybe?

AFAICS not (according to providers how-to) - but note that it's the providers responsibility. Note that the "routed" version is a little bit tricky. Try rather use the "bridged" method https://wiki.hetzner.de/index.php/Netzkonfiguration_Debian/en#Bridged - all you need is to order an additional MAC. Then you have clear conditions. If it's still not working it's maybe caused by wrong webserver configuration.

Florian Schaal · Sep 6, 2019

Retuas said:
I can ssh it with the additional ip address i routet onto vmbr0. But if i try to connect to it via browser and http/https i get somthing like a redirect to my main ip

Are you sure, that this is really a problem with your network? If ssh works and http fails, it looks more like an error with your web-server.

Florian Schaal · Sep 6, 2019

Richard said:
Note that the "routed" version is a little bit tricky.

Really? We run several servers and all are setup with the routed setup.

t.lamprecht · Sep 6, 2019

Florian Schaal said:
Really? We run several servers and all are setup with the routed setup.

Nobody said it's impossible

But for people starting out the bridged one is often easier.

Florian Schaal · Sep 6, 2019

Maybe.

But you have to request additional MAC addresses and this is only possible for single IPs. Multiple IP addresses cannot be used in one VM. IPv6 can only be used on the host or in a single VM.
Beside this: we have a script, that can create the network-config for your server(s) with a routed setup: https://git.schaal-it.com/hetzner/proxmox

proxminent · Mar 13, 2020

Hi
For Subnet IP's (example /29) Hetzner accept only routed setup and is not posible to request additional MAC addresses.
you have to request additional MAC addresses and this is only possible for single IPs.
Warning ! Problem with ifupdown2.
see https://forum.proxmox.com/threads/no-internet-on-all-vms-after-update.64404/#post-300739

t.lamprecht · Mar 13, 2020

proxminent said:
Warning ! Problem with ifupdown2

not anymore - at least shouldn't be

bebraver_Axel · Mar 20, 2020

Florian Schaal said:
Maybe. But you have to request additional MAC addresses and this is only possible for single IPs. Multiple IP addresses cannot be used in one VM. IPv6 can only be used on the host or in a single VM.
Beside this: we have a script, that can create the network-config for your server(s) with a routed setup: https://git.schaal-it.com/hetzner/proxmox

Thank you Florian!

thereiter · May 5, 2020

Hello!
Didn’t want to start a new thread.
Having trouble configuring routed network between host and VM’s @ Hetzner.
I have to attach 2 external ips to my VM. I successfully set up bridge configuration, but unfortunately Hetzner do not allow more than 1 ip per VM. So I had to look for routed setup.
Thanks Florian Schaal for wonderful script https://git.schaal-it.com/hetzner/proxmox,
but unfortunately network on VM still not working.

So what I have:
1) Hetzner root server
2) 3 ips:
Main ip:
123.215.228.229
Gateway: 123.215.228.193
Netmask: 255.255.255.192
Broadcast: 123.215.228.255
Two additional ips with separate mac’s
123.215.228.228,
123.215.228.227

On host:
cat /proc/sys/net/ipv4/conf/enp196s0/proxy_arp → 1
cat /proc/sys/net/ipv4/ip_forward → 1

Host configuration:

Code:

# /etc/network/interfaces

### generated using Proxmox-Setup Tool 1.1 from schaal @it UG
### https://schaal-it.com/script-to-install-proxmox-5-x-and-6-x-on-a-dedicated-hetzner-server/
###
### Network-Type routed

# loopback device
auto lo
iface lo inet loopback
iface lo inet6 loopback

# network device
auto enp196s0
iface enp196s0 inet static
    address        123.215.228.229
    netmask        255.255.255.255
    gateway        123.215.228.193
    pointopoint    123.215.228.193
    post-up echo 1 > /proc/sys/net/ipv4/ip_forward
        post-up echo 1 > /proc/sys/net/ipv4/conf/enp196s0/proxy_arp

iface enp196s0 inet6 static
    address        2a01:4f8:241:4859::1
    netmask        128
    gateway        fe80::1
    up sysctl -p

auto vmbr0
iface vmbr0 inet static
    address        123.215.228.229
    netmask        255.255.255.255
    broadcast     123.215.228.255
    bridge_ports    none   
    bridge_stp    off
    bridge_fd    0
    up ip route add 123.215.228.227/32 dev vmbr0
    up ip route add 123.215.228.228/32 dev vmbr0

iface vmbr0 inet6 static
    address        2a01:4f8:241:4859::2
    netmask        64

Guest configuration:

Code:

auto ens18
iface ens18 inet static
    address 123.215.228.227
    netmask 255.255.255.192
    post-up ip route add 123.215.228.229 dev ens18
    post-up ip route add default via 123.215.228.229 dev ens18
    post-down ip route del default via 123.215.228.229 dev ens18
    post-down ip route del 123.215.228.229 dev ens18
    dns-nameservers 8.8.8.8 4.4.4.4 1.1.1.1

Host can ping VM, VM can ping host. But VM don’t have access to the internet. Not even ping to the gateway.
Firewalls seems to be off in Proxmox.
Just in case iptables flushed:

Code:

iptables -F
iptables -X
iptables -t nat -F
iptables -t nat -X
iptables -t mangle -F
iptables -t mangle -X
iptables -P INPUT ACCEPT
iptables -P FORWARD ACCEPT
iptables -P OUTPUT ACCEPT

Please, help!

thereiter · May 5, 2020

thereiter said:
Hello!
Didn’t want to start a new thread.
Having trouble configuring routed network between host and VM’s @ Hetzner.
I have to attach 2 external ips to my VM. I successfully set up bridge configuration, but unfortunately Hetzner do not allow more than 1 ip per VM. So I had to look for routed setup.
Thanks Florian Schaal for wonderful script https://git.schaal-it.com/hetzner/proxmox,
but unfortunately network on VM still not working.

So what I have:
1) Hetzner root server
2) 3 ips:
Main ip:
123.215.228.229
Gateway: 123.215.228.193
Netmask: 255.255.255.192
Broadcast: 123.215.228.255
Two additional ips with separate mac’s
123.215.228.228,
123.215.228.227

On host:
cat /proc/sys/net/ipv4/conf/enp196s0/proxy_arp → 1
cat /proc/sys/net/ipv4/ip_forward → 1

Host configuration:

Code:

# /etc/network/interfaces ### generated using Proxmox-Setup Tool 1.1 from schaal @it UG ### https://schaal-it.com/script-to-install-proxmox-5-x-and-6-x-on-a-dedicated-hetzner-server/ ### ### Network-Type routed # loopback device auto lo iface lo inet loopback iface lo inet6 loopback # network device auto enp196s0 iface enp196s0 inet static address 123.215.228.229 netmask 255.255.255.255 gateway 123.215.228.193 pointopoint 123.215.228.193 post-up echo 1 > /proc/sys/net/ipv4/ip_forward post-up echo 1 > /proc/sys/net/ipv4/conf/enp196s0/proxy_arp iface enp196s0 inet6 static address 2a01:4f8:241:4859::1 netmask 128 gateway fe80::1 up sysctl -p auto vmbr0 iface vmbr0 inet static address 123.215.228.229 netmask 255.255.255.255 broadcast 123.215.228.255 bridge_ports none bridge_stp off bridge_fd 0 up ip route add 123.215.228.227/32 dev vmbr0 up ip route add 123.215.228.228/32 dev vmbr0 iface vmbr0 inet6 static address 2a01:4f8:241:4859::2 netmask 64

Guest configuration:

Code:

auto ens18 iface ens18 inet static address 123.215.228.227 netmask 255.255.255.192 post-up ip route add 123.215.228.229 dev ens18 post-up ip route add default via 123.215.228.229 dev ens18 post-down ip route del default via 123.215.228.229 dev ens18 post-down ip route del 123.215.228.229 dev ens18 dns-nameservers 8.8.8.8 4.4.4.4 1.1.1.1

Host can ping VM, VM can ping host. But VM don’t have access to the internet. Not even ping to the gateway.
Firewalls seems to be off in Proxmox.
Just in case iptables flushed:

Code:

iptables -F iptables -X iptables -t nat -F iptables -t nat -X iptables -t mangle -F iptables -t mangle -X iptables -P INPUT ACCEPT iptables -P FORWARD ACCEPT iptables -P OUTPUT ACCEPT

Please, help!

The problem resolved by revoking separate macs on hetzner

ssldn · Aug 10, 2020

Florian Schaal said:
Maybe. But you have to request additional MAC addresses and this is only possible for single IPs. Multiple IP addresses cannot be used in one VM. IPv6 can only be used on the host or in a single VM.
Beside this: we have a script, that can create the network-config for your server(s) with a routed setup: https://git.schaal-it.com/hetzner/proxmox

Hello dear fellows,
BTW am I missing anything?
When I php call the script only for networking:

Code:

Generate the config for the Server with the IP [88.11.444.91]: yes
Use NIC []: no
Using robot.conf.php for your API-Login
Enter your credentials for the Hetzner-API
robot_url [https://robot-ws.your-server.de]: https://robot-ws.your-server.de
robot_user []: the_user   
robot_password []: PPPPPWWWW
PHP Notice:  Undefined property: stdClass::$error in /root/proxmox/lib/RobotClient.class.php on line 79
PHP Notice:  Trying to get property 'message' of non-object in /root/proxmox/lib/RobotClient.class.php on line 79
PHP Notice:  Undefined property: stdClass::$error in /root/proxmox/lib/RobotClient.class.php on line 79
PHP Notice:  Trying to get property 'code' of non-object in /root/proxmox/lib/RobotClient.class.php on line 79
PHP Fatal error:  Uncaught RobotClientException in /root/proxmox/lib/RobotClient.class.php:79
Stack trace:
#0 /root/proxmox/lib/RobotRestClient.class.php(104): RobotClient->executeRequest()
#1 /root/proxmox/lib/RobotClient.class.php(691): RobotRestClient->get('https://robot-w...')
#2 /root/proxmox/lib/hetzner_network.inc.php(64): RobotClient->serverGet('yes')
#3 /root/proxmox/network-manual.php(73): hetzner_network->network('routed')
#4 {main}
  thrown in /root/proxmox/lib/RobotClient.class.php on line 79

ssldn · Aug 20, 2020

thereiter said:
The problem resolved by revoking separate macs on hetzner

Good, that I saw you used the network config script and posted it. Cause the network script (alone, no copmlete setup) did not work on my hetzner route. And sadly He didnt reply so far.
But I wll positively mooded now try your setup. Cause I digged thruough the whole net for finding a routed setup.

PetePal · Apr 2, 2021

Florian Schaal said:
Maybe. But you have to request additional MAC addresses and this is only possible for single IPs. Multiple IP addresses cannot be used in one VM. IPv6 can only be used on the host or in a single VM.
Beside this: we have a script, that can create the network-config for your server(s) with a routed setup: https://git.schaal-it.com/hetzner/proxmox

I only registered to tell you that you are my favorite person in the world, sir. I was struggling with sorting this out for a week now, and your script did the job.

My guest /etc/network/interfaces, in case anybody fighting with config:

Code:

auto lo
iface lo inet loopback
iface lo inet6 loopback

auto eth0
iface eth0 inet static
        address { ADDITIONAL IP }/32
# --- BEGIN PVE ---
        post-up ip route add { MAIN IP } dev eth0
        post-up ip route add default via { MAIN IP } dev eth0
        pre-down ip route del default via { MAIN } dev eth0
        pre-down ip route del { MAIN IP } dev eth0
# --- END PVE ---

Proxmox GUI setup:

Code:

MAC Address: leave blank, Proxmox will sort it out
Bridge: vmbr0
IPv4/CIDR: {ADDITIONAL IP}/32
Gateway (IPv4): {MAIN IP}

Search

Search

Hetzner Proxmox routed configuration halfway working

Retuas

Member

Richard

Renowned Member

Florian Schaal

Active Member

Florian Schaal

Active Member

t.lamprecht

Proxmox Staff Member

Florian Schaal

Active Member

proxminent

Well-Known Member

t.lamprecht

Proxmox Staff Member

bebraver_Axel

Member

thereiter

New Member

thereiter

New Member

ssldn

Member

ssldn

Member

PetePal

Member