Proxmox Network Config on Hetzner

[Kaan Karaca]

Hi,

It feels like I read the half of the internet including this forum, but the network config for Proxmox on a Hetzner server still doesn't work...
It was so easy with earlier versions and I don't get why it is so hard with the lastest ones.

Our aimed setup is actually nothing spectacular nor uncommon.
A host with its own IP, one or more container with their own public IP and some CT with private IPs.
To start simple we booked a server and bought an additional IP in the Hetzner Management Interface.

Proxmox is installed and we can login, download templates and create CTs.

Unfortunately no Container is reachable from the outside nor has access to internet.
After trying and testing several configs and following tutorials, threads etc. it still doesn't work.

Your help/tip/support is very welcome as we don't know what else we can do.

Let me share some config details with you:
Forwarding is enabled in /etc/sysctl.conf

net.ipv4.ip_forward=1
net.ipv6.conf.all.forwarding=1

These are the network details from Hetzner:

Mein-IP: 46.4.1.100
Gateway: 46.4.1.150
Broadcast: 46.4.1.190
Netmask: 255.255.255.224

Second-IP: 46.4.1.200
Separate Mac: 00:11:22:33:44

Given these network details, the settings in /etc/network/interfaces are as following:
We disabled IPv6 for the time being until we solved IPv4 first...

### Hetzner Online GmbH installimage

source /etc/network/interfaces.d/*

auto lo
iface lo inet loopback
iface lo inet6 loopback

auto enp6s0
iface enp6s0 inet static
  address 46.4.1.100
  #original setting: netmask 255.255.255.224
  netmask 255.255.255.255
  gateway 46.4.1.150
  pointopoint 46.4.1.150
  up route add -net 46.4.1.149 netmask 255.255.255.224 gw 46.4.1.150 dev enp6s0


auto vmbr0
iface vmbr0 inet static
  address 46.4.1.100 # Server Basis IP
  netmask 255.255.255.255 # Netzmaske neu gesetzt
  bridge_ports none
  bridge_stp off
  bridge_fd 0
    up ip route add 46.4.1.200/32 dev vmbr0 # erste VM mit erster Zusatz-IP

#iface enp6s0 inet6 static
#  address 2a03:5f7:123:123::2
#  netmask 64
#  gateway fe80::1

With these settings we create a new container with theses params

Bridge: vmbr0 
IP (static): 46.4.1.200/32 
MAC: 00:11:22:33:44
Gateway: 46.4.1.100

The container starts and can be attached via "lxc-attach -n 100", but it can't talk to the internet nor it can be reached from the internet.
As already said any advise is really appreciated.

Cheers
derkaan

 

[Sebastian Helfinger]

Hi Kaan,

I had the same problem like you. I've hit every single google result to get to know why. On Debian 8 it worked Debian 9 it didn't at all. I sit down and was thinking, why. I stumbled on my last proxmox server (I dropped and switched). Notice: I refresh your post every day to see if maybe someone came up with a solution, now who I finally got the problem solved, I register to answer you.

### Hetzner Online GmbH installimage
Block first the source interface directory (It's nothing inside, it will delay only for few ns)
#source /etc/network/interfaces.d/*

auto lo
iface lo inet loopback
iface lo inet6 loopback

auto enp6s0
# First we setup IPv4
iface enp6s0 inet manual

# Second we setup IPv6
iface enp6s0 inet6 manual

# Now comes our "magic" trick
auto vmbr0
iface vmbr0 inet static
  address 46.4.1.100
  netmask 255.255.255.255
  gateway 46.4.1.150
  pointopoint 46.4.1.150
  broadcast 46.4.1.159 # Important change this to your broadcast IP!!!
  bridge_ports enp6s0
  bridge_stp off
  bridge_fd 0
  up ip route add 46.4.1.200/32 dev vmbr0

iface enp6s0 inet6 static
  address 2a03:5f7:123:123::2
  netmask 64
  gateway fe80::1

Please change the broadcast settings, this is for our IP and VM's IP.

After setting up, reboot (Not just restart networking), IPv4 and IPv6 should work directly. Setup also a Internal Network "vmbr1" for internal communication (It's better to have one for future development). I copy you below my own settings

auto vmbr1
iface vmbr1 inet static
        address  10.10.10.1
        netmask  255.255.255.0
        bridge_ports none
        bridge_stp off
        bridge_fd 0

If you want also to have outside connection you will need postroute it (IPtables).

        post-up   echo 1 > /proc/sys/net/ipv4/ip_forward
        post-up   iptables -t nat -A POSTROUTING -s '10.10.10.0/24' -o vmbr0-j MASQUERADE
        post-down iptables -t nat -D POSTROUTING -s '10.10.10.0/24' -o vmbr0 -j MASQUERADE

I hope I could help you, setting up the proxmox networking can be sometimes "head-to-keyboard" featuring...

 

[DerDanilo]

Thanks for your post.
This is about the broadcast? Why does it then work with the original PVE ISO without setting the broadcast?

 

[Sebastian Helfinger]

Yes, the PVE ISO doesn't has this option, I just added since I've this option (Hetzner IP settings field). Also I want to avoid future "problems".

In Hetzner they have IP assign Racks, which means, if you are in the same Rack like example Repo from some Linux Dists. It's better to have the broadcast assigned. You can communicate in the same network (Efficiently). Don't get me wrong on this, but I don't want anymore "Head-to-Keyboard"... In Hetzner wiki is also nothing written, but my old IP-range had example clearos repo's so I thought I should add this.

Of course, you can complete erase this option, since Broadcast would be only helpful for "LAN-Neighbors" in the private sectors.

 

[DerDanilo]

So does your Hetzner Subnet work on Stretch with minimal setup?

 

[Sebastian Helfinger]

Yes, with all IP's. Debian 8/Debian 9 from Hetzner Images

The VM Container looks like this:

auto lo
iface lo inet loopback

auto eth0
iface eth0 inet static
        address <Main IP>
        netmask 255.255.255.255
# --- BEGIN PVE ---
        post-up ip route add <Main IP> dev eth0
        post-up ip route add default via <Main IP> dev eth0
        pre-down ip route del default via <Main IP> dev eth0
        pre-down ip route del <Main IP> dev eth0
# --- END PVE ---
        dns-nameservers 213.133.100.100 213.133.98.98 213.133.99.99

auto eth1
iface eth1 inet static
        address 10.10.10.8
        netmask 255.255.255.255
# --- BEGIN PVE ---
        post-up ip route add 10.10.10.1 dev eth1
        post-up ip route add default via 10.10.10.1 dev eth1
        pre-down ip route del default via 10.10.10.1 dev eth1
        pre-down ip route del 10.10.10.1 dev eth1
# --- END PVE ---

iface eth0 inet6 static
        address <Sub IPv6>
        netmask 32
        gateway <Main IPv6>

Also the "Remote-IP" is visible on the VM see here.

212.237.59.163 - - [21/Aug/2017:09:07:42 +0000] "GET /MyAdmin/scripts/setup.php HTTP/1.1" 404 480 "-" "ZmEu"

Traceroute (See screenshot) "jumped" the holder (Main IP).

 

[DerDanilo]

Basically the issue was that the Hetzner Robot used a 2ndary MAC address and tried to route the additional subnet/IP to this one. Of course the subnet and the IPs were not routed correctly. Once corrected in the robot our initial Stretch network config worked fine.

I may post a config example to the Proxmox Wiki for Jessie and one for Stretch.

- Broadcast is not required
- Point-to-Point is not required
- Routes within VMs/LXC are not required

- Using Hetzner DNS Servers for VMs is quiet nice, have not done this often out of laziness

 

[Sebastian Helfinger]

It seems like you are keeping things out, because of laziness, I add things in (When it's not required) because of enthusiasmus... Fair deal

 

[montaropdf]

Hi,

I am facing the same problem, container inaccessible from the outside world and unable to access the internet from the container.

Below you will find the Information provided by Hetzner.

Dedicated server info:

IP address:  88.99.213.160
Netmask:  255.255.255.192
Gateway:  88.99.213.129
Broadcast: 88.99.213.191

IP subnet info:

Statically routed on the IP 88.99.213.160.
Subnet:    94.130.94.152 /29
Netmask:    255.255.255.248
Broadcast:    94.130.94.159

Usable IP addresses:

94.130.94.153 to 94.130.94.158

Host configuration

# network interface settings; autogenerated
# Please do NOT modify this file directly, unless you know what
# you're doing.
#
# If you want to manage part of the network configuration manually,
# please utilize the 'source' or 'source-directory' directives to do
# so.
# PVE will preserve these directives, but will NOT its network
# configuration from sourced files, so do not attempt to move any of
# the PVE managed interfaces into external files!

source /etc/network/interfaces.d/*

auto lo
iface lo inet loopback

iface lo inet6 loopback

auto enp0s31f6
iface enp0s31f6 inet static
        address  88.99.213.160
        netmask  255.255.255.192
        gateway  88.99.213.129
        up route add -net 88.99.213.128 netmask 255.255.255.192 gw 88.99.213.129 dev enp0s31f6
# route 88.99.213.128/26 via 88.99.213.129

iface enp0s31f6 inet6 static
        address  2a01:4f8:10a:3e9f::2
        netmask  64
        gateway  fe80::1

auto vmbr0
iface vmbr0 inet static
        address  94.130.94.152
        netmask  255.255.255.248
        bridge_ports none
        bridge_stp off
        bridge_fd 0

Guest configuration

auto lo
iface lo inet loopback

auto eth0
iface eth0 inet static
        address 94.130.94.154
        netmask 255.255.255.255
# --- BEGIN PVE ---
        post-up ip route add 94.130.94.152 dev eth0
        post-up ip route add default via 94.130.94.152 dev eth0
        pre-down ip route del default via 94.130.94.152 dev eth0
        pre-down ip route del 94.130.94.152 dev eth0
# --- END PVE ---
        dns-nameservers 213.133.100.100 213.133.98.98 213.133.99.99

1. Reading this thread I have the feeling that I don't need a subnet, but just 1 or 2 individual IP address.

I want to deploy multiple VMs/Containers behind a reverse-proxy (another VM/Container). So, if my understanding is correct, only the reverse-proxy needs to be available from the internet and so needs an IP from Hetzner. The other ones could get there IP from the vmbr1 bridge pool of IP, in the exemple above.

I suppose that if I want the VM/container to access the internet, for update I need another VM which will act as a proxy?

2. Is there any changes needed for KVM VMs?

3. How can I translate that configuration to CentOS 7 network configuration?

4. I have found the following documents in the Hetzner wiki, but they confuses me so far

wiki.hetzner.de/index.php/Zusaetzliche_IP-Adressen/en
wiki.hetzner.de/index.php/Proxmox_VE/en

Hope everything is clearly explained.

 

[Sebastian Helfinger]

Hi montaropdf,

I want to deploy multiple VMs/Containers behind a reverse-proxy (another VM/Container). So, if my understanding is correct, only the reverse-proxy needs to be available from the internet and so needs an IP from Hetzner. The other ones could get there IP from the vmbr1 bridge pool of IP, in the exemple above.

You are right, reverse-proxy needs to have a IP address to be connected, all VM can be routed with the reverse-proxy. The VM's itself can have access via vmbr1 (From my settings above) internet.

Is there any changes needed for KVM VMs?

Yes, you will need to give each VM a static IP address (Private Network), in the example above it's 10.10.10.0/24

How can I translate that configuration to CentOS 7 network configuration?

Last time I checked CentOS Networking configuration it's pretty the same like above, it was CentOS 6. I'm not sure if CentOS has changed everything...

I have found the following documents in the Hetzner wiki, but they confuses me so far

Still you confuse here, Reverse-Proxy no need for additional IP setup. It's worthless, since you want to setup a reverse-proxy, but you building now a "routing". I'm pretty sure you don't know Reverse-Proxy, that's why read more about it.

 

[montaropdf]

I think my confusion comes more from my lack of understanding about networking than what is a reverse-proxy. This is one of the reason for this project, which include, deploying some services to access them from various remote location and understand better, things related to virtualisation, networking, firewalling and the like.

Concerning CentOS, for what I see, it uses files in /etc/sysconfig/network-scripts and their format is quite different from Debian, see screenshot attached.

Now, my remaining problems, so far, are:
1. None of my VMs/Container (using an IP provided by Hetzner) can access the outside world.
Firewall is disabled at the level of Proxmox, but I have configured the SSH template in the Hetzner robot webinterface

2. SSH login from the dedicated server to the CentOS 7 VM, is slow and always fails
Surely linked to some pam authentication modules like polkit and SELinux, I will have a look at it.

 

[Sebastian Helfinger]

I think my confusion comes more from my lack of understanding about networking than what is a reverse-proxy. This is one of the reason for this project, which include, deploying some services to access them from various remote location and understand better, things related to virtualisation, networking, firewalling and the like.

I guess... you totally wrong. You setup a "reverse-proxy" with a public IP address. Ehm? What does reverse-proxy? Why not directly assigning domain/sub-domain to the IP, and stop doing some weird stuff.

Now, my remaining problems, so far, are:
1. None of my VMs/Container (using an IP provided by Hetzner) can access the outside world.
Firewall is disabled at the level of Proxmox, but I have configured the SSH template in the Hetzner robot webinterface

So far as I can tell, your dedicated server config is wrong. You have two possibility A.) Setup like I posted above or B.) Setup a private network (described above also) and do a ipv4_forward/masquerade

2. SSH login from the dedicated server to the CentOS 7 VM, is slow and always fails
Surely linked to some pam authentication modules like polkit and SELinux, I will have a look at it.

Routing problem.

*Tip* if you are not familiar with a Linux system, use it in VMWare local on your PC/Laptop, before you try to use it on a Server.

 

[montaropdf]

I guess... you totally wrong. You setup a "reverse-proxy" with a public IP address. Ehm? What does reverse-proxy? Why not directly assigning domain/sub-domain to the IP, and stop doing some weird stuff.

A reverse-proxy is a server with a piece of software that is supposed to stand between a network of clients and one or more web applications. So doesn't the reverse-proxy require a public IP so that the clients can access it?

*Tip* if you are not familiar with a Linux system, use it in VMWare local on your PC/Laptop, before you try to use it on a Server.

I am familiar with Linux System (its the system on my laptop and what I deal with at work), but not with all aspects of it, either, because:
- I don't use those aspects.
- Their is a lot of automation to install and manage them, inside an existing corporate infrastructure.
- They were installed ages ago, and my options were limited to "play" with them at the time.

 

[Sebastian Helfinger]

A reverse-proxy is a server with a piece of software that is supposed to stand between a network of clients and one or more web applications. So doesn't the reverse-proxy require a public IP so that the clients can access it?

One IP (Public) and a Private 24 network. If you use Reverse-Proxy with multiple "Public-IPs" what is the sense of Reverse-Proxy?

 

[montaropdf]

ok, then, I think their is misunderstanding, misjudment, misinterpretation and confusion in both of us.

As I said, by reading this thread I was not so sure a subnet was necessary in the end, and as I stated later, my knowledge of networking is poor, so I was not aware, or didn't realise, that it was possible to create a private network out of thin air in an existing infrstructure. But in this case I suppose this is a feature provided by the virtual environment (proxmox) and not the Hetzner network itself.

When I post my first message, I had already acquired the subnet and I am currently trying to understand how to configure VMs and containers using what I have already available (i.e. the subnet of IPs)

 

[DerDanilo]

@montaropdf If you still need help on how to configure your network let us know. For a small tip I can also connect to your machines and show you how to configure those (after my holidays). Happy configuration.

 

[Sebastian Helfinger]

@DerDanilo, for a "small tip" you can explain how to configure on CentOS here.... Of course after your holidays.

 

[montaropdf]

@DerDanilo , Sorry for my lack of communication, I was "playing" with proxmox and some VM/container.

Actually, the only remaining configuration problem concerns the public IP, I succeed in configuring a bridge with a private IP subnet, but I still struggle with the public one.

I have cancelled the subnet of public IP, as it is not necessary, but before ordering a single IP I want to give more context on what is my plan for that dedicated server.

I plan to deploy the following services (not necessarily available publicly):
- Seafile
- GitLab
- Radicale
- A mail server (postfix) + IMAP server
- My blog (maybe), hosted somewhere else ATM

So basically, I was thinking about exposing the public applications through a reverse proxy and, configuring a VPN to access the administration part, of the whole system. So that means I would need 2 or 3 public IP, one for the RP, one for the VPN and maybe a third one for the mail server, except if I use one of the other IP and set up a forwarding using some firewall rule.

Have a happy vacation time.

 

[navigator]

[QUOTE = "DerDanilo, post: 179949, membro: 42891"]
[USER = 49519] @montaropdf [/ USER] Se você ainda precisar de ajuda sobre como configurar sua rede, informe-nos. Para uma pequena dica, também posso conectar-me às suas máquinas e mostrar-lhe como configurá-las (após minhas férias). Configuração feliz.
[/ CITAR]
Would you mind to help me install mine?

I already have Proxmox Cluster on my Local Network,im using Hetzner to have a second cluster ready to action, but i discovered that things work very different on their Network and im having a lot of dificulties

 

[DerDanilo]

Sure I can help. Shouldn't be difficult. Please write me a PM so we can find a timeslot to answer your questions.