Proxmox Network Config on Hetzner

Discussion in 'Proxmox VE: Installation and configuration' started by Kaan Karaca, Aug 10, 2017.

  1. Kaan Karaca

    Kaan Karaca New Member

    Joined:
    Jul 17, 2013
    Messages:
    1
    Likes Received:
    0
    Hi,

    It feels like I read the half of the internet including this forum, but the network config for Proxmox on a Hetzner server still doesn't work...
    It was so easy with earlier versions and I don't get why it is so hard with the lastest ones.

    Our aimed setup is actually nothing spectacular nor uncommon.
    A host with its own IP, one or more container with their own public IP and some CT with private IPs.
    To start simple we booked a server and bought an additional IP in the Hetzner Management Interface.

    Proxmox is installed and we can login, download templates and create CTs.

    Unfortunately no Container is reachable from the outside nor has access to internet.
    After trying and testing several configs and following tutorials, threads etc. it still doesn't work.

    Your help/tip/support is very welcome as we don't know what else we can do.

    Let me share some config details with you:
    Forwarding is enabled in /etc/sysctl.conf
    Code:
    net.ipv4.ip_forward=1
    net.ipv6.conf.all.forwarding=1
    

    These are the network details from Hetzner:
    Code:
    Mein-IP: 46.4.1.100
    Gateway: 46.4.1.150
    Broadcast: 46.4.1.190
    Netmask: 255.255.255.224
    
    Second-IP: 46.4.1.200
    Separate Mac: 00:11:22:33:44
    

    Given these network details, the settings in /etc/network/interfaces are as following:
    We disabled IPv6 for the time being until we solved IPv4 first...
    Code:
    ### Hetzner Online GmbH installimage
    
    source /etc/network/interfaces.d/*
    
    auto lo
    iface lo inet loopback
    iface lo inet6 loopback
    
    auto enp6s0
    iface enp6s0 inet static
      address 46.4.1.100
      #original setting: netmask 255.255.255.224
      netmask 255.255.255.255
      gateway 46.4.1.150
      pointopoint 46.4.1.150
      up route add -net 46.4.1.149 netmask 255.255.255.224 gw 46.4.1.150 dev enp6s0
    
    
    auto vmbr0
    iface vmbr0 inet static
      address 46.4.1.100 # Server Basis IP
      netmask 255.255.255.255 # Netzmaske neu gesetzt
      bridge_ports none
      bridge_stp off
      bridge_fd 0
        up ip route add 46.4.1.200/32 dev vmbr0 # erste VM mit erster Zusatz-IP
    
    #iface enp6s0 inet6 static
    #  address 2a03:5f7:123:123::2
    #  netmask 64
    #  gateway fe80::1
    
    

    With these settings we create a new container with theses params
    Code:
    Bridge: vmbr0 
    IP (static): 46.4.1.200/32 
    MAC: 00:11:22:33:44
    Gateway: 46.4.1.100
    
    The container starts and can be attached via "lxc-attach -n 100", but it can't talk to the internet nor it can be reached from the internet.
    As already said any advise is really appreciated.

    Cheers
    derkaan
     
  2. Sebastian Helfinger

    Sebastian Helfinger New Member

    Joined:
    Aug 21, 2017
    Messages:
    8
    Likes Received:
    0
    Hi Kaan,

    I had the same problem like you. I've hit every single google result to get to know why. On Debian 8 it worked Debian 9 it didn't at all. I sit down and was thinking, why. I stumbled on my last proxmox server (I dropped and switched). Notice: I refresh your post every day to see if maybe someone came up with a solution, now who I finally got the problem solved, I register to answer you.

    Code:
    ### Hetzner Online GmbH installimage
    Block first the source interface directory (It's nothing inside, it will delay only for few ns)
    #source /etc/network/interfaces.d/*
    
    auto lo
    iface lo inet loopback
    iface lo inet6 loopback
    
    auto enp6s0
    # First we setup IPv4
    iface enp6s0 inet manual
    
    # Second we setup IPv6
    iface enp6s0 inet6 manual
    
    # Now comes our "magic" trick
    auto vmbr0
    iface vmbr0 inet static
      address 46.4.1.100
      netmask 255.255.255.255
      gateway 46.4.1.150
      pointopoint 46.4.1.150
      broadcast 46.4.1.159 # Important change this to your broadcast IP!!!
      bridge_ports enp6s0
      bridge_stp off
      bridge_fd 0
      up ip route add 46.4.1.200/32 dev vmbr0
    
    iface enp6s0 inet6 static
      address 2a03:5f7:123:123::2
      netmask 64
      gateway fe80::1
    Please change the broadcast settings, this is for our IP and VM's IP.

    After setting up, reboot (Not just restart networking), IPv4 and IPv6 should work directly. Setup also a Internal Network "vmbr1" for internal communication (It's better to have one for future development). I copy you below my own settings

    Code:
    auto vmbr1
    iface vmbr1 inet static
            address  10.10.10.1
            netmask  255.255.255.0
            bridge_ports none
            bridge_stp off
            bridge_fd 0
    
    If you want also to have outside connection you will need postroute it (IPtables).

    Code:
            post-up   echo 1 > /proc/sys/net/ipv4/ip_forward
            post-up   iptables -t nat -A POSTROUTING -s '10.10.10.0/24' -o vmbr0-j MASQUERADE
            post-down iptables -t nat -D POSTROUTING -s '10.10.10.0/24' -o vmbr0 -j MASQUERADE
    I hope I could help you, setting up the proxmox networking can be sometimes "head-to-keyboard" featuring...
     
    #2 Sebastian Helfinger, Aug 21, 2017
    Last edited: Aug 21, 2017
  3. DerDanilo

    DerDanilo Member
    Proxmox Subscriber

    Joined:
    Jan 21, 2017
    Messages:
    266
    Likes Received:
    24
    Thanks for your post.
    This is about the broadcast? Why does it then work with the original PVE ISO without setting the broadcast?
     
  4. Sebastian Helfinger

    Sebastian Helfinger New Member

    Joined:
    Aug 21, 2017
    Messages:
    8
    Likes Received:
    0
    Yes, the PVE ISO doesn't has this option, I just added since I've this option (Hetzner IP settings field). Also I want to avoid future "problems".

    In Hetzner they have IP assign Racks, which means, if you are in the same Rack like example Repo from some Linux Dists. It's better to have the broadcast assigned. You can communicate in the same network (Efficiently). Don't get me wrong on this, but I don't want anymore "Head-to-Keyboard"... In Hetzner wiki is also nothing written, but my old IP-range had example clearos repo's so I thought I should add this.

    Of course, you can complete erase this option, since Broadcast would be only helpful for "LAN-Neighbors" in the private sectors.
     
  5. DerDanilo

    DerDanilo Member
    Proxmox Subscriber

    Joined:
    Jan 21, 2017
    Messages:
    266
    Likes Received:
    24
    So does your Hetzner Subnet work on Stretch with minimal setup?
     
  6. Sebastian Helfinger

    Sebastian Helfinger New Member

    Joined:
    Aug 21, 2017
    Messages:
    8
    Likes Received:
    0
    Yes, with all IP's. Debian 8/Debian 9 from Hetzner Images

    The VM Container looks like this:

    Code:
    auto lo
    iface lo inet loopback
    
    auto eth0
    iface eth0 inet static
            address <Main IP>
            netmask 255.255.255.255
    # --- BEGIN PVE ---
            post-up ip route add <Main IP> dev eth0
            post-up ip route add default via <Main IP> dev eth0
            pre-down ip route del default via <Main IP> dev eth0
            pre-down ip route del <Main IP> dev eth0
    # --- END PVE ---
            dns-nameservers 213.133.100.100 213.133.98.98 213.133.99.99
    
    auto eth1
    iface eth1 inet static
            address 10.10.10.8
            netmask 255.255.255.255
    # --- BEGIN PVE ---
            post-up ip route add 10.10.10.1 dev eth1
            post-up ip route add default via 10.10.10.1 dev eth1
            pre-down ip route del default via 10.10.10.1 dev eth1
            pre-down ip route del 10.10.10.1 dev eth1
    # --- END PVE ---
    
    iface eth0 inet6 static
            address <Sub IPv6>
            netmask 32
            gateway <Main IPv6>
    Also the "Remote-IP" is visible on the VM see here.
    Code:
    212.237.59.163 - - [21/Aug/2017:09:07:42 +0000] "GET /MyAdmin/scripts/setup.php HTTP/1.1" 404 480 "-" "ZmEu"
    Traceroute (See screenshot) "jumped" the holder (Main IP).
     

    Attached Files:

  7. DerDanilo

    DerDanilo Member
    Proxmox Subscriber

    Joined:
    Jan 21, 2017
    Messages:
    266
    Likes Received:
    24
    Basically the issue was that the Hetzner Robot used a 2ndary MAC address and tried to route the additional subnet/IP to this one. Of course the subnet and the IPs were not routed correctly. Once corrected in the robot our initial Stretch network config worked fine.

    I may post a config example to the Proxmox Wiki for Jessie and one for Stretch.

    - Broadcast is not required
    - Point-to-Point is not required
    - Routes within VMs/LXC are not required

    - Using Hetzner DNS Servers for VMs is quiet nice, have not done this often out of laziness :D
     
  8. Sebastian Helfinger

    Sebastian Helfinger New Member

    Joined:
    Aug 21, 2017
    Messages:
    8
    Likes Received:
    0
    It seems like you are keeping things out, because of laziness, I add things in (When it's not required) because of enthusiasmus... Fair deal
     
  9. montaropdf

    montaropdf New Member

    Joined:
    Aug 24, 2017
    Messages:
    14
    Likes Received:
    0
    Hi,

    I am facing the same problem, container inaccessible from the outside world and unable to access the internet from the container.

    Below you will find the Information provided by Hetzner.

    Dedicated server info:
    Code:
    IP address:  88.99.213.160
    Netmask:  255.255.255.192
    Gateway:  88.99.213.129
    Broadcast: 88.99.213.191
    IP subnet info:
    Code:
    Statically routed on the IP 88.99.213.160.
    Subnet:    94.130.94.152 /29
    Netmask:    255.255.255.248
    Broadcast:    94.130.94.159
    
    Usable IP addresses:
    
    94.130.94.153 to 94.130.94.158
    Host configuration
    Code:
    # network interface settings; autogenerated
    # Please do NOT modify this file directly, unless you know what
    # you're doing.
    #
    # If you want to manage part of the network configuration manually,
    # please utilize the 'source' or 'source-directory' directives to do
    # so.
    # PVE will preserve these directives, but will NOT its network
    # configuration from sourced files, so do not attempt to move any of
    # the PVE managed interfaces into external files!
    
    source /etc/network/interfaces.d/*
    
    auto lo
    iface lo inet loopback
    
    iface lo inet6 loopback
    
    auto enp0s31f6
    iface enp0s31f6 inet static
            address  88.99.213.160
            netmask  255.255.255.192
            gateway  88.99.213.129
            up route add -net 88.99.213.128 netmask 255.255.255.192 gw 88.99.213.129 dev enp0s31f6
    # route 88.99.213.128/26 via 88.99.213.129
    
    iface enp0s31f6 inet6 static
            address  2a01:4f8:10a:3e9f::2
            netmask  64
            gateway  fe80::1
    
    auto vmbr0
    iface vmbr0 inet static
            address  94.130.94.152
            netmask  255.255.255.248
            bridge_ports none
            bridge_stp off
            bridge_fd 0
    
    Guest configuration
    Code:
    auto lo
    iface lo inet loopback
    
    auto eth0
    iface eth0 inet static
            address 94.130.94.154
            netmask 255.255.255.255
    # --- BEGIN PVE ---
            post-up ip route add 94.130.94.152 dev eth0
            post-up ip route add default via 94.130.94.152 dev eth0
            pre-down ip route del default via 94.130.94.152 dev eth0
            pre-down ip route del 94.130.94.152 dev eth0
    # --- END PVE ---
            dns-nameservers 213.133.100.100 213.133.98.98 213.133.99.99
    
    1. Reading this thread I have the feeling that I don't need a subnet, but just 1 or 2 individual IP address.

    I want to deploy multiple VMs/Containers behind a reverse-proxy (another VM/Container). So, if my understanding is correct, only the reverse-proxy needs to be available from the internet and so needs an IP from Hetzner. The other ones could get there IP from the vmbr1 bridge pool of IP, in the exemple above.

    I suppose that if I want the VM/container to access the internet, for update I need another VM which will act as a proxy?

    2. Is there any changes needed for KVM VMs?

    3. How can I translate that configuration to CentOS 7 network configuration?

    4. I have found the following documents in the Hetzner wiki, but they confuses me so far
    Code:
    wiki.hetzner.de/index.php/Zusaetzliche_IP-Adressen/en
    wiki.hetzner.de/index.php/Proxmox_VE/en
    Hope everything is clearly explained.
     
  10. Sebastian Helfinger

    Sebastian Helfinger New Member

    Joined:
    Aug 21, 2017
    Messages:
    8
    Likes Received:
    0
    Hi montaropdf,

    You are right, reverse-proxy needs to have a IP address to be connected, all VM can be routed with the reverse-proxy. The VM's itself can have access via vmbr1 (From my settings above) internet.

    Yes, you will need to give each VM a static IP address (Private Network), in the example above it's 10.10.10.0/24

    Last time I checked CentOS Networking configuration it's pretty the same like above, it was CentOS 6. I'm not sure if CentOS has changed everything...

    Still you confuse here, Reverse-Proxy no need for additional IP setup. It's worthless, since you want to setup a reverse-proxy, but you building now a "routing". I'm pretty sure you don't know Reverse-Proxy, that's why read more about it.
     
  11. montaropdf

    montaropdf New Member

    Joined:
    Aug 24, 2017
    Messages:
    14
    Likes Received:
    0
    I think my confusion comes more from my lack of understanding about networking than what is a reverse-proxy. This is one of the reason for this project, which include, deploying some services to access them from various remote location and understand better, things related to virtualisation, networking, firewalling and the like.

    Concerning CentOS, for what I see, it uses files in /etc/sysconfig/network-scripts and their format is quite different from Debian, see screenshot attached.

    Now, my remaining problems, so far, are:
    1. None of my VMs/Container (using an IP provided by Hetzner) can access the outside world.
    Firewall is disabled at the level of Proxmox, but I have configured the SSH template in the Hetzner robot webinterface

    2. SSH login from the dedicated server to the CentOS 7 VM, is slow and always fails
    Surely linked to some pam authentication modules like polkit and SELinux, I will have a look at it.
     

    Attached Files:

  12. Sebastian Helfinger

    Sebastian Helfinger New Member

    Joined:
    Aug 21, 2017
    Messages:
    8
    Likes Received:
    0
    I guess... you totally wrong. You setup a "reverse-proxy" with a public IP address. Ehm? What does reverse-proxy? Why not directly assigning domain/sub-domain to the IP, and stop doing some weird stuff.

    So far as I can tell, your dedicated server config is wrong. You have two possibility A.) Setup like I posted above or B.) Setup a private network (described above also) and do a ipv4_forward/masquerade
    Routing problem.

    *Tip* if you are not familiar with a Linux system, use it in VMWare local on your PC/Laptop, before you try to use it on a Server.
     
  13. montaropdf

    montaropdf New Member

    Joined:
    Aug 24, 2017
    Messages:
    14
    Likes Received:
    0
    A reverse-proxy is a server with a piece of software that is supposed to stand between a network of clients and one or more web applications. So doesn't the reverse-proxy require a public IP so that the clients can access it?

    I am familiar with Linux System (its the system on my laptop and what I deal with at work), but not with all aspects of it, either, because:
    - I don't use those aspects.
    - Their is a lot of automation to install and manage them, inside an existing corporate infrastructure.
    - They were installed ages ago, and my options were limited to "play" with them at the time.
     
  14. Sebastian Helfinger

    Sebastian Helfinger New Member

    Joined:
    Aug 21, 2017
    Messages:
    8
    Likes Received:
    0
    Reverse-Proxy.png

    One IP (Public) and a Private 24 network. If you use Reverse-Proxy with multiple "Public-IPs" what is the sense of Reverse-Proxy?
    No-Reverse-Proxy.png
     
  15. montaropdf

    montaropdf New Member

    Joined:
    Aug 24, 2017
    Messages:
    14
    Likes Received:
    0
    ok, then, I think their is misunderstanding, misjudment, misinterpretation and confusion in both of us.

    As I said, by reading this thread I was not so sure a subnet was necessary in the end, and as I stated later, my knowledge of networking is poor, so I was not aware, or didn't realise, that it was possible to create a private network out of thin air in an existing infrstructure. But in this case I suppose this is a feature provided by the virtual environment (proxmox) and not the Hetzner network itself.

    When I post my first message, I had already acquired the subnet and I am currently trying to understand how to configure VMs and containers using what I have already available (i.e. the subnet of IPs)
     
  16. DerDanilo

    DerDanilo Member
    Proxmox Subscriber

    Joined:
    Jan 21, 2017
    Messages:
    266
    Likes Received:
    24
    @montaropdf If you still need help on how to configure your network let us know. For a small tip I can also connect to your machines and show you how to configure those (after my holidays). Happy configuration.
     
  17. Sebastian Helfinger

    Sebastian Helfinger New Member

    Joined:
    Aug 21, 2017
    Messages:
    8
    Likes Received:
    0
    @DerDanilo, for a "small tip" you can explain how to configure on CentOS here.... Of course after your holidays.
     
  18. montaropdf

    montaropdf New Member

    Joined:
    Aug 24, 2017
    Messages:
    14
    Likes Received:
    0
    @DerDanilo , Sorry for my lack of communication, I was "playing" with proxmox and some VM/container.

    Actually, the only remaining configuration problem concerns the public IP, I succeed in configuring a bridge with a private IP subnet, but I still struggle with the public one.

    I have cancelled the subnet of public IP, as it is not necessary, but before ordering a single IP I want to give more context on what is my plan for that dedicated server.

    I plan to deploy the following services (not necessarily available publicly):
    - Seafile
    - GitLab
    - Radicale
    - A mail server (postfix) + IMAP server
    - My blog (maybe), hosted somewhere else ATM

    So basically, I was thinking about exposing the public applications through a reverse proxy and, configuring a VPN to access the administration part, of the whole system. So that means I would need 2 or 3 public IP, one for the RP, one for the VPN and maybe a third one for the mail server, except if I use one of the other IP and set up a forwarding using some firewall rule.

    Have a happy vacation time.
     
  1. This site uses cookies to help personalise content, tailor your experience and to keep you logged in if you register.
    By continuing to use this site, you are consenting to our use of cookies.
    Dismiss Notice