Yet another network configuration problem - Hetzner and additional IP

5

5k7

New Member
Sep 28, 2022
16
0
1
Hi guys,

I spent 2 days struggling with simple configuration on hetzner dedicated server. I've read many articles, docs, search here on forum but still additional IP is not working.
It should be something really easy and straight forward but I see many people are having same issue over and over again. And now me too ;)

Going into details, i have dedicated server with standard IPv4 and IPv6 subnet. Additionally i bought another IP address and now i want to have second IPv4 working with container.

PVE HOST (/etc/network/interfaces):
Code: 
#Main IPv4
auto eno1
iface eno1 inet static
        address XX.XX.76.215 [MAIN IP]
        netmask 255.255.255.192 [MAIN SUBNET]
        gateway XX.XX.76.193 [MAIN GATEWAY]
        up route add -net XX.XX.76.192 netmask 255.255.255.192 gw XX.XX.76.193 dev eno1
        #pointopoint XX.XX.76.193

Code: 
#VM routed
auto vmbr0
iface vmbr0 inet static
        address XX.XX.76.215 [MAIN IP]
        netmask 255.255.255.255
        bridge-ports none
        bridge-stp off
        bridge-fd 0
        bridge_maxwait 0
        pre-up brctl addbr vmbr0
        up ip route add XX.XX.76.213/32 dev vmbr0 [ADDITIONAL IP]

HOST routes (route -n):

Code: 
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         XX.XX.76.193    0.0.0.0         UG    0      0        0 eno1
10.10.10.0      0.0.0.0         255.255.255.0   U     0      0        0 vmbr1
XX.XX.76.192    XX.XX.76.193    255.255.255.192 UG    0      0        0 eno1
XX.XX.76.192    0.0.0.0         255.255.255.192 U     0      0        0 eno1
XX.XX.76.213    0.0.0.0         255.255.255.255 UH    0      0        0 vmbr0

Guest (/etc/network/interfaces):
Code: 
auto eth0

iface eth0 inet static
        address XX.XX.76.213/32 [ADDITIONAL IP]
# --- BEGIN PVE ---
        post-up ip route add XX.XX.76.215 dev eth0 [MAIN IP]
        post-up ip route add default via XX.XX.76.215 dev eth0 [MAIN IP]
        pre-down ip route del default via XX.XX.76.215 dev eth0 [MAIN IP]
        pre-down ip route del XX.XX.76.215 dev eth0 [MAIN IP]
# --- END PVE ---
        dns-nameservers 1.1.1.1 8.8.8.8

Guest routes (route -n):

Code: 
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         XX.XX.76.215    0.0.0.0         UG    0      0        0 eth0
XX.XX.76.215    0.0.0.0         255.255.255.255 UH    0      0        0 eth0


Additionally in guest container I added mac address in gui. Currenlty I cannot even ping main gateway - XX.XX.76.193 from guest.

Can you advise what I'm doing here wrong or where is the problem ?

Thanks
 
you have one Server ?
you have the management IP, 1 Public IP and now want a second public IP ?
dont forget the MAC restrictions not
 
Last edited:
First public IPv4 is for PVE (dedicated server IP), second IPv4 is for this one particular container. Rest I will manage via ipv6.
 
i configured it on hetzner for some servers and it took me some time to get everything sorted. i have a slighlty different config.
i forward all traffic to a firewall, which manage the subnets.
if you are interested in such config (instead of putting the container directly in internet), let me know - i can send you the conf files.
 
my network interface looks like

auto enp41s0
iface enp41s0 inet manual
#1GB UPLINK


auto vmbr0
iface vmbr0 inet static
address public_IP
gateway Public_GW
bridge-ports enp41s0
bridge-stp off
bridge-fd 0
bridge-vlan-aware yes
bridge-vids 2-4094
#1GB UPLINK Public
 
Of course, thank you for the proposal. I sent you a pm. I'd like to check bridge version with firewall to see if this will solve all my problems.

I'm just still wondering what is the issue with routed one I used. I tried so many combinations and variants so I think it has to be something small which is missing all the time. Might be Hetzner specific. I tried even with confirmed working solution from different thread here, but t's still not working for me somehow.

Does anyone has proposal what would be a good idea to check/verify ?
 
its the infra i build
its still not finished yet. need to change the vxlan to evpn.
the tricky thing is - all 4 servers has 4 management IPs (not a problem) and 4 piblic IPs and a virtual switch (assigned to all 4 servers).
i want the traffic going in and out of the environment on all 4 IPs (right now only 1 is used for all traffic). my idea is per evpn put all 4 Public interfaces in one subnet and connect to the opnsense (with fixed MAC). but i dont know yet how to balance the outgoing traffic (income traffic will be balanced over the vswitch - i guess - maybe somebody can correct me, if i am wrong)
 

Attachments

  • infra.JPG
    infra.JPG
    170.7 KB · Views: 25
I managed almost everything i need with bridge mode. The only missing part here is IPv6. Well, maybe not missing but not complete. In bridge mode i assign additional IP normally on guest and everything is working just fine. I want achieve the same on ipv6. Currenlty i can reach only main host via
2a01:XXX:XXX:XXXX::2, but not guest (2a01:XXX:XX:XXXX::3).

What I'm still missing here ?

HOST (PVE):

Code: 
auto eno1
iface eno1 inet manual

auto vmbr0
iface vmbr0 inet static
        address XX.XX.76.215/32 #Main IP
        gateway XX.XX.76.193 #MAIN GATEWAY
        bridge-ports eno1
        bridge-stp off
        bridge-fd 0
        
iface vmbr0 inet6 static
        address 2a01:XXX:XXX:XXXX::2/64 #IPv6 Subnet
        gateway fe80::1


GUEST:
Code: 
auto lo
iface lo inet loopback

auto eth0
iface eth0 inet static
        address XX.XX.76.213/32 #ADDITIONAL IP
        gateway XX.XX.76.193 #MAIN GATEWAY 
# --- BEGIN PVE ---
        post-up ip route add XX.XX.76.193 dev eth0
        post-up ip route add default via XX.XX.76.193 dev eth0
        pre-down ip route del default via XX.XX.76.193 dev eth0
        pre-down ip route del XX.XX.76.193 dev eth0
# --- END PVE ---
        dns-nameservers 1.1.1.1 8.8.8.8
        hwaddress ether 00:50:XX:XX:XX:XX #MAC GENERATED BY HETZNER

iface eth0 inet6 static
        address 2a01:XXX:XX:XXXX::3/64
        gateway fe80::1
 
I solved it on my end. Didn't have time to post answer yet. Where exactly your problem is?
 
Just like the one you described...

HOST:
Code: 
auto eno1
iface eno1 inet static
        address 195.xx.yy.205/26
        gateway 195.xx.yy.193
        up route add -net 195.xx.yy.192 netmask 255.255.255.192 gw 195.xx.yy.193 dev eno1
# route 195.xx.yy.192/26 via 195.xx.yy.193


iface eno1 inet6 static
        address 2a01:4f8:13b:faat::2/128
        gateway fe80::1


auto vmbr0
iface vmbr0 inet static
        address 195.xx.yy.205/32
        bridge-ports none
        bridge-stp off
        bridge-fd 0


iface vmbr0 inet6 static
        address 2a01:4f8:13b:faat::3/64

GUEST:
Code: 
auto lo
iface lo inet loopback


auto eth0
iface eth0 inet6 static
        address 2a01:4f8:13b:faat::10/10
        gateway 2a01:4f8:13b:faat::2

The problem is I want to use IPv6, but I can't get the routing to work.
 
Last edited:
OK, first off all i solved it via bridge mode. But maybe I will check later if i can now do the same via routed version.
In your case you're not probably looking for additional IP which i struggled with, so I believe this one should work for you. Additionally there is vmbr1 network with NAT.

Code: 
### IPv4 ###
# Main IPv4 from Host
auto eno1
iface eno1 inet static
  address XX.XX.33.203
  netmask 255.255.255.255
  gateway XX.XX.33.193
  pointopoint XX.XX.33.193

### IPv6 ###
# Main IPv6
iface eno1 inet6 static
  address 2a01:XX:XXX:XXXX::2
  netmask 128
  gateway fe80::1
  up sysctl -p


### VM-Routed
auto vmbr0
iface vmbr0 inet static
  address X.XX.33.203
  netmask 255.255.255.255
  bridge_ports none
  bridge_stp off
  bridge_fd 0
 
#VM-Routed
iface vmbr0 inet6 static
  address 2a01:XX:XX:XXX::3
  netmask 64
  up ip -6 route add 2a01:XXX:XX:XXX::/64 dev vmbr0

### Private NAT used by Proxmox
auto vmbr1
iface vmbr1 inet static
  address  10.10.10.1
  netmask  255.255.255.0
  bridge_ports none
  bridge_stp off
  bridge_fd 0
  bridge_maxwait 0
  post-up   iptables -t raw -I PREROUTING -i fwbr+ -j CT --zone 1
  post-down iptables -t raw -D PREROUTING -i fwbr+ -j CT --zone 1
  post-up   iptables -t nat -A POSTROUTING -s '10.10.10.0/24' -o eno1 -j MASQUERADE
  post-down iptables -t nat -D POSTROUTING -s '10.10.10.0/24' -o eno1 -j MASQUERADE

And for your guest

Code: 
auto eth0
iface eth0 inet6 static
        address 2a01:XXX:XXX:XXX::6/64
        gateway 2a01:XXX:XXX:XXX::3

auto eth1
iface eth1 inet static
        address 10.10.10.3/24
        gateway 10.10.10.1


With this configuration you can reach your guest via IPv6 and also still have IPv4 through NAT


And remember:
Code: 
With a routed setup, the bridge vmbr0 is not connected with the physical interface. IP forwarding needs to be activated on the host system. Please note that packet forwarding between network interfaces is disabled for the default Hetzner installation. Activate it with the following command:

sysctl -w net.ipv4.ip_forward=1

Forwarding for IPv6 needs to be activated as well. This is also available in the Hetzner standard installation and only needs to be activated:

sysctl -w net.ipv6.conf.all.forwarding=1

The commands above only change runtime values. To make a persistent configuration, please adjust /etc/sysctl.conf file.

Source: https://community.hetzner.com/tutorials/install-and-configure-proxmox_ve
 
Last edited:
Here is also my working solution with bridge mode:

Code: 
auto lo
iface lo inet loopback

iface lo inet6 loopback

auto eno1
iface eno1 inet manual

auto vmbr0
iface vmbr0 inet static
        address XXX.XX.76.215/32
        gateway XXX.XX.76.193
        bridge-ports eno1
        bridge-stp off
        bridge-fd 0

iface vmbr0 inet6 static
        address 2a01:XXX:XXX:XXXX::1/128
        gateway fe80::1

auto vmbr1
iface vmbr1 inet static
        address 10.10.10.1/24
        bridge-ports none
        bridge-stp off
        bridge-fd 0

iface vmbr1 inet6 static
        address 2a01:XXX:XXX:XXX::2/64

post-up   iptables -t raw -I PREROUTING -i fwbr+ -j CT --zone 1
post-down iptables -t raw -D PREROUTING -i fwbr+ -j CT --zone 1
post-up   iptables -t nat -A POSTROUTING -s '10.10.10.0/24' -o vmbr0 -j MASQUERADE
post-down iptables -t nat -D POSTROUTING -s '10.10.10.0/24' -o vmbr0 -j MASQUERADE

up ip -6 route add 2a01:XXX:XXX:XXXX::/64 dev vmbr0

For guest additional IPv4 + IPv6:

Code: 
auto eth0
iface eth0 inet static #vmbr0
        address XX.XX.76.213/32 #Additional IP
# --- BEGIN PVE ---
        post-up ip route add XX.XX.XX.193 dev eth0
        post-up ip route add default via XX.XX.76.193 dev eth0
        pre-down ip route del default via XX.XX.76.193 dev eth0
        pre-down ip route del XX.XX.76.193 dev eth0
# --- END PVE ---
        dns-nameservers 1.1.1.1 8.8.8.8
       
auto eth1 #vmbr1
iface eth1 inet6 static
        address 2a01:XXX:XXX:XXXX::3/64
        gateway 2a01:XX:XX:XXXX::2

Other guests (IPV4 with NAT + IPV6):

Code: 
auto eth0
iface eth0 inet static
        address 10.10.10.2/24
        gateway 10.10.10.1

iface eth0 inet6 static
        address 2a01:XXX:XXX:XXXX::4/64
        gateway 2a01:XXX:XXX:XXXX::2
 
Thanks man, appreciate the input.

Sorry if I didn't express myself very well - I didn't mean that I particularly wanted to use routing, just to be able to reach the internet in the least complicated way. I just use the vmbr0 thingy.

So with your updates I got it working to a degree, but I'm still missing some bits.

My config now is
Code: 
source /etc/network/interfaces.d/*


auto lo
iface lo inet loopback


iface lo inet6 loopback


auto eno1
iface eno1 inet static
        address 195.xx.yy.205/26
        gateway 195.xx.yy.193
        up route add -net 195.xx.yy.192 netmask 255.255.255.192 gw 195.xx.yy.193 dev eno1
# route 195.xx.yy.192/26 via 195.xx.yy.193


iface eno1 inet6 static
        address 2a01:4f8:13b:fatt::2/128
        gateway fe80::1


auto vmbr0
iface vmbr0 inet static
        address 195.xx.yy.205/32
        bridge-ports none
        bridge-stp off
        bridge-fd 0


iface vmbr0 inet6 static
        address 2a01:4f8:13b:fatt::3/64


and client

Code: 
auto lo
iface lo inet loopback


auto eth0
iface eth0 inet6 static
        address 2a01:4f8:13b:fatt::10/64
        gateway 2a01:4f8:13b:fatt::3


I can ssh into both the host and the client using IPv6, and also resolve domain names and run apt update etc from both. I was able to install docker on the client, but trying to pull a docker test package, I got this error:

Code: 
# docker compose pull
[+] Running 1/0
 ✘ app Error                                                                                                       0.1s
Error response from daemon: Get "https://registry-1.docker.io/v2/": dial tcp 34.194.164.123:443: connect: network is unreachable

The same thing happens if I try to ping - IPv6 addresses are fine but IPv4 throw unreachable error.

I realize I'm into deeper water here - IPv6 is new territory for me - but I guess this is caused by this website not supporting IPv6 and/or my VM not having an IPv4 address? Is this how it works, is there any way around this or are we forced to keep using IPv4 until forever when every last server on the internet has changed to IPv6 ?
 
Last edited:
5k7 said:
Here is also my working solution with bridge mode:

Code: 
auto lo
iface lo inet loopback

iface lo inet6 loopback

auto eno1
iface eno1 inet manual

auto vmbr0
iface vmbr0 inet static
        address XXX.XX.76.215/32
        gateway XXX.XX.76.193
        bridge-ports eno1
        bridge-stp off
        bridge-fd 0

iface vmbr0 inet6 static
        address 2a01:XXX:XXX:XXXX::1/128
        gateway fe80::1

auto vmbr1
iface vmbr1 inet static
        address 10.10.10.1/24
        bridge-ports none
        bridge-stp off
        bridge-fd 0

iface vmbr1 inet6 static
        address 2a01:XXX:XXX:XXX::2/64

post-up   iptables -t raw -I PREROUTING -i fwbr+ -j CT --zone 1
post-down iptables -t raw -D PREROUTING -i fwbr+ -j CT --zone 1
post-up   iptables -t nat -A POSTROUTING -s '10.10.10.0/24' -o vmbr0 -j MASQUERADE
post-down iptables -t nat -D POSTROUTING -s '10.10.10.0/24' -o vmbr0 -j MASQUERADE

up ip -6 route add 2a01:XXX:XXX:XXXX::/64 dev vmbr0

For guest additional IPv4 + IPv6:

Code: 
auto eth0
iface eth0 inet static #vmbr0
        address XX.XX.76.213/32 #Additional IP
# --- BEGIN PVE ---
        post-up ip route add XX.XX.XX.193 dev eth0
        post-up ip route add default via XX.XX.76.193 dev eth0
        pre-down ip route del default via XX.XX.76.193 dev eth0
        pre-down ip route del XX.XX.76.193 dev eth0
# --- END PVE ---
        dns-nameservers 1.1.1.1 8.8.8.8
      
auto eth1 #vmbr1
iface eth1 inet6 static
        address 2a01:XXX:XXX:XXXX::3/64
        gateway 2a01:XX:XX:XXXX::2

Other guests (IPV4 with NAT + IPV6):

Code: 
auto eth0
iface eth0 inet static
        address 10.10.10.2/24
        gateway 10.10.10.1

iface eth0 inet6 static
        address 2a01:XXX:XXX:XXXX::4/64
        gateway 2a01:XXX:XXX:XXXX::2
Click to expand...

I set up a nat bridge pretty much exactly as you list it here, but still no cigar.
I can ping 10.10.10.1 from the client, I can resolve names (though it appears to do that throught the IPv6 interface) but anything else times out. What am I missing?

Code: 
root@4100:~# ping -q -4 -c1 10.10.10.2
PING 10.10.10.2 (10.10.10.2) 56(84) bytes of data.


--- 10.10.10.2 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.056/0.056/0.056/0.000 ms
root@4100:~# ping -q -4 -c1 10.10.10.1
PING 10.10.10.1 (10.10.10.1) 56(84) bytes of data.


--- 10.10.10.1 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.048/0.048/0.048/0.000 ms
root@4100:~# ping -q -4 -c1 1.1.1.1
PING 1.1.1.1 (1.1.1.1) 56(84) bytes of data.


--- 1.1.1.1 ping statistics ---
1 packets transmitted, 0 received, 100% packet loss, time 0ms
 
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom
Back