Hetzner subnet with additional IPs - Driving me crazy for days!

[ShihabSoft]

Hi,

This issue has been driving me crazy for days....

I've a Hetzner Dedi, I've 6 additional IPs for various projects. I needed more IPs so I ordered a /29 subnet from them too. Totalling to 12 IPs...

Here's my /etc/network/interfaces for the Host.

auto lo
iface lo inet loopback

iface enp0s31f6 inet manual
pre-up /usr/sbin/ethtool -K enp0s31f6 tso off gso off
post-up echo 1 > /proc/sys/net/ipv4/ip_forward
post-up echo 1 > /proc/sys/net/ipv4/conf/enp0s31f6/proxy_arp

auto vmbr0
iface vmbr0 inet static
address 195.x.x.194
netmask 255.255.255.248
gateway 195.x.x.193
bridge-ports enp0s31f6
bridge-stp off
bridge-fd 0
up route add -net 195.x.x.192 netmask 255.255.255.248 gw 195.x.x.193 dev vmbr0

auto vmbr1
iface vmbr1 inet static
address 176.x.x.57 (First usable IP in the subnet)
netmask 255.255.255.248
bridge_ports none
bridge_stp off
bridge_fd 0

source /etc/network/interfaces.d/*


My subnet is 176.x.x.56/29 (.56 to .63)


I've followed Hetzner's networking configuration guide and opted the "Network Configuration Bridged" method as all my VMs are KVM.

Bridge vmbr0 is used for VMs with additional IPs and each VM has unique Mac addresses assigned as generated by Hetzner.

VMs with additional single IPs are working perfectly without any issues, that is, it can access the internet and also anyone can access the VMs from the internet.

Bridge vmbr1 is the subnet and used for VMs that comes under the subnet.

Here comes my real problem and the headache of the week.

VMs can ping the host system, also it can access other VMs with additional single IPs.

BUT...

it can't access the internet nor it can't be accessed from the internet.

When I try to ping 1.1.1.1 or any internet IP, the gateway pushes out (Destination host is unreachable)

/etc/network/interface of client VM that comes under the subnet

auto ens18
iface ens18 net static
address 176.9.193.58 (Second usable IP)
netmask 255.255.255.248
gateway 176.9.193.57 (Host subnet IP)

Some command outputs from the host

>>>brctl show
bridge name bridge id STP enabled interfaces
fwbr100i0 8000.4ade6b0f7240 no fwln100i0
tap100i0
fwbr101i0 8000.1e5f25c45715 no fwln101i0
tap101i0
fwbr102i0 8000.c23a9abd20d6 no fwln102i0
tap102i0
fwbr103i0 8000.1e1ef8b0df77 no fwln103i0
tap103i0
fwbr104i0 8000.8e9a5105ab37 no fwln104i0
tap104i0
fwbr105i0 8000.32d707c73ec4 no fwln105i0
tap105i0
vmbr0 8000.901b0ef795e2 no enp0s31f6
fwpr100p0
fwpr101p0
fwpr102p0
fwpr103p0
fwpr104p0
fwpr105p0
vmbr1 8000.8ab69a1f1e36 no tap106i0


>>>sysctl -p
net.ipv4.ip_forward = 1
net.ipv6.conf.all.forwarding = 1

 

[ShihabSoft]

After a lot of fiddling and meddling. Finally I found a solution.

First and foremost, for this bridged and routed combination to work.

When you boot up the Proxmox Dedi

DO NOT START ANY VM.

Connect to SSH shell of the Proxmox node and put the vmbr0 interface down, which is the bridged interface.

Now start any one of the VM which is configured to be used in a subnet (routed setup). Once it is booted up, try pinging an Internet address it should work.

After then you can do ifup vmbr0, bring the bridged interface up.

Finally now you can start all the VMs and get the network working without any issue.

I also added a test VM to test if there's a need for reboot when configuring the network in the new VM, but no it worked out right out of the box.

 

[bvdl]

Thanks for sharing your solution!

Can't wrap my mind around it though...