Hetzner with PVE 6.05, bridged VMs, not available

thommie · Aug 19, 2019

Hi all,
I am a newbie to proxmox, so please apologize if my questions are a bit "basic"

I have a running proxmox 6.05 on a Hetzner EX42 server with debian 10, bridged setup and two bridged VMs (ubuntu 18 lts). Setup was done following the documentation at the proxmox wiki for Debian Buster plus other sources from hetzner dokuwiki plus some older public tutorials which I adapted for ubuntu 18 and debian 10 (especially as ubuntu 18 now uses netplan).

My problem: the bridged VMs do not connect to the outside world and vice versa. But both VMs can reach each other and the proxmox server, so I assume that the vmbr0 setup is principally ok. From my understanding of the routing, bridged VMs are routing directly to the Hetzner gareway, which leads to this guest configuration:

network:
version: 2
renderer: networkd
ethernets:
ens18:
dhcp4: no
addresses: [REDACTED/26]
gateway4: [REDACTED]
nameservers:
addresses: [REDACTED,REDACTED]
routes:
- to: 0.0.0.0/0
via: [REDACTED]
on-link: true

On the proxmox server I use the "out of the box" configuration (firewall is off on datacenter level). I need some hints where to search from now ...

Thanks, Thommie

spirit · Aug 19, 2019

any mac address restriction with Hetzner ?

thommie · Aug 19, 2019

I have separate MACs for both VMs and added these MACs in the MAC Address field in "Edit: Network device" on the VM level. Just re-checked if I have the correct MACs and all seems to be fine ... Are there any additional checks I can make on the cli level?

spirit · Aug 19, 2019

can you send your host /etc/network/interfaces ?

thommie · Aug 19, 2019

Code:

auto lo
iface lo inet loopback

iface lo inet6 loopback

auto enp0s31f6
iface enp0s31f6 inet static
        address  [REDACTED]
        netmask  255.255.255.192
        gateway  [REDACTED]
        up route add -net [REDACTED] netmask 255.255.255.192 gw [REDACTED] dev enp0s31f6
        # route [REDACTED]/26 via [REDACTED]

iface enp0s31f6 inet6 static
        address  [REDACTED]
        netmask  64
        gateway  fe80::1

auto vmbr0
iface vmbr0 inet static
  address [REDACTED]
  netmask 255.255.255.192
  # pointopoint [REDACTED]
  # gateway [REDACTED]
  bridge_ports none
  bridge_stp off
  bridge_fd 0
# additional IPs
  up ip route add [REDACTED]/32 dev vmbr0
  up ip route add [REDACTED]/32 dev vmbr0

iface vmbr0 inet6 static
  address 2a01:4f8::3
  netmask 64
  up ip -6 route add [REDACTED]/64 dev vmbr0

From the proxmox machine itself everything seems to be OK ....

spirit · Aug 19, 2019

are you sure that Hetzner allow a bridge setup for the vms ?
(I'm seeing up "ip route add [REDACTED]/32 dev vmbr0" on your host, and it's the vm ip address).
sound like a routed setup, where your need to use your proxmox ip address as gateway for your vms.

thommie · Aug 19, 2019

Hm, I used the documentation at https://community.hetzner.com/tutorials/install-and-configure-proxmox_ve/de?title=Proxmox_VE
But maybe I mixed something up here ...

I could also switched to a routed setup if this has advantages. Which one is better, seen from the PVE point of view?

spirit · Aug 19, 2019

thommie said:
Hm, I used the documentation at https://community.hetzner.com/tutorials/install-and-configure-proxmox_ve/de?title=Proxmox_VE
But maybe I mixed something up here ...

I could also switched to a routed setup if this has advantages. Which one is better, seen from the PVE point of view?

It's depend if you need live migration between multiple proxmox hosts. (In this case you need a bridged setup)

thommie · Aug 19, 2019

no, this is my first proxmox server, migration will be done manually from another virtualbox server

thommie · Aug 19, 2019

ok, the problem was the bridged setup, the routed setup works fine:

Code:

auto lo
iface lo inet loopback

iface lo inet6 loopback

auto enp0s31f6
iface enp0s31f6 inet static
        address  [REDACTED]
        netmask  255.255.255.192
        gateway  [REDACTED]
        pointopoint [REDACTED]
        gateway [REDACTED]
        # from bridged setup
        # up route add -net [REDACTED] netmask 255.255.255.192 gw [REDACTED] dev enp0s31f6
        # route [REDACTED] via [REDACTED]

iface enp0s31f6 inet6 static
        address  [REDACTED]
        netmask  64
        gateway  fe80::1

auto vmbr0
iface vmbr0 inet static
  address [REDACTED]
  netmask 255.255.255.192
  bridge_ports none
  bridge_stp off
  bridge_fd 0
# additional IPs
  up ip route add [REDACTED]/26 dev vmbr0
  up ip route add [REDACTED]/26 dev vmbr0

iface vmbr0 inet6 static
  address [REDACTED]
  netmask 64
  up ip -6 route add [REDACTED]/64 dev vmbr0

Retuas · Aug 31, 2019

Can you please say at least which IP (Main, Additional 1 etc) goes in your configuration? "REDACTED" wont help me here.

bofh · Sep 1, 2019

thommie said:
I have separate MACs for both VMs and added these MACs in the MAC Address field in "Edit: Network device" on the VM level. Just re-checked if I have the correct MACs and all seems to be fine ... Are there any additional checks I can make on the cli level?

uhm what?
what do you mean exactly you added those mac adress es on the vmlevel?

you need to add the VMs mac adresses in hetzners robot.
take one of the assigned ips an ad the corresponding mac adress to this ip.
public ip on the vm need to match with the mac of the vm in hetzners portal.

otherwise hetzners switches gonna ignore you.

thommie · Sep 1, 2019

I am now using a routed setup instead and all works fine. The original problem is solved and the thread can be closed.

PS. yes, I know, the macs for bridged setup have to be requested and set in the hetzner robot

Carsten Martens · Sep 2, 2019

thommie said:
ok, the problem was the bridged setup, the routed setup works fine:
[...]

I have the same Problem for our Net. Kann you tell uns wich ist the Main-IP, wich one the the gateway and so on. I think it would help us.

Tanks a lot

thommie · Sep 2, 2019

Example network/interfaces for the proxmox server (check your own settings as provided in the hetzner robot):

auto enp0s31f6
iface enp0s31f6 inet static
address [public ip of hetzner root server]
netmask 255.255.255.192
gateway [hetzner gateway]
pointopoint [hetzner gateway]
gateway [hetzner gateway]

iface enp0s31f6 inet6 static
address 2a01:4f8:172:d22::2
netmask 64
gateway fe80::1

auto vmbr0
iface vmbr0 inet static
address [public ip of hetzner root server]
netmask 255.255.255.192
bridge_ports none
bridge_stp off
bridge_fd 0
# additional IPs
up ip route add [additional ips//26 dev vmbr0
up ip route add [additional ips//26 dev vmbr0
up ip route add [additional ips//26 dev vmbr0

iface vmbr0 inet6 static
address 2a01:4f8::3
netmask 64
up ip -6 route add 2a01:4f8::/64 dev vmbr0

========================================================

Example for a VM (netplan yaml for ubuntu 10.04 lts):

network:
version: 2
renderer: networkd
ethernets:
ens18:
dhcp4: no
addresses: [vm ip]
gateway4: [ip of proxmox bridge]
nameservers:
addresses: [hetzner dns servers]
routes:
- to: 0.0.0.0/0
via: [ip of proxmox bridge]
on-link: true

Carsten Martens · Sep 2, 2019

Tank you very much.
Did you get a Subnet or three additional ips?

thommie · Sep 2, 2019

additional ips

bofh · Sep 3, 2019

for hetzner i use a NAT setup.

outgoing and incomming nat port mappings done with fwbuilder.
resons for this is:

moving VMs to another server requires manual intervention anyway except you pay an extra premium for an failoverip
your ips are bound to the host.
that can have major implications when moving VMs specially when they use services with each other.

advantages:
- i can use one ip for multiple servers as long their services/ports dont overlap
- a bit better isolation and easier config for internal services and mappings without a second networkdevice each VM
-with hetzners vlans i can even go with a hackidy solution to simply map that vlan to that bridge
result is:
all hosts share the same subnet for the vms
each host can be used by each vm as a gateway if i want to (kinda virtual routers for the poor)
-moving an vm to another host now requires no ip change on the host, (which can be major on certain services like IP ACLs on mysql servers)
instead you change the default gateway on the VM and make shure NAT is set accordingly on the Host
-at least temporarly use an IP from the other host if lets say you run out of IPs on the other

so basically each promox host becomes a virtual router (that ofc is doing not routing but nat)

shure its not optimal, but thats in the nature of the beast hetzner.
with manual switching failoverips (that costs a premium), or nonswitchable ips the use of public ips for vms
is questionable.

i dont see any real benefit here, just headaches at migration when you need to reconfigure every vm.
in best case only its networking, work case a lot of deamons, acls, dns/make shure dns is updated) have more downtimes...

thommie · Sep 3, 2019

Both strategies (routed/bridged or NAT) have their pros and cons and may be adequate, depending on the detailed use cases.
In my case, I used a NATted setup on a previous hetzner server in combination with headless virtualbox VMs, now I switched to a routed setup on a new server and for proxmox/KVM-based virtualisation. The changes on the VM side are minimal. In both cases I need separate public IP adresses for 1 €/per month (which is OK compared to the price of the root server itself).

bofh · Sep 3, 2019

thommie said:
In both cases I need separate public IP adresses for 1 €/per month (which is OK compared to the price of the root server itself).

while 1 € is still extortion, the mean thing is those are bound to the host, and failoverips costs you 5€ a month. and you need to buy them seperatly because a fialover network can only be switched together.

and to compare that ovh ip cost you 2€once no monthly.

so if you need a lot of public ips, as i do for one farm, i run there 100ish vms each with public ips, i save alone 20€ a month per host by not paying that 1€, plus they are failover (another 80 compared to hetzner).
and now the dediserver with server hardware and better networking, dedicated private lan via vrack and IPMI cost you the same as a lowend consumerhardware box at hetzner

just saying dont understimate little costs for little things.
the moment you scale up you gonna be surprised how prices merge and how much it matters what exactly you beeing offered as a included service

Hetzner with PVE 6.05, bridged VMs, not available

Member

Distinguished Member

Member

Distinguished Member

Member

Distinguished Member

Member

Distinguished Member

Member

Member

Member

Well-Known Member

Member

New Member

Member

New Member

Member

Well-Known Member

Member

Well-Known Member