High Availability on Hetzner Proxmox setup

[ersefa]

Hi everyone.

I searched for hours but found nothing about the following issue. No sure if it is because is a simply task or the contrary.

We have 3 proxmox nodes on 3 hetzner machines attached to the same vswitch.
Right now its all working smoothly, the guests kvm/cts can ping each other on the different nodes and can reach internet trough their gateways. Each node is the gateway for their guest machines. Each node guests are on a a different subnet, they can communicate with other guests on other nodes trough routing.

The problem appeared when we wanted to activate the HA feature on Proxmox. The guests machines are relocated on other nodes without problem, on node failure, etc. BUT their network connection is lost because the gateway configured on the guest belongs to the old node the machine comes from.

I wonder how we have to configure our network to have HA , maybe a single gateway for all the machines or how we can achieve this on hetzner. On a "physical setup", all the three nodes would be attached to a switch and reach internet trough a router o

This is the configuration of one of our nodes:

auto lo
iface lo inet loopback
iface lo inet6 loopback

#PUBLIC LAN
auto enp4s0
iface enp4s0 inet static
  address public-ip for the node
  netmask 255.255.255.192
  gateway gw-ip
  pointopoint gw-ip

#CLUSTER VSWITCH LAN
auto enp4s0.4000
iface enp4s0.4000 inet static
  address  192.168.100.11
  netmask  255.255.255.0
  vlan-raw-device enp4s0
  mtu 1400
  up route add -net 192.168.12.0 netmask 255.255.255.0 gw 192.168.100.12 dev enp4s0.4000
  up route add -net 192.168.13.0 netmask 255.255.255.0 gw 192.168.100.13 dev enp4s0.4000
  up route add -net 192.168.14.0 netmask 255.255.255.0 gw 192.168.100.14 dev enp4s0.4000

#GUEST LAN FOR PROXMOX
auto vmbr0
iface vmbr0 inet static
address  192.168.11.1
  netmask  255.255.255.0
  bridge-ports none
  bridge-stp off
  bridge-fd 0
  post-up echo 1 > /proc/sys/net/ipv4/ip_forward
  post-up   iptables -t nat -A POSTROUTING -s '192.168.11.0/24' -o enp4s0 -j MASQUERADE
  post-down iptables -t nat -D POSTROUTING -s '192.168.11.0/24' -o enp4s0 -j MASQUERADE

 

[spirit]

I'm currently working to implement sdn on proxmox , with vxlan and internal routing, and anycast gateway (each vmbr on differents host have the same ip).
It's not 100% ready, but I'm targeting it for proxmox 6.2. (or betwwen 6.1 and 6.2)

 

[ersefa]

Thanks Spirit. That sounds nice.
Are you part of the Proxmox team? That will be an official release or something like a plugin or patch?

In the meantime, Hetzner answers that we can buy a subnet for the vswitch the host machines are attached. I suppose that is like providing the virtual switch with an internet gateway. If each host machine have a vswitch subnet public ip, then all guests can have the same gateway to internet (through their vmbr on the host machine and the vswitch gateway).

I dont know if my assumption its right.

If anyone configured something like this, i will be grateful.
Thanks in advance.

 

[spirit]

>>Are you part of the Proxmox team?
No, I'm a contributor since a long time. (and we need some missing features for my company, I develop them and try to push it upstream)

>> That will be an official release or something like a plugin or patch?
Yes, it'll be official. (some part are already integrated). see https://pve.proxmox.com/wiki/Roadmap.

>>In the meantime, Hetzner answers that we can buy a subnet for the vswitch the host machines are attached. I suppose that is like providing the >>virtual switch with an internet gateway. If each host machine have a vswitch subnet public ip, then all guests can have the same gateway to internet >>(through their vmbr on the host machine and the vswitch gateway).
yes, if you have a vswitch on Hetzner, it'll be more easy. (same with ovh vrack for example).
I don't known too much Hetzner, but I think it should work.

 

[ersefa]

Thanks again Spirit.

Im eager for this update to be released. Keep the hard work and thanks in advance for your contribution.

To be honest im not sure how to achieve this setup with the public subnet, All i can find its how to add those Ips to the hosts servers but no clue how the Guests (guest network) have to be configured

 

[ersefa]

Im going to explain my setup furthermore, maybe someone can help:

I have 3 physycal machines with the next interfaces / networks:
- One public IP in their physical NIC
- One private Vswitch network that connect the 3 physycal machines.
- One private vmbr routed network for their guest VMs.

In the private vswitch network, we added routes for communicate each vmbr subnet in each node. That way, every VM can communicate with the rest of VMs in the differente nodes. That provides us with the ability to create cluster configs, like Kafka / Elastic / Galera clusters.

We have this config on another client, that uses Nomad + Consul. In a docker orquestration like that we didnt need to move nodes (the network config never changes), because in case of failure its the docker service what is moved, not the entire VM.
But this new client didnt want a microservice arquitectura, and thats why we are stuck.

We need for example a Maxscale to load balance our Galera. We cant use keepalived approach (with a virtual IP) because we need every maxscale node in the same network (our config right now is a subnet for each node communicated with the router config on physical machines). Also we cannot use a Proxmox HA approach because when the Maxscale node is moved, his network config becomes obsolete and needs to manuallu be changed.

In other words, we need:
- a unique network for all our VMs to maintain the network config validity on a node migration. (We dont know how to achieve this. Maybe with another vswitch and a bridge on each physycal machine to his own vmbrs?)
- a unique gateway for al this VMs to mantain Internet connection.



Knowing that, i dont know if the options are still valid because we not only need to keep inteernet connection but also to mantain the private IP validity on a failover migration.

Thanks in advance for your time and help.

 

[SebastianS]

Hi ersefa,

you can add an additional VLAN to your vswitch for internet connectivity.
For this VLAN you can order IP addresses or a network within the VLAN configuration of your vswitch.
This way you can configure servers with a public IP from your assigned network and move them around between your Proxmox cluster members.
They will be reachable via their public IP no matter, which server they are running on.
You will need to create a VLAN interface on each host and a linux network bridge to make this network available for your guest VMs.

Hope that helps.

Best regards
Sebastian