Going to add a NIC dedicated for pve inter-node traffic, asking for recommendation.

[Tony]

Hi,

I have a Proxmox cluster at Hetzner (no shared storage). I use a feature supported by Hetzner called vSwitch which let customers connect servers via a virtual private network across different locations. My Proxmox nodes at Hetzner are inter-connected (joined) via a private subnet on a vSwitch dedicated for pve traffic.

vSwitch is cool, but unfortunately it's unreliable. So I would like to use a dedicated physical network for pve inter-node traffic. The preparation has been done, all nodes have been moved to a single rack, each node has 2 NICs: one main public NIC and one extra NIC for pve traffic.

Now I am not sure how I should switch to this. The relevant part from /etc/network/interfaces looks like this:

iface enp41s0.4010 inet manual
auto vmbr4010
iface vmbr4010 inet manual
# net4010 private
  bridge_ports enp41s0.4010
  bridge_stp off
  bridge_fd 0
  mtu 1400
  up ip addr add 192.168.110.11/24 dev vmbr4010

enp41s0 is the primary physical NIC. The node uses 192.168.110.11 as the node IP to join the cluster.

My plan is following:

- add to /etc/network/interfaces the following stanza:

auto enp33s0
iface enp33s0 inet static
  address 192.168.110.11
  netmask 255.255.255.0

- bring vmbr4010 down:

ifdown vmbr4010

- bring enp33s0 up:

ifup enp33s0

Is it going to work? I am not sure what happens if the nodes are disconnected from the current vSwitch and reconnected to the physical switch, one by one.

Thanks in advance for any hint.

 

[aaron]

Without knowing or having much experience with Hetzner, the overall approach would be, once the machines boot with the new NIC, to configure a new vmbr that uses this new NIC as bridge port. Use the same X in vmbrX on both servers. Then you can add a second NIC to the VMs that connects to the new vmbr and configure a network on it.

The VMs should now be able to communicate via that new network. Make sure to use an IP subnet that is not in use somewhere else yet.

 

[Tony]

thank you @aaron for your answer.

Since the extra NIC and new network is dedicated to inter-node traffic only, I assume I don't need a bridge for it because no VM would be ever connected to that bridge. Is is correct?

I am also worried about this: if a node is disconnected from the cluster for a while, will split-brain happen? Every node will be disconnected from vmbr4010 (the current vlan for pve cluster), then connected to the new (physical) net over the 2nd NIC using the IP that was on vmbr4010.

 

[aaron]

wait, no VM traffic needs to go accross this? then yeah, you can save the bridge and configure the IP directly on the interface.

Regarding the cluster: It is a 2 node cluster right? Do you use HA? If not, then nothing dramatically will happen. VMs will keep running and once the nodes can talk to each other again you are fine. In the meantime, if the majority of votes is not available, you won't be able to do certain things like starting a guest or changing the config of a guest.

 

[Tony]

I don't use HA. The cluster has more than 2 nodes; does it make a difference?

 

[aaron]

are all going to be rebootet at the same time or one by one? If it is one by one, then you should always have a majority with the remaining nodes. If all reboot at the same time, then that is also okay as they will hopefully find each other after the reboot. If not, you need to troubleshoot the network config

 

[Tony]

I plan to do the switching one by one, starting with less important nodes first and see how it goes.

thank you @aaron again for your help, much appreciated!