Can proxmox node with 2 interfaces/IPs join a cluster

[aveekkumar]

Hi Community,

We have a proxmox cluster with 3 nodes. Each node has a Mellanox 100G [ConnectX-5] NIC which is split into 2 interfaces. We typically configure these two interfaces as a bond for redundancy.

==================================
auto bond0
iface bond0 inet manual
bond-slaves eno1 eno2
bond-miimon 100
bond-mode 802.3ad
bond-xmit-hash-policy layer3+4
mtu 9000

auto vmbr0
iface vmbr0 inet static
address 10.0.20.24/22
gateway 10.0.23.254
bridge_ports bond0
bridge_stp off
bridge_fd 0
bridge-vlan-aware yes
mtu 9000
==================================

However, the interfaces are connected to switches that do not have MLAG capabilities and cannot be bonded. Our networking team has suggested assigning two different IPs to both the interfaces for redundancy.

Is it possible to add the 2nd IP of the node to the proxmox cluster?

This is the current status of the cluster where each node has 1 IP.
==================================
Cluster information
-------------------
Name: sys-ny4-dmz
Config Version: 3
Transport: knet
Secure auth: on

Quorum information
------------------
Date: Sat Nov 4 08:29:21 2023
Quorum provider: corosync_votequorum
Nodes: 3
Node ID: 0x00000002
Ring ID: 1.1a
Quorate: Yes

Votequorum information
----------------------
Expected votes: 3
Highest expected: 3
Total votes: 3
Quorum: 2
Flags: Quorate

Membership information
----------------------
Nodeid Votes Name
0x00000001 1 10.0.20.9
0x00000002 1 10.0.20.24 (local)
0x00000003 1 10.0.20.25
==================================

 

[mgabriel]

If you want to achieve redundancy for the cluster connection, corosync has already all you need.

just use two independent interfaces with different IP addresses in different ranges and assign them to two corosync links/rings.

let's say the first interface of each node is in the 10.10.10.0/24 network, the second interface is in the 10.10.20.0/24 network. all you need to do is connecting the first interface of each server to the first switch and the second interface of each server to the second switch.

see https://pve.proxmox.com/pve-docs/pve-admin-guide.html#pvecm_redundancy for more details.

 

[VictorSTS]

You should use a dedicated network connection for corosync, at least for one of its links. Still, if you want to share corosync traffic with everything else, simply use a different type of bond that does not require switch configuration/compatibility [1]. Anyone except broadcast and round robin should give you decent performance and reliability.

[1] https://pve.proxmox.com/wiki/Network_Configuration#sysadmin_network_bond