Making best use of multiple NICs per Node

[maxim.webster]

Dear all,

I am upgrading the hardware of my 3-node-homelab-cluster. Currently, every node has a single NIC (GbE).
The new hardware has 4 NICs per node (2.5GbE) and I want to make best use of them.

How should I set-up routing for my cluster, given
* corosync
* multi-vlan-traffic from and to the network using a vlan-aware bridge (vmbr0)
* no distributed storage (Ceph), but ZFS mirrors on each node with HA and replication

Thanks in advance

Maxim

 

[JD-iteee]

Hi Maxim,

with 4x 2.5GbE per node and your setup using Corosync, HA and ZFS replication without Ceph, I would separate the traffic types as much as possible to keep the cluster stable and easier to troubleshoot.

I would use one dedicated NIC only for Corosync traffic with static IPs and without bonding. This keeps cluster communication isolated and avoids issues during high network load.

For the normal VM, management and VLAN traffic, I would use two NICs together for the VLAN-aware bridge (vmbr0). Active/Passive bonding is completely sufficient for a homelab and gives good redundancy. If your switch supports proper LACP/802.3ad, you could also use that instead.

The remaining NIC would then be used exclusively for ZFS replication traffic. Having a dedicated replication network keeps large replication jobs away from Corosync and VM traffic and improves overall cluster stability during failover or sync operations.

If you do not need LACP or other switch-based features, you could also use two direct connections between the Proxmox nodes only for replication traffic by using two ports per node for an isolated internal replication network. That keeps the replication traffic completely separated from the main network and avoids unnecessary load on the switch infrastructure.

That is actually the setup I use in my own homelab and it works very reliably for ZFS replication.

Regards
Jonas