HA architecture: 5-node Proxmox cluster vs. separate compute and storage clusters

[TS Guru]

Hi,

we are currently designing a relativelly small HA infrastructure and would appreciate some opinions from experienced Proxmox users.

Our infrastructure consists of:

• C1, C2 – Proxmox compute servers
• S1, S2 – TrueNAS storage servers providing NFS, with DRBD replication
• W1 – separate witness/voting node
• All servers are connected through a dedicated fibre network (internal) as well as a regular LAN.

We are currently discussing two possible architectures:

Scenario 1:
Create one cluster consisting of C1, C2, S1, S2 and W1, with a 3-of-5 quorum. S1 and S2 would also provide compute resources in addition to their storage role.

Scenario 2:
Keep the roles separated: C1 + C2 with W1/QDevice as the Proxmox compute cluster, while S1 + S2 form a separate storage HA cluster using DRBD/TrueNAS.

The main discussion is whether the additional compute capacity and 3-of-5 quorum of Scenario 1 provide a real HA advantage, or whether separating compute and storage as in Scenario 2 provides better fault isolation and a more robust architecture.

I would be interested in your experience and opinions, particularly regarding the advantages/disadvantages, quorum behavior, failure scenarios, maintenance, and potential risks of running Proxmox workloads on the storage nodes.

Which architecture would you recommend for a production HA environment, and why?

 

[driley]

hi,
It might be worth looking into Ceph as well for your high-availability cluster, as it is natively integrated into Proxmox VE. This could potentially eliminate the need for a separate TrueNAS layer while taking full advantage of your hardware for fault tolerance.

You can read up on it here: PVE Ceph Docs

 

[LnxBil]

some opinions from experienced Proxmox users.

I would suggest that experiencesd Proxmox VE users would go with a simple, but supported solution that is battle proved like 3 or 5 node CEPH cluster.

It's always complicated to setup a migration on available hardware instead of planing a new environment and buying proper hardware for this.
I would not go with scenario 1 or 2 but have no better solution to use all compute nodes besides buying more disks and going with 3 CEPH nodes and keep most of the 4rd node as replacement parts.