Node requirements for a new cluster

D

duncanh

New Member
Feb 18, 2025
3
0
1
Greetings all,

I am new to Proxmox and currently investigating the possibility of setting up a scalable virtual infrastructure using Proxmox VE and have what is probably a very "noob" sort of question.

Is there a requirement for the hardware nodes in a cluster to be matching (i.e. CPU, RAM, manufacturer? etc.) or can the cluster grow organically over time using a variety of hardware?

My current thinking is to start with a set of 3 (or more) matching servers and a SAN, then adding extra "tin" in the future to allow us to migrate some of our existing VMs into the same virtual environment.

Many thanks in advance
 
Hello duncanh! If you want to perform live migrations between the nodes in the cluster, there are some restrictions regarding the CPU type explained in the documentation. In general, having exactly the same CPU on all servers and setting the VM's CPU to host should theoretically give you the highest performance, but you can also use one of the more general QEMU CPU types, or one of the Intel/AMD CPU types, to make sure that you can migrate between servers with different CPUs. However, as noted in the documentation, keep in mind that
Live migrations between Intel and AMD host CPUs have no guarantee to work.
Click to expand...
You can still perform offline migrations though, if you can afford some downtime for the VMs.

As you can see, having similar (or the same) hardware is not strictly necessary, but does have its advantages.

Last but not least, if you didn't buy the servers yet, make sure to take a look at the System Requirements. Enterprise hardware is recommended for the best reliability, including ECC RAM and datacenter-grade SSDs (with power-loss protection).
 
Last edited:
You will also need to think about the shared storage ( Ceph would be one possibility or reusing an existing NAS/SAN) and the network requirements. The cluster needs a dedicated network for cluster communnication, Ceph at least a 10/ 25 Gbit/s network just for ceph:
https://pve.proxmox.com/wiki/Cluster_Manager#_cluster_network

https://pve.proxmox.com/wiki/Deploy...r#_recommendations_for_a_healthy_ceph_cluster

UdoB

[TUTORIAL] Thread 'FabU: can I use Ceph in a _very_ small cluster?'

Ceph is great, but it needs some resources above the theoretical minimum to work reliably. My assumptions for the following text:
  • you want to use Ceph because... why not?
  • you want to use High Availability - which requires Shared Storage (note that a complete solution needs more things like a redundant network stack and power supplies)
  • you want to start as small (and cheap) as possible, because this is... “only” a Homelab
You plan for three Nodes. Each node has s single dedicated disk for use as an “OSD”. This is the documented minimum...
  • Like


Regarding scalability: aAccording to the docs you can't scale too much vecause at some point stuff like network latencies get you into trouble, even with a dedicated cluster network. But there are reports of working clusters up to 30-50 nodes.
 
l.leahu-vladucu said:
Hello duncanh! If you want to perform live migrations between the nodes in the cluster, there are some restrictions regarding the CPU type explained in the documentation. In general, having exactly the same CPU on all servers and setting the VM's CPU to host should theoretically give you the highest performance, but you can also use one of the more general QEMU CPU types, or one of the Intel/AMD CPU types, to make sure that you can migrate between servers with different CPUs. However, as noted in the documentation, keep in mind that

You can still perform offline migrations though, if you can afford some downtime for the VMs.

As you can see, having similar (or the same) hardware is not strictly necessary, but does have its advantages.

Last but not least, if you didn't buy the servers yet, make sure to take a look at the System Requirements. Enterprise hardware is recommended for the best reliability, including ECC RAM and datacenter-grade SSDs (with power-loss protection).
Click to expand...
Thanks for this, it is very helpful and good to know. I hadn't actually considered the migration across different hardware types so thank you for the heads up on that potential pitfall.

In terms of hardware, I think I would be looking for Enterprise hardware anyway as (for me at least) the end game is to run business critical VMs in here so we would need all of the added goodness that comes with that level of hardware anyway.
 
Johannes S said:
You will also need to think about the shared storage ( Ceph would be one possibility or reusing an existing NAS/SAN) and the network requirements. The cluster needs a dedicated network for cluster communnication, Ceph at least a 10/ 25 Gbit/s network just for ceph:
https://pve.proxmox.com/wiki/Cluster_Manager#_cluster_network

https://pve.proxmox.com/wiki/Deploy...r#_recommendations_for_a_healthy_ceph_cluster

UdoB

[TUTORIAL] Thread 'FabU: can I use Ceph in a _very_ small cluster?'

Ceph is great, but it needs some resources above the theoretical minimum to work reliably. My assumptions for the following text:
  • you want to use Ceph because... why not?
  • you want to use High Availability - which requires Shared Storage (note that a complete solution needs more things like a redundant network stack and power supplies)
  • you want to start as small (and cheap) as possible, because this is... “only” a Homelab
You plan for three Nodes. Each node has s single dedicated disk for use as an “OSD”. This is the documented minimum...
  • Like


Regarding scalability: aAccording to the docs you can't scale too much vecause at some point stuff like network latencies get you into trouble, even with a dedicated cluster network. But there are reports of working clusters up to 30-50 nodes.
Click to expand...
Thanks. I had read some quite horrifying posts about Ceph needing at least 5 nodes to be feasible and some others that said 3 nodes was fine. My thinking on the scalability was to initially buy nodes that have spare slots rather than adding more nodes so we could start small and then scale the existing nodes as well as adding additional nodes as our usage grows. I'm not sure we would ever get to the lofty height of 30 nodes (not sure we have the physical space :D
) so can't see us pushing the envelope on that front.

My thinking for a Ceph setup was to have fibre channels to a SAN controller (volume configured on collection of RAIDed disks) but, having just read the Ceph article you linked to, using RAID controllers isn't recommended so I may need to reconsider that.
 
You must log in or register to reply here.
Top Bottom