Flexible number of nodes in a Proxmox cluster

[hoba]

Hi,

I'm using a Proxmox cluster with 6 nodes (HP DL360 Gen8 servers) on which also a Ceph cluster is configured. Is it possible to flexible power-off nodes in times where less performance is used and power them on again when I need the performance? In my scenario I would like to power-off 3 servers out of 6 to save energy costs. Is that possible and even recommended with retaining a healthy cluster state (Proxmox + Ceph)?

Thanks for any input,
hoba

 

[oguz]

hi,

to maintain quorum in the cluster you'll need more than half (so for 6 servers, you'll need 4) of the nodes running.

you could look into setting up a qdevice [0] to add an extra vote in your cluster. then you should be able to take the 3 of them offline without losing quorum (since with the extra vote you'll have 7, then taking 3 offline would leave 4 votes in the cluster)

[0]: https://pve.proxmox.com/pve-docs/chapter-pvecm.html#_corosync_external_vote_support

 

[hoba]

Okay I see, however as a downside if I understand correctly, then in such a case the high-availability would be gone.

Edit: would it instead be an option to edit the "quorum_votes" in the cluster/corosync config?

Example:
nodes pve1, pve2, pve3 = quorum_votes = 7
nodes pve4, pve5, pve6 = quorum_votes = 2

So if pve4, pve5, pve6 and e.g. pve1 are down, then the quorum votes still are 14/27 which remains quorum.

Would that work?

 

[UdoB]

Would that work?

Probably. But keep in mind:

• with 6 machines two may fail without loosing quorum of the complete cluster
• with 3+QDevice none may fail! It just fulfills the absolute minimum requirement. If any one device fails, the cluster functionality will stop to work. (I am not really sure about fencing to apply, triggering reboots.)

At least this is my understanding... please correct me if I'm wrong.

 

[need2gcm]

I want to bring up that Ceph cluster again. Are you expecting there to be OSDs on the nodes you power off to save power? If so Ceph will have to rebalance every time you turn servers off/on, and that will have considerable wear on your storage medium, slow down your storage pools, and cause increased resource use on every Ceph server.

If you only run OSDs on servers that will always be on, then this won't be an issue for Ceph. You are always planning on having three servers on, so that is enough for Ceph redundancy until you have one node failure.

With an external fencing system, you could have one of the off servers brought online if an always on server fails, but there will be delay time while you wait for the server to fully boot, and if you don't have quorum for that whole time, that could cause issues.

 

[hoba]

My idea with the Ceph cluster is, to remove the OSDs on the 3 nodes I want to power off and let Ceph rebalance itself. After that, I would turn the nodes off (considering the quorum_votes are configured as mentioned in my previous post). The nodes would not be turned on/off frequetly but rather 1-2 times a year.

I don't know if that idea is maybe too naive...

 

[hoba]

Hi,

I wanted to provide some feedback as I've applied the above changes to the cluster, and it seems to work pretty well. So to recap, I've done the following steps to power-off 3 nodes, out of 6:

1. Change "quorum_votes" for pve1-pve3 to 7, for pve4-pve6 to 2.
2. On nodes pve4-pve6, put all OSDs to status "down" and "out".
   • allow enough time for Ceph cluster rebalance itself.
3. Power-off nodes pve4-pve6.

Cluster quorum remains, Ceph is in healthy status and it's still high-availability as it allows for 1 node to fail.

To power-on the nodes pve4-pve6 again, I need to reverse the above order.

I'm happy for any feedback and hope, I could help others with similar questions.

Regards,
hoba