Proxmox VE Cluster vs Non-Cluster

[silverstone]

I am (re)debating this Issue, as I have done in the Past.

My use Case, as a Homelab user, is that some/many Hosts are up only when needed (at specific Times, in order to reduce Power Consumption), and so the "normal" way with Quorum doesn't really work.

I had a look at https://forum.proxmox.com/threads/proxmox-cluster-no-ha-only-config.131497/ and it brings more Questions than Answers.

In the Past I coded some workarounds (https://forum.proxmox.com/threads/cluster-made-of-permanent-adhoc-nodes.53591/) but not sure if the same Concept would work in this Case.

I could have 4-5 "Main" / "Master" Hosts permanently up with say 20 Votes each, then all other Hosts would work whether up or down.

I do NOT require HA.

What I mainly miss are:
- Migrate / Move a VM to a different Host (permanently). This is the main Feature I miss right now. I would otherwise Export to .raw Image, copy Config File, then Re-import on the Destination VM. I could also of course do a zfs send | ssh zfs receive Manually and scp the config file, although I feel like that Snapshots that have been taken through the GUI do NOT get migrated

Other Feature I might miss but I guess can be worked around (I guess I could just use salt or scp to Synchronize those):
- (When I'll start using them) Synchronized Firewall Security Profile / IP Set across the Hosts
- (When I'll start using them) Synchronized Firewall Rules across the Hosts
- Synchronized Tags Names/Colors/etc across the Hosts

 

[leesteken]

What I mainly miss are:
- Migrate / Move a VM to a different Host (permanently). This is the main Feature I miss right now. I would otherwise Export to .raw Image, copy Config File, then Re-import on the Destination VM. I could also of course do a zfs send | ssh zfs receive Manually and scp the config file, although I feel like that Snapshots that have been taken through the GUI do NOT get migrated

There is ongoing work on migration of VMs without clustering ( https://forum.proxmox.com/threads/multi-datacenter-management.66830/page-2#post-515033 ) but AFAIK it's still command-line only. I have no experience with that but maybe you can find more details now you know that it exists.

 

[esi_y]

I am (re)debating this Issue, as I have done in the Past.

My use Case, as a Homelab user, is that some/many Hosts are up only when needed (at specific Times, in order to reduce Power Consumption), and so the "normal" way with Quorum doesn't really work.

I had a look at https://forum.proxmox.com/threads/proxmox-cluster-no-ha-only-config.131497/ and it brings more Questions than Answers.

In the Past I coded some workarounds (https://forum.proxmox.com/threads/cluster-made-of-permanent-adhoc-nodes.53591/) but not sure if the same Concept would work in this Case.

I could have 4-5 "Main" / "Master" Hosts permanently up with say 20 Votes each, then all other Hosts would work whether up or down.

There's multiple configurations of corosync with most notable being the last man standing, lots of caveats like waiting for all necessary. Similar could be achieved with a single Q device (this is also functionality not well tested), but like most here, I think you really do not need a cluster. Not to mention the inability to help you on the forum later because no one wants to spend time on perfectly normal corosync config that they have however not tested well.

I do NOT require HA.

And unfortunately this is always on hardcoded like many pre-decided things in PVE design.

What I mainly miss are:
- Migrate / Move a VM to a different Host (permanently). This is the main Feature I miss right now.

As previously mentioned, there's the qm remote-migrate but like many things, not officially supported beyond "technical preview" (for a long time now).

I would otherwise Export to .raw Image, copy Config File, then Re-import on the Destination VM. I could also of course do a zfs send | ssh zfs receive Manually and scp the config file, although I feel like that Snapshots that have been taken through the GUI do NOT get migrated

ZFS send/receive (you may want to really pipe it via mbuffer) is really the way to go, another of those missing features.

Other Feature I might miss but I guess can be worked around (I guess I could just use salt or scp to Synchronize those):
- (When I'll start using them) Synchronized Firewall Security Profile / IP Set across the Hosts
- (When I'll start using them) Synchronized Firewall Rules across the Hosts
- Synchronized Tags Names/Colors/etc across the Hosts

Did you consider you have been using the wrong tool all along?

 

[UdoB]

HA:

And unfortunately this is always on hardcoded like many pre-decided things in PVE design.

You must differentiate between two things: 1) Quorum to operate the cluster and 2) HA to ensure the availability of a service. (Ignoring Ceph here as it is not used here.)

HA will restart (fence) a Node if it is supplying a HA-enabled service and Quorum is lost. If there is no HA enabled... it will NOT fence/reboot a node. If you do not specify explicitly services (a VM) to "be HA" then there is none. So no fear for "unexpected reboots".

The "normal" Quorum is required to operate the cluster. When it is lost all VMs will keep their state, be it "stopped" or "running" - no interruption will happen.

But you can not start/stop/create things. Tools like an external/cheap/simple Quorum Device help very small clusters with an even number of Nodes to skip that hurdle. And there are ways to define different numbers of votes for some cluster members - those that are up and running always may have two votes while the others have only one.

At there very end of the list of options there is the famous "pvecm expected x" command, which is useful in a Homelab with some Nodes being turned off most of the time.


Personally I prefer to have a cluster in my Homelab. Some services (DNS/Pihole, DHCP, Router/Firewall, VPNs, selfhosted Cloud services ... ...) are important enough for me to make sure they are reliably available...

Just my 2 €¢ on a Sunday morning

 

[esi_y]

If there is no HA enabled... it will NOT fence/reboot a node. If you do not specify explicitly services (a VM) to "be HA" then there is none. So no fear for "unexpected reboots".

This is by the docs, unfortunately there's phantom bugs in the codepath and the watchdog is active at all times. Major issue of PVE is cannot really turn off HA other than by nasty manual effort.

EDIT: See e.g. https://forum.proxmox.com/threads/getting-rid-of-watchdog-emergency-node-reboot.136789/#post-635602
(This is for the OP, I just noticed @UdoB was there too. I now cannot find the other thread where CTO reacted on how it should not be happening, etc. It was never filed as a bug as far as I know. I could not reproduce it at the time, but strangely there's multiple threads with the same topic over time on the forum.)

Personally I prefer to have a cluster in my Homelab.

I do not want speak on behalf of the OP, but he never said his members get to 1 at any point, he just wanted to have flexible quorum as his solution scales up and down - very normal requirement nowadays.