High Availability with local ZFS storage

[Dihaxu]

Hi, I have inherited a Proxmox cluster with a configuration I'm not familiar with, and am trying to get to the bottom of it.

There are three servers in the cluster, each with identical, local ZFS storage.

Replication is set up between them, with a single VM replicating every 15 minutes from one host to another, and every 30 minutes from the first host to the other.

Everything I have read about HA suggests it needs shared storage, or Ceph, but this setup has neither.

There is an HA group configured with quorum, and the single VM added as a resource on each host. The states are "started" for the running VM, and "ignored" for the others. Fencing just says "Use watchdog based fencing."

Is this an acceptable configuration for automatic failover of the VM in case of host downtime? Obviously if it does work there would be data loss based on the last replication time, but I haven't seen anything to suggest this would even work, as everything I have seen talks about shared storage or Ceph.

Thanks for any thoughts on this.

 

[narrateourale]

Yes, using the VM replication feature in combination with HA is a possible way.

The shortest possible replication interval is 1 minute: */1.

The downside is, that in case of a HA recovery, the VM will be accessing the disk image with the data since the last successful replication. The upside though is, that there is no need for a central shared storage which might be a bit overkill (Ceph) or could be a single point of failure (network share to another box/NAS).
Additionally, if the latency between the nodes is a bit higher, the running VM is not impacted by it since the access to the disk image happens locally within the nodes and the replication itself is asynchronous.

 

[Dihaxu]

Perfect, thank you for the confirmation and information, I appreciate it.

 

[Marco Iñiguez]

Please guys, give me a guide about HA configuration ... I try this configuration and the virtual machines don't boot because they have no quorum. I have to login to the node via terminal and run "pvecm expected 1" to continue booting....

 

[Astraea]

@Marco Iñiguez - How many nodes do you have in your cluster?

 

[Marco Iñiguez]

@Marco Iñiguez - How many nodes do you have in your cluster?

Hi friend! thanks for your fast reply! two nodes...

 

[Astraea]

You cannot do anything HA without 3 nodes or 2 nodes and a QDevice

 

[Dunuin]

Quorum means you always need greater than 50% of your nodes running. If you want to be able to reboot or shutdown any of the two nodes without the other node stop working, you need 3 nodes or 2 nodes + a third machine as a qdevice providing the third vote. That's why 3 machines is the minimum requirement of a cluster.

I would highly recommend that you read this whole article to learn about the cluster basics: https://pve.proxmox.com/wiki/Cluster_Manager

 

[Marco Iñiguez]

Hello friends, thank you for your help... I misunderstood the publication of this thread thinking that if it was possible to configure HA with 2 nodes only, however I see that it is not posible... greetings

 

[dualm]

I have two nodes plus a QDevice on a vm on another hypervisor (ESXi). But you can run the QDevice on a Raspberry Pi. Easy to setup.
HA is working great in the Proxmox cluster with local drives.

 

[Ashkaan4]

I have 8 nodes with properly working HA, local ZFS on each (identical), and replication. When I reboot a host, a FRESH replication occurs instead of making a delta and it takes forever. Anyone have any ideas why it doesn't use the existing data as a base?

 

[justinclift]

@Ashkaan4 In theory (!), it's definitely supposed to do deltas when there's an existing ZFS volume for the same VM.

It's been almost a month since you posted that, have you gotten it figured out in the meantime?

 

[Ashkaan4]

Thank you for asking. We never got it. Between this and some other issues (in another thread), we had to change strategies completely.

 

[justinclift]

No worries at all. Still using Proxmox?

 

[Ashkaan4]

No, I'm trying XCP-ng for this deployment. To my surprise, it seems to be working really well. I have other smaller issues, but I think those will be easier to work out than this issue was.

 

[justinclift]

Interesting. Haven't really looked into XCP-ng before, as the admin interface project on GitHub looked like it was dead when I last checked.

Might take a look at it at some point then.

 

[Ashkaan4]

It's very active!

https://xcp-ng.org/blog/2024/07/12/xcp-ng-8-3-release-candidate-1/

 

[esi_y]

admin interface project on GitHub

What's that? XO? Or the "lite" one?

 

[justinclift]

No idea, it was months ago.

 

[esi_y]

No idea, it was months ago.

I think you meant the self-built XO. Now this is PVE forum, but talking of this architecture-wise, I always found it non-that-much beneficial to have all nodes in PVE serving JS SPA and then relaying it as opposed to the traditional outside-of-the-cluster control pane. It's harder to debug, harder to license meaningfully and I am not sure if anyone e.g. uses round robin to access the web GUI or knows how to really utilise it well.

I get that I do not have to use the other nodes' pveproxy services if I do not wish so, but I definitely MISS with PVE is to be able to deploy that pveproxy standalone.