PVE 3-Notes with HA and ZFS, how to force destination of HA migration?

[Adi M]

I installed an test cluster with 2 Notes (pve02 and pve03) with ZFS for HA as described here https://tonyfernandeztech.wordpress...uring-proxmox-to-replicate-between-two-hosts/
and here https://florianmuller.com/setup-a-p...berrypi-as-a-third-quorum-vote-device-qdevice
But for using HA I need third notes (pve01). I use an low power pc without ZFS.

Now I simulated an system failure on pve03 and the system migrated test CT to pve01 with no ZFS and cannot start (storage is missing, of course).

Q1: How can I bring back CT to pve03?
Q2: How can I force only HA-migration pve02 <-> pve03?

 

[tom]

First, such a setup does not make sense for me, you always loose data, as the zfs replication is not realtime. So this is NOT supported.

Regarding your question, start reading on:

https://pve.proxmox.com/pve-docs/pve-admin-guide.html#chapter_ha_manager

 

[spirit]

Q1: How can I bring back CT to pve03?

manually ? with live migration + storage migration.
Auto with HA? set a priority weight on a node in HA group.

Q2: How can I force only HA-migration pve02 <-> pve03?

simply create an HA group with your 2 zfs nodes with option "restricted".

 

[Adi M]

Thank you for your help

@tom: I know, my setup is not really practicable, but the only way I have found until now for a file system for cluster, raid and snapshots available. I am looking for a step-by-step cookbook.
@spirit: Thank you for answer Q2, that's what I'm looking for. For Q1: CT is not running again it's on error state. I cannot migrate back manually because of storage missing. Maybe I have change something in config file on pve back-end?

 

[tom]

You can run a 2-node cluster, you use zfs and you have snapshots.

But do NOT enable HA. The basic idea of HA is increasing availability of your services, but in your case, you just reduce up-time.

 

[Adi M]

Ok, I see. What about Ceph? Is it possible to take snapshots and HA?
And what about raid? Can Ceph handle this or have I to use tools like mdadm?

 

[Adi M]

Ok there is a point I don't understand :
When I use ZFS on BVE02 and PVE03 and replication is activated (no HA), how can I migrate CT from PVE03 to PVE02 if PVE03 is crashed?
When I select the CT on "death" PVE03 and press migrate, I get error in GUI "Connection error 595: No route to host".
On PVE02 the desired CT is not available.

 

[spirit]

Ok there is a point I don't understand :
When I use ZFS on BVE02 and PVE03 and replication is activated (no HA), how can I migrate CT from PVE03 to PVE02 if PVE03 is crashed?
When I select the CT on "death" PVE03 and press migrate, I get error in GUI "Connection error 595: No route to host".
On PVE02 the desired CT is not available.

you need to do it manually through command line

on working node :

#pvecm expected 1 (to be able to write in /etc/pve)
#mv /etc/pve/nodes/failednode/qemu-server/<vmid>.conf /etc/pve/nodes/workingnode/qemu-server/<vmid>.conf

 

[Adi M]

Ok, thank you