How to start VMs when the primary node fails in two-node cluster without HA

[pxmx_sys]

Hi,
I'm going to bring up a 2-node production proxmox cluster. Already purchased 2x Dell R760 with 60TB storage in each. I'm exploring different options for setting up a cluster on them. As R760 has a BOSS card, I install OS on XFS and the rest of the storage in 2 different ZFS Pools (fast and slow Raidz2 storage).

I have a Synology NAS in the rack and can run a VM in it as Qdevice (3rd node), but I want to find out if I don't want to enable HA and a proper 3-node quorated cluster, what are my options with 2-node cluster and using "two_node: 1" in corosync.conf. so i will create ZFS snapshot on both ZFS pools (fast, and slow) between 2 nodes. my question is how to bring up the VMs in the backup node in case the primary node becomes unavailable.

the applications are low latency apps that use in-memory files. So, we may not be able to use HA so trying to revive the primary node, might be the better solution in my use case and so need to explore the basic operations of a two-node setup.

thanks

 

[mgabriel]

If you just want to be able to start your VMs/CTs on the remaining node, you can issue a pvecm expected 1 on the commandline of the remaining node to reduce the number of votes to 1 and regain quorum.

 

[pxmx_sys]

@mgabriel thanks for your response, i know about "pvecm e 1", also in case of two_node: 1 option, it might be not required anymore. my question was that when the primary node goes down, then in the other node the output of "qm list" is empty. do i need to recreate the VM with "qm create" again or there is a way to start the VM from the zfs snapshot without making too many changes in the backup node, because when the issue of primary node is resolved, you want to keep using the VM in primary node. i haven't seen any documentation or post about this yet.

 

[VictorSTS]

1.- Don't try to reinvent the wheel: use a qdevice in that Synology NAS and don't mess with pvecm expected 1. Such command is for disaster recovery, not for typical operations and may induce configuration conflicts if misused in a split brain situation. Also, corosync "two_node:" isn't officially supported as again in a split brain situation will give quorum to both sides and potentially cause trouble with PMXCFS and VM data, specially if you use replication (both nodes will start the same VM with their local drive).
2.- Use HA + qdevice: VMs will be moved automatically to backup node if primary fails.
3.- If you still insist on doing it manually (or as disaster recovery): just move VM config files from /etc/pve/nodes/pve01/qemu-server to /etc/pve/nodes/pve02/qemu-server from backup server (replace pve01 and pve02 with your hostnames).

 

[pxmx_sys]

@VictorSTS Thanks for your response, you are right. I will use Synology NAS to bring up the 3rd cluster node and set up HA + Qdevice in the cluster. I have another question about the HA setup. As I said, I already purchased 2 x R760 servers (each 0.5TB RAM and 60TB storage). Does using HA Setup mean that all VMs should run on the primary server and the backup server (which in my case is a powerful server) to receive the snapshot every 10 minutes and do nothing unless the primary server becomes unavailable? It would be good if i could run some VMs also in the backup server to use the resources properly. sorry for my dumb questions as I don't have much experience in Proxmox clustering at all.

 

[VictorSTS]

You can have VM/CT in both servers. You can even put some VMs in HA and other not so critical ones without HA and will not be started if the node they are runnin on fails. Also plan to use at least two corosync links to ensure cluster stability due to fencing [1].

[1] https://pve.proxmox.com/wiki/High_Availability#ha_manager_fencing

 

[pxmx_sys]

Thanks, @VictorSTS, so I will put some Critical VMs in the Primary node and will enable HA (and of course ZFS snapshot) for them. I'll scatter non-critical VMs across both nodes and will only create ZFS snapshots for them from Primary to backup and vice versa for the VMs in the Backup node. In this case, i will have a good backup for non-critical machines as well