Disadvantages of non-HA cluster?

[sshaikh]

I have two nodes running at home. They each use local storage for images (backup, HA or fast migration aren't really necessary here), with each guests "manually" accessing a NAS for non transient data.

I'm looking at running the two nodes in a cluster, if only to manage them from a single web UI as well as easy VM migration (which I assume will involve an image copy since everything is local).

Are there any downsides to setting up such a trivial cluster? It won't have a dedicated cluster network, although it doesn't have a storage network either.

Is it easy to tear down the cluster afterwards? Is there a management/admin burden I'll be adding for little gain?

 

[sshaikh]

Also to add: would I be able to shut down one node without affecting the other? As I said I'm not too interested in the HA side of things (can HA be disabled?)

 

[Mitch Maris]

Following...

 

[guletz]

Maybe, you can think at HA/cluster I was in exactly your situation, and I discover that a box like Raspberry Pi(I have Odroid-c2), can be use for the 3rd node. Now my Odroid-c2, is running all the time, and sometimes I have one server that is running 16 hours /day(sometimes I use both servers). But I can use all the good things... live migration, pve-sync, and so on.

 

[sshaikh]

Do you need a third vacuous node at all? My understanding was that a third was needed just for HA (something I'm not necessarily looking for)

 

[Denny]

You don't need the third, it is just recommended. You can change the expected quorom votes to one via

pvecm expected 1

You can pseudo live migrate with such a setup by setting up storage that is available via nfs on both machines and marking them shared. The trick is move the disk first and then migrate the machine.

I would really recommend using the third node though.

 

[sshaikh]

Hmm. I think what I need is some kind of feature matrix comparing:

• 2 standalone nodes.
• 2 nodes as a cluster.
• 2 nodes + a "dummy" third node.

I suspect I'm not quite after the features the third solution will give me (afaik: HA and live migration), so my question is: is the cheap (ie software only configuration) solution of clustering 2 nodes worth doing, or will it add potential problems over keeping them separate?

 

[Mitch Maris]

As having been through this exact same dilemma myself, here's what I ran into and my conclusion.

-Requirements I wanted to run a single server and have no downtime of my VMs for updates of the hypervisor (proxmox). For this I wanted Live Migration, no need for HA in my use case.

-my setup.
2x 1u supermicro servers (1x 8c16t cpu with 64gb ram) with raid 1 of fast 1tb SSD's on each.
1x 1u (2c4t with 8gb ram) with raid 1 on spinners for quorum and VPN/remote access/DNS

I run 35-40 VM's, depending on the load and what I'm testing at the moment.

my network was 4x 1gb nics on each SM server, and 2x 1gb on the smaller one. I created 3 VLANS, lets say prod (for VM traffic), mgmt (for accessing proxmox gui (and other service VM's) and stor (this is for storage between servers)

on both of my SM servers, I created the zfs pool and exported them via NFS on the stor vlan. We'll call the shares pve1_nfs_1tb and pve2_nfs_1tb.

On my setup, I aggregated the 4x 1gb nics into 2x2 (1 group for prd and mgmt, the other for stor) to get the most from my NFS share.

So, What I ended up with, if I wanted to run all my VM's on pve1 while I updated/rebooted pve2 I storage migrated all running VM's from pve2 storage to pve1. This was working well, but felt slower than I expected and was quite a pain in the butt migrating 50-75 vm disks in my setup 1 by 1 from the gui (could have scripted this out but wanted it all from the gui) Then I live migrated the VM's from pve2 to pve1. This was FAST as it was just the VM config and ram snapshot as all the disks were already on the pve1 nfs storage.

My storage network was the bottleneck, I needed 10gb I felt to really get good nfs performance. Likely I could have done more tuning but I left it.

In the end, I purchased a single, 16c32t(2p) 128gb server to replace all three and just accept the downtime for updating proxmox. It's MUCH less overhead on my physically and it's just simpler. Saves some power in the end.

I also tried the 5.0 beta with local storage live migration but it's not via the gui yet and still has some maturing to do before I'd consider it production ready.

5.0 WILL BE THE ANSWER to all of these SMB issues, I am very excited for this release!!

 

[sshaikh]

Thanks Mitch. That's kind of why I have zero expectation of HA / live migration facilities - I know my HW isn't up to scratch. I have no problem with maintenance downtime either.

My question is whether creating a trivial cluster has any benefits - whether that's the ability to manage the two nodes from a single UI or anything else. Actually the question is more "will I have to pay for those small features with blood and sweat later?"

Sounds like no one has an answer... So I guess I'll have to just try it and see!

 

[Denny]

As you said, the advantage is managing both machines from one interface. With minimal effort you can move virtual machines from one to the other. The gotchas are when you add the node to the cluster it must have no vms on it. Tearing the cluster down later is trivial and shouldn't concern you.

 

[sshaikh]

Thanks Denny. I went ahead and bit the bullet and things seem to be going well. I did notice that all my local lvm-thin volumes from node1 now have (inactive) entries on node 2. Why is that? The two nodes will have different storage HW and configuration so this seems a bit superfluous.

OTOH there was a samba mount on node1 that has also been replicated to node 2, except it is missing the content I see via node1.

Finally, both nodes came with their default "local-lvm" storage entries - are these still separate or have they now been merged somehow?

 

[Mitch Maris]

well, when you read what happens to the second node as it's added to the cluster, it inherits your storage config from the first node plus what was installed on it. This all becomes visible to the "cluster" but you can't use pve2's local-zfs on pve1

 

[Denny]

local-lvm is indeed separate. Since I use zfs I wasn't aware of the lvm-thin containers being re-created on the other node. Setup the samba mount in fstab of node2 and it will be available on both machines.

 

[dswartz]

I think the recommended way of doing a 2-node cluster is by adding the 'two_node: 1' setting to the corosync file. Like this:

quorum {
provider: corosync_votequorum
two_node: 1
wait_for_all: 0
}

There's a gotcha there though: while that allows the cluster to be quorate with only one node present, this is NOT true when the cluster is coming up. By default setting two_node true also sets wait_for_all true (both nodes need to be present before cluster is quorate). In my case, the 2nd node is only booted when I need to upgrade the primary node, so I set wait_for_all false. You don't want to do this if the guests are using shared storage like NFS or iSCSI, due to possible corruption. In my case, the primary node hosts guest storage on an NVME zfs mirror, so there is no possibility of this.

 

[guletz]

Finally, both nodes came with their default "local-lvm" storage entries - are these still separate or have they now been merged somehow?

You can restrict that X storage resurce to exist only on node N1.

 

[sshaikh]

You can restrict that X storage resurce to exist only on node N1.

I think this is what I want - care to elaborate how? Right now the entries are a bit superfluous and possibly even confusing.

 

[guletz]

I think this is what I want - care to elaborate how? Right now the entries are a bit superfluous and possibly even confusing.

dir: diverse-store
       path /mzpool/diverse-store
       content backup,vztmpl,iso,images
       maxfiles 4
       nodes node01
       shared 1

dir: diverse-ossec
       path /mzpool/diverse-ossec
       content backup,iso,vztmpl,images
       maxfiles 4
       nodes node03
       shared 1

 

[sshaikh]

Thanks guletz, that worked a treat.

I think the recommended way of doing a 2-node cluster is by adding the 'two_node: 1' setting to the corosync file.
...
There's a gotcha there though: while that allows the cluster to be quorate with only one node present, this is NOT true when the cluster is coming up.

This has opened up a bit of a rabbit hole for me I'm afraid! On reading, it really does seem that I don't want a cluster at all really - both nodes should/would run independently if they want to. If I turn two_node on, but leave wait_for_all as true, what happens if I just start up the one? It won't work? Actually... what does quorate mean in this context?

Considering I'm not that bothered then, I think I can go for a looser cluster - as I mentioned there will be no shared storage so I might just copy your config in this regard. OTOH.... it seems to be working "as is"... so yeah, rabbit hole

 

[sshaikh]

For completion: I've stuck with the default behaviour (both nodes need to be up for anything to work), and use the following command to "force" a quorum in the rare situations I need to start something up:

pvecm expected 1

This seems to be a decent middle ground.