Cluster Hardware

[Kayser-Sosa]

Hoping someone can point me in a direction.

I'm trying to find out if it would be worth my while for me to setup a Proxmox cluster of 3.

My Current Proxmox Setup
Proxmox is running on a single old desktop hardware (CPU: i7, RAM: 32GB, plenty of HDD space for the VMs), looking to upgrade in the near future. I dont have room to rackmount anything. Proxmox has 6 active VMs (Emby, Nextcloud, media editor, Apache Guacomole, NUT/Zabbix, Home Assistant). All but the media editor are linux, and there are a few more VMs but they are usually powered off and only for testing.

Reason I'm looking at a cluster would be the availability, incase something went down or for updates. It can be a pain to have to power off all VMs to update the Proxmox host.

My other question is related to cluster hardware and if it would work:
Node 1: single large server with lots of RAM, better CPU, and all hard drives for VMs and Proxmox OS
Node 2 & 3: NUC-like systems with less RAM, CPU, and HDDs only for Proxmox OS
Basically the NUC-like systems would only be there to create the cluster and hold the fort down while the main system was updated. I assume I would probably need to add external SSDs or something to the NUC-like systems just to hold the VMs when they would travel over.

Any thoughts, recommendations, or direction is greatly appreciated.

 

[Dunuin]

I don't see the point. If your main server will fail your storage is offline so the NUCs can't do much if they only got a single disk for the OS.

Wouldn't it be more useful to have a cluster with HA? So for example only 2 but identical servers with a replicated ZFS storage or better a 3/4 node CEPH setup with 10+ Gbit NICs to have a real shared storage with failover?

 

[Kayser-Sosa]

I was starting to think of that while typing it up. If the NUC-like systems had an external USB drive large enough to hold the VMs as well, such as an external 1 or 2 TB SSD?

 

[Dunuin]

You can only use HA/failover if you are using a shared storage (gluster/NFS/CEPH/ZFS) and you want that storage to be redundant (so NFS isn't a real option). And ZFS isn't a real shared storage, its just two local storages that get synchronized every several minutes so you always loose some minutes of data, because it will never be perfectly synced. If you want a really good shared storage with redundancy all your storage disks need to be spread accross all servers and all servers need to use the network to keep the storages perfectly in sync. Thats also why Ceph needs a dedicated storage backend network with atleast 10Gbit NIC.
You can't just add some USB disk and hope PVE will make it magically work.

So you might want to read abit about CEPH and ZFS replication. Both wont be cheap. With CEPH you will need expensive NICs and managed switches and everything needs to be stored atleast 3 times on 3 different servers.
And with ZFS you need to store everything atleast twice and you need a lot of RAM.
And for both you should get expensive enterprise grade hardware so stuff like cheap consumer SSDs like a Samsung Evo isn't recommended.

Or you don't care about HA and just keep your old server and let it run as a PBS server with PVE also installed to it.
In that case your VMs will only run on the new server and if that new server fails all your guests will be offline. But if you got daily backups on that second server you could atleast restore them on your second server and rum them on that until you repaired your main server. Of cause you can only work with 1 day old copies. But that wouldn't be a problem if you just want to steam some media or something.like that.

 

[Kayser-Sosa]

Thanks for the info. I'm just trying to figure out what may and may not work before pulling an triggers on hardware. I'm only weekend warrior'ing any tech stuff and have very limited time.

I only have one VM that I'm that I really need to remain up and I do have a temp backup plan if the system goes down. So my current setup does work

For the single server and 2 NUC-like system idea to work, they need to use external storage to store the VMs? I do have another physical TrueNAS box up and could possibly use that as the storage for the VMs.

Or do all 3 systems need to have the same or very similar CPU and RAM specs with external VM storage?

 

[Dunuin]

Thanks for the info. I'm just trying to figure out what may and may not work before pulling an triggers on hardware. I'm only weekend warrior'ing any tech stuff and have very limited time.

I only have one VM that I'm that I really need to remain up and I do have a temp backup plan if the system goes down. So my current setup does work

For the single server and 2 NUC-like system idea to work, they need to use external storage to store the VMs? I do have another physical TrueNAS box up and could possibly use that as the storage for the VMs.

Or do all 3 systems need to have the same or very similar CPU and RAM specs with external VM storage?

What you are talking about with an external TrueNAS as a VM storage is what I told about NFS. That will work but you don't got any redundancy. If your TrueNAS will fail there is no working NFS storage so none of your 3 servers can run any VMs. So you still got a single point of failure and your 3 server cluster wouldn't be more reliable than just a single server. Best would be if every of your servers will run some VMs and every server will store all the data of all VMs with a storage like CEPH. In that case it doesn't matter which of your server will fail because you got redundancy over the network so all data is still accessible even if a single server is offline. And the HA of PVE will automatically migrate the VMs from the offline server to one of the remaining servers that has enough ressources free. For such a setup it would make the most sense to have similar hardware for all nodes. Especially if you want to set the cpu type to "host" for better performance. I didn't ran CEPH myself but as far as I know you will need atleast 3 or 4 nodes running 24/7 so thats really not cheap if you care about the power consumption.

So for example if you want 12 VMs running all the time and all 12 VMs together need 2TB of storage it would be best to have 4 identical servers each with atleast 2TB of storage and enough RAM+CPU to run 4 VMs. So in normal operation each server would run 3 VMs from the local 2TB storage and CEPH will keep the 2TB synchronized in the background so all 4 server will always have access to the exact same 2TB of data. In case any of these 4 servers will go offline the remaining 3 servers will resume the work of the failed node so the 3 VMs that are now offline will be spread across the 3 remaining server so each of them is now running 4 instead of 3 VMs. Thats why you want all of them being capable fof running atleast 4 VMs even if they only need to run 3 VMs in normal operation. But yes, thats 8TB of storage (+ some additional TB as spare) if you want mirroring for just 2TB of usable storage. And buying 4 servers isn't cheap either, especially if you want enterprise stuff that is more reliable. The best storage engine will be crap when running on a unreliable hardware so you might want stuff like ECC RAM, UPS, SSDs with powerloss protection and so on. And NUCs won't really work because you won't have PCIe slot for 10Gbit or 40Gbit NICs, not enough SATA/SAS/M.2 ports for all the drives (and USB isn't that reliable) and so on.

I also thought about creating a cluster but that really doesn't make sense for a homlab. I would need atleast another 3000€ for new servers and switches and I would easily pay 900€ more per year for electricity. So I only got that one PVE node that is running 24/7 and send daily backups to my PBS. And in case my single PVE node dies I can install PVE to another old PC and restore some backups so atleast the most important VMs are running again while I'm repairing the main server.
My plan for the future is to get another smaller server and set that up as a second node without any HA or failover. All VMs that need to run 24/7 will stay on the main node and I will only use the second node for stuff that I only need from time to time. So the second node can be offline most of the time and I only switch it on if I want to work with it. And in case the main Node will fail I already got a running PVE node where I could restore some important VMs to.

Edit:
Here is an example on how to use a HA cluster based on ZFS replication.