[Small Office/Home Office Deployment] Any Benefit to Combining Clustering and PDM-Based Management on Small-Scale Local Deployment?

SInisterPisces

SInisterPisces

Well-Known Member
Proxmox Subscriber
Sep 1, 2022
541
204
53
42
Hello,

That topic was probably not as clear as I might hope, so let me start out by describing what I've got set up in my home office. I use these machines for a mixture of work-from-home productivity, home network administration, backups, and hobbyist stuff. I'm trying to figure out if there's any benefit to clustering any of these four machines given what PDM can do to manage (and live-migrate between) single-node clusters. I want to avoid over-complicating my setup--well, over-complicating it more than I already have by somehow ending up with 4 PVE nodes and a PBS VM (it all seemed like a good idea at the time).

I should also prefix this by saying that all of these physical nodes and the PDM VM are isolated on the same subnet, and they're all physically in the same room in my office, connected to the same switches.
  • [Single Node Cluster: Pisces] Pisces1:
    • Proxmox Datacenter Manager lives in a VM here
    • Other management VMs and LXCs (e.g., Unifi's console, Pulse, etc.).
    • This single node cluster is meant to give me a centralized place to configure and run (and more importantly backup and restore) as much of my management plane as I can centralize in one place.
    • This lives in its own cluster so that if any of the other three servers go sideways, it will hopefully remain functional.
  • [Single Node Cluster: Cassiopeia] Cassiopeia1: An N300-based 2U server with a stupid amount of storage relative to its compute power, and a single 10Gbps NIC.
    • Proxmox Backup Server in a VM, with passed through 4x 2TB enterprise SSDs and 4x 480GB enterprise SSDs + 2x 1TB u.2 NVME. Backup destination for Pisces, Cassiopeia, and Andromeda nodes.
    • Logging and Metrics (local storage): Syslog server, metrics server (e.g., InfluxDB, etc.). Logging and metrics destination for all nodes in all clusters.
    • Intended to remain on 24/7, and be fairly impervious to weird things happening to it since it runs set it and forget it workloads.
  • [Not sure how to cluster these:] Andromeda1 and Andromeda2: 12th gen Intel workstations.
    • Similar but not identical hardware (e.g., Andromeda1 has a dedicated NVIDIA GPU and multiple 10 Gbps NICs; Andromeda 2 has an iGPU and fewer 10 Gbps NICs).
    • Primary production workload (I actually do stuff on these that's not management or backups).
    • I'm not interested in high availability between them.
      • Their hardare is just different enough to make this awkard.
      • This is a level of technical complexity and maintanence effort that I'm not quite ready to deal with yet as a hobbyist/home office worker.
      • Andromeda2 (the mini PC node) has such a lower power draw that my current priority is stuffing as much work onto it as possible so I can turn Andromeda1 (the larger workstation (with ~4x idle power draw)) off when I don't need it.
Leading into an actual question, I promise:
  • Originally, I was going to cluster Andromeda 1 and 2 for management on a single pane of glass, with all the hardware complications (e.g., q-device, dedicated switch or not/dedicated cluster NICs or not, etc.) that implied. I started planning all this well before PDM was even announced, and then got distracted from my hobbyist stuff for months due to…chaos, and then by the time I got back around to things PDM was at version 1.1 and looking an order of magnitude more capable than it did back in alpha.
  • I've been experimenting with PDM for the first time this weekend; and I've noticed (but not tested) that it does live migration between clusters, including single node clusters.
The question, as promised: What would the advantage be (or rather, when would there be an advantage) to clustering Andromeda 1 and Andromeda 2 together and giving PDM a three node [2x PVE + qdevice] cluster to manage versus managing each of them as a single-node cluster in PDM?
 
The main advantage I see of having two PVE servers in the same cluster is that you can easily set up and manage replication between the hosts using the PVE GUI. When you need to do maintenance to one of the servers, you can hot migrate all of the VMs to the other host in a just a few minutes, since it only needs to send the disk data that has changed since the last replication (as well as the VM's memory).

However, if you have a two node cluster and are planning to turn off one of the servers most of the time, I don't really see any point to it all...

Also, for the GUI-based ZFS replication, the two PVE hosts in the cluster will need to have the same storage naming.

Lastly, you can't join two PVE hosts together in a cluster if they already have VMs running, and it's also a bit of effort to remove a host from a cluster since you have to wipe it afterwards (there are unsupported hacks to avoid this).

So you are probably going to either need to buy three identical nodes and set up a proper cluster, or stick with what PDM offers.
 
  • Like
Reactions: UdoB
For replication clustering isn't actually needed since you could also use pve-sync for it. The story is different for high-availability though, that feature needs a cluster. Only @SInisterPisces know whether he needs it or not though ;)

One thing to consider: I wouldn't add PBS and it's host to PDM to avoid a bad actor taking over PDM also gets access to the backups. I would also consider running PBS bare metal ( can be done parallel on the metrics PVE ) so for restoring backups no working PVE host is needed.
 
  • Like
Reactions: UdoB
Just to clarify, PVE-zsync is command-line only and doesn't support hot migration. So it's better for DR, IMHO. With PVE ZFS storage replication, being able to do rolling host updates without having to shut your VMs down is a big quality of life improvement.
 
SInisterPisces said:
This lives in its own cluster so that if any of the other three servers go sideways, it will hopefully remain functional.
Click to expand...
It is by definition a Single-Point-of-Failure.

With an actual cluster you would eliminate that aspect ;-)

And a cluster makes frequent maintenance so much easier. How many Kernel-upgrades did we have this year? I've rebooted my systems 20 to nearly 30 times: I just activate "maintenance mode", wait for HA to automatically evacuate the node, run full-upgrade and reboot. No service-hickup noticed and easy as cake. With manual handling via PDM it is not really so smart, right?

Disclaimer: I do not have much experience with PDM - it is a great tool I do not need :-)
 
PDM is extremely limited as a centralized management tool. The only feature you really get that PVE doesn't offer in the GUI is the shared-nothing migration. Otherwise, it is just a lot of graphing and logs along with some basic tasks like power on/off, snapshot, console and check for updates. And to actually launch the updates you are just redirected to the native PVE or PBS GUI. So no "update all my stuff automatically" for example.

So if PDM goes down or has to be rebuilt, it's not going to impact any running systems.
 
J-Rod said:
And to actually launch the updates you are just redirected to the native PVE or PBS GUI. So no "update all my stuff automatically" for example.
Click to expand...
Yeah.

I am definitely a fan of Proxmox - both the company and the software suite. But it lacks some comfort; compare this: "Rolling Node Updates - Update Proxmox nodes one by one with automatic VM evacuation" - https://docs.pegaprox.com/automation/automation/
 
You must log in or register to reply here.
Top Bottom
Back