[Small Office/Home Office Deployment] Any Benefit to Combining Clustering and PDM-Based Management on Small-Scale Local Deployment?

[SInisterPisces]

Hello,

That topic was probably not as clear as I might hope, so let me start out by describing what I've got set up in my home office. I use these machines for a mixture of work-from-home productivity, home network administration, backups, and hobbyist stuff. I'm trying to figure out if there's any benefit to clustering any of these four machines given what PDM can do to manage (and live-migrate between) single-node clusters. I want to avoid over-complicating my setup--well, over-complicating it more than I already have by somehow ending up with 4 PVE nodes and a PBS VM (it all seemed like a good idea at the time).

I should also prefix this by saying that all of these physical nodes and the PDM VM are isolated on the same subnet, and they're all physically in the same room in my office, connected to the same switches.

• [Single Node Cluster: Pisces] Pisces1:
  • Proxmox Datacenter Manager lives in a VM here
  • Other management VMs and LXCs (e.g., Unifi's console, Pulse, etc.).
  • This single node cluster is meant to give me a centralized place to configure and run (and more importantly backup and restore) as much of my management plane as I can centralize in one place.
  • This lives in its own cluster so that if any of the other three servers go sideways, it will hopefully remain functional.
• [Single Node Cluster: Cassiopeia] Cassiopeia1: An N300-based 2U server with a stupid amount of storage relative to its compute power, and a single 10Gbps NIC.
  • Proxmox Backup Server in a VM, with passed through 4x 2TB enterprise SSDs and 4x 480GB enterprise SSDs + 2x 1TB u.2 NVME. Backup destination for Pisces, Cassiopeia, and Andromeda nodes.
  • Logging and Metrics (local storage): Syslog server, metrics server (e.g., InfluxDB, etc.). Logging and metrics destination for all nodes in all clusters.
  • Intended to remain on 24/7, and be fairly impervious to weird things happening to it since it runs set it and forget it workloads.
• [Not sure how to cluster these:] Andromeda1 and Andromeda2: 12th gen Intel workstations.
  • Similar but not identical hardware (e.g., Andromeda1 has a dedicated NVIDIA GPU and multiple 10 Gbps NICs; Andromeda 2 has an iGPU and fewer 10 Gbps NICs).
  • Primary production workload (I actually do stuff on these that's not management or backups).
  • I'm not interested in high availability between them.
    • Their hardare is just different enough to make this awkard.
    • This is a level of technical complexity and maintanence effort that I'm not quite ready to deal with yet as a hobbyist/home office worker.
    • Andromeda2 (the mini PC node) has such a lower power draw that my current priority is stuffing as much work onto it as possible so I can turn Andromeda1 (the larger workstation (with ~4x idle power draw)) off when I don't need it.

Leading into an actual question, I promise:

• Originally, I was going to cluster Andromeda 1 and 2 for management on a single pane of glass, with all the hardware complications (e.g., q-device, dedicated switch or not/dedicated cluster NICs or not, etc.) that implied. I started planning all this well before PDM was even announced, and then got distracted from my hobbyist stuff for months due to…chaos, and then by the time I got back around to things PDM was at version 1.1 and looking an order of magnitude more capable than it did back in alpha.
• I've been experimenting with PDM for the first time this weekend; and I've noticed (but not tested) that it does live migration between clusters, including single node clusters.

The question, as promised: What would the advantage be (or rather, when would there be an advantage) to clustering Andromeda 1 and Andromeda 2 together and giving PDM a three node [2x PVE + qdevice] cluster to manage versus managing each of them as a single-node cluster in PDM?

 

[J-Rod]

The main advantage I see of having two PVE servers in the same cluster is that you can easily set up and manage replication between the hosts using the PVE GUI. When you need to do maintenance to one of the servers, you can hot migrate all of the VMs to the other host in a just a few minutes, since it only needs to send the disk data that has changed since the last replication (as well as the VM's memory).

However, if you have a two node cluster and are planning to turn off one of the servers most of the time, I don't really see any point to it all...

Also, for the GUI-based ZFS replication, the two PVE hosts in the cluster will need to have the same storage naming.

Lastly, you can't join two PVE hosts together in a cluster if they already have VMs running, and it's also a bit of effort to remove a host from a cluster since you have to wipe it afterwards (there are unsupported hacks to avoid this).

So you are probably going to either need to buy three identical nodes and set up a proper cluster, or stick with what PDM offers.

 

[Johannes S]

For replication clustering isn't actually needed since you could also use pve-sync for it. The story is different for high-availability though, that feature needs a cluster. Only @SInisterPisces know whether he needs it or not though

One thing to consider: I wouldn't add PBS and it's host to PDM to avoid a bad actor taking over PDM also gets access to the backups. I would also consider running PBS bare metal ( can be done parallel on the metrics PVE ) so for restoring backups no working PVE host is needed.

 

[J-Rod]

Just to clarify, PVE-zsync is command-line only and doesn't support hot migration. So it's better for DR, IMHO. With PVE ZFS storage replication, being able to do rolling host updates without having to shut your VMs down is a big quality of life improvement.

 

[UdoB]

This lives in its own cluster so that if any of the other three servers go sideways, it will hopefully remain functional.

It is by definition a Single-Point-of-Failure.

With an actual cluster you would eliminate that aspect ;-)

And a cluster makes frequent maintenance so much easier. How many Kernel-upgrades did we have this year? I've rebooted my systems 20 to nearly 30 times: I just activate "maintenance mode", wait for HA to automatically evacuate the node, run full-upgrade and reboot. No service-hickup noticed and easy as cake. With manual handling via PDM it is not really so smart, right?

Disclaimer: I do not have much experience with PDM - it is a great tool I do not need

 

[J-Rod]

PDM is extremely limited as a centralized management tool. The only feature you really get that PVE doesn't offer in the GUI is the shared-nothing migration. Otherwise, it is just a lot of graphing and logs along with some basic tasks like power on/off, snapshot, console and check for updates. And to actually launch the updates you are just redirected to the native PVE or PBS GUI. So no "update all my stuff automatically" for example.

So if PDM goes down or has to be rebuilt, it's not going to impact any running systems.

 

[UdoB]

And to actually launch the updates you are just redirected to the native PVE or PBS GUI. So no "update all my stuff automatically" for example.

Yeah.

I am definitely a fan of Proxmox - both the company and the software suite. But it lacks some comfort; compare this: "Rolling Node Updates - Update Proxmox nodes one by one with automatic VM evacuation" - https://docs.pegaprox.com/automation/automation/

 

[SInisterPisces]

Interesting discussion. I'm confident in my approach now. For my needs, PDM should be fine by itself without setting up actual clusters. If I had identical machines and wanted to use clustering to sync up their network configuration with SDN or something, that might be different, but I'm not there yet. Thanks. Always nice to be able to push forward with confidence I won't regret what I'm doing five minutes after I'm done.

I should clarify that, knowing me, I'll likely end up leaving Andromeda1 on 24/7 eventually. For now, I'm trying to get overall power usage down, so I need to get about ~150w of crap out of my office before I leave Andromeda1 on all the time. Luckily, I'm about to replace a 12 year old 4K LCD that draws 150w with a 2026 model that should draw ~50w or less. (I've got two of the 150w monitors. Hopefully I'll be able to replace the other one soonish. Probably not.)

I don't care about high availability at this time, so I don't have to worry about that. It sounds cool and I'd love to set it up, but I absolutely don't need it, and I'd rather focus on setting up useful workloads than keep spending money to optimize my PVE infrastructure.

(Put it another way: I'm at the stage where I've built up a muscle car to the point it works well and will be fun to drive, so I need to actually get it out on the road. I can upgrade it later if I find it's not got enough oomph.)

Honestly, if I ever find myself with workloads that need HA, I'll likely build an identical three node mini-PC cluster with dedicated switch for that. But for my daily productivity use, ZFS-based replication will be sufficient--especially since I don't keep my production data inside the VMs. I won't care if my Windows 11 VM travels back in time 1 hour because all my data lives elsewhere.

Andromeda1 and Andromeda2 both have the same storage capacity for VMs and LXCs, so I'd already planned to use the ZFS replication feature to keep copies of their VMs and LXCs between them, just as another layer of redundancy in addition to PBS.

It is by definition a Single-Point-of-Failure.

Yes. That's not ideal. As to PBS, I'll be running a second PBS as a VM on TrueNAS that will Pull synchronize to the PBS on Cassiopeia1, and Cassiopeia1 will also Pull sync to my Tuxis cloud PBS.

For logging/metrics redundancy, I'll either need to set up a separate Cassiopeia2 PVE node (ideal, but not a fun prospect in The Decade of the RAMpocalypse), or cave and run production VM workloads on TrueNAS (TrueNAS is a great platform for this, and I'll be running a virtualized PBS there, but I really wanted to keep all my VM and LXC stuff on Proxmox as much as possible just to keep everything running on one platform. I'm not great at PVE, but I'm much better with it and comfortable that I know what I'm doing than I am with TrueNAS).

One thing to consider: I wouldn't add PBS and it's host to PDM to avoid a bad actor taking over PDM also gets access to the backups.

I'll be wary of this. Thanks for calling it out.

That said, I've got strong passwords with 2FA on everything. If someone breaches my PDM (or anything else), it means they have access to my entire password manager, including all my credit card numbers, bank account passwords, everything passwords…

Or, it means that I'm sitting in front of a keyboard with a gun to my head and being made to log into my PDM.

Either way, someone breaching my PDM would be a symptom of a much larger disaster.