Design - Replication and backups

[symcbean]

I am setting up a tiny cluster - I only need more than one machine to provide fault tolerance (will be using 3 for quorum / simpler upgrade and maintenance cycles). Given that I have 3 physical hosts which is very redundant, I'm not planning on mutliple PSU's/NICs/RAID. To provide resillience against a node failing completely I want the containers/VM disks (I'm expecting this to be a mixture) replicated or backed up across all the devices. Although I want low RTO, I can accomodate quite high RPO (24 hours +).

I am seeking very high uptime in the absence of node failures. Indeed some of the services (HTTP, SMTP) will be replicated across instances pinned to specific physical nodes, however at least in the case of a NFS server (it's using separate iSCSI storage) implementing this on concurrent hosts is rather esoteric. Hence the backup/replication must be minimally disruptive.

While Proxmox provides replication and snapshots, this appears to be dependent on ZFS or Ceph for storage. From my research, both appear to have major performance overheads at such a small scale.

1) Is it worth considering using filesystem level snapshots (e.g. BTRFS/LVM), simply remounting the snapshot version/copying out the files and relying on crash recovery for bringing these images back online?

2) Should I just configure local backups in Proxmox and replicate these seperately which a scheduled rsync?

3) Which of CephFS, Ceph RBD and local ZFS is least bad for performance at this scale?

 

[apoc]

I think the setup will mainly depend on your available gear. What do you have on your list?

 

[symcbean]

Buying new hardware for this - planning to spend around 7000 GBP (9000 USD) for 3 x pizza boxes and a 10G switch. Each box with 32G / 256G NvMe and 500Gb spinning rust. Why does the hardware affect the choice?

 

[apoc]

If you have existing gear for that project it might shoot one or the other option out into the sky.

If you plan for a cluster you should have redundant networking. Otherwise a switch reboot or failure will blow your whole system away.

I am running a single host since roughly 5 years, have only one PSU but a UPS. Aside the maintenance schedules for reboot the following has given me trouble:
1. Booting from a USB stick (bad idea)
2. Networking issues (switch needs a reboot after 200 days)
3. Smashed ZFS pool due to multiple HDD failures at the same time (I was not monitoring their age, and after 7+ years multiple disks failed the same time)

What I want to say with this. Maybe it is better to avoid ha clusters "on small scale" and use 2 single hosts with replication.

And with that in mind you could spend your budget different

 

[Nitinlahoti]

Great information! thanks for explaining it