Cluster (storage) layout advice and backup strategy

[Cromagnonaut]

Hello everyone,

first of all: I am not sure if this topic is more backup (server) related, so feel free to move this thread in the correct subforum.
Also, I am doing this as a hobby only, so I am far from professional knowledge on these topics. So please correct me if I am wrong on these topics!

...and sorry for the long post. I really hope somebody has time to read this wall of text, but after month of research, I thought I need to have another perspective on my thoughts.

Currently, I am rebuilding my homelab environment, as I want to learn clustering and increasing uptime. I am moving from one PVE instance + dedicated TrueNAS host to three hosts + one powered on demand, all build more or less on the same base hardware (see below).
The whole build is with energy efficiency in mind: I have set my power budget for the rack devices to 30 - 50 watts (idle, 24/7 uptime), because of my ecological footprint and energy prices in Germany. Also, I did backups to the NAS, but poorly managed, so I would like to improve on this part (3-2-1 rule...). Therefore, I would like to implement a Proxmox Backup Server that will be used as a daily "playground" for quick daily backups and on demand (so always online) and an additional Proxmox Server for long term backups and disaster recovery on a dedicated machine powered on demand.

Currently I have about 3-5 small Linux headless VM's and about 15 containers, but I like to expand a bit in the future.

In short, my primary goals summarized:

• Getting the most of my hardware, consolidating as much as possible, because right now, my node is happily idling around most of the time
• Cluster of at least 2 nodes, third one optional (see considerations)
• Increasing uptime of guests in case one node goes down (mission critical high availability not required)
• Energy efficency: Max. 30 -50 Watts idle in sum.
• Proper backups via Proxmox Backup Server(s) (1 always online and one powered on demand ("cold" strorage), if this is advisable (see considerations)
• I like play around but keep things simple in the long term

Restrictions:

• Additional hardware (more or less, no hard limit here, I was thinking of additonal NMVe's or small datacenter SATA SSD's for example)
• No 10G ethernet (for now, will maybe be upgraded to DAC later)
• ZFS as filesystem on PVE hosts

I did my research over the past 2 months through several forum posts on reddit and here to find the optimal configuration for my use case and ordered the first hardware. But now, I am kinda in a "valley of tears" regarding my configuration. Mainly regarding storage configuration. Originally I had 3 different configurations in mind (avaiable hardware listed at the end of the post):

Spoiler: Considerations for host configuration

In all cases: Additional Proxmox Backup Server on dedicated server for long term backups powered on demand via WOL (or IPMI?)

1. Option A: Using Host 1 & 2 as PVE nodes for VM's and containers with ZFS replication and using the third host for TrueNAS Scale, with a small VM as a qdevice and one VM for Proxmox Backup Server with a blockdevice as a datastore on a TrueNAS dataset. . Consumer SSD's used as mirror for rootfs and datacenter NVMe's as mirror for guest pool on PVE hosts. This was my original idea.
2. Option B: Using Host 1- 3 as PVE nodes. Node 1 and 2 will be used for VM's and containers with ZFS replication. SSD's used for rootfs and NVMe's as mirror for guest pool on node 1&2.
   Node 3 will be used for only used with the SATA controller passed through to a TrueNAS VM (8x SATA in one IOMMU group..). The always-on Backup Server will be a nested VM inside the TrueNAS VM with a blockdevice as a datastore on a TrueNAS dataset.
   On the third node I am undecided how to configure the storage, as I do not want to store the VM's on consumer NVMe's.
3. Option C: Nearly the same as Option B.
   Using Host 1- 3 as PVE nodes. Node 1 and 2 will be used for VM's and containers with ZFS replication. SSD's used for rootfs and NVMe's as mirror for guest pool on node 1&2.
   Node 3 will be used for only used with the SATA controller passed through to a TrueNAS VM (8x SATA in one IOMMU group..). The always-on Proxmox 0Backup Server will be a VM on the PVE host.
   On the third node I am undecided how to configure the storage, as I do not want to store the VM's on consumer NVMe's, I've no slots for dedicated disks for the always-on backup server left.
   I also have a HBA available, which would solve the problem, but it uses about 7 Watts alone , which is nearly as much as the whole server, so too much for my taste.

So, finally some specific questions:

1. Which option should I go for? I really like the idea of having a three-node cluster, but option B & C seems to be kinda convoluted. Option A is more straight-forward I think.
   With option C I have to get additional hardware (and with that, additional power draw). In all cases, the NAS instance and Proxmox Backup Server instances itself would not be included in a backup.
2. Maybe there is better solution for my use-case, that is not listed here? Should the disks be layed out differently?
3. Is the Additional Backup Server really necessary with another on virtualized or vise versa?
4. When using ZFS replication, could I possibly reduce the NVMe pool to one disk instead of a mirror? (also mentioned in this thread, without a clear result). So, the 4 disks available disks could be spread out to all hosts / used for passthrough)
5. More of a PBS question: Am I okay with using only one disk for the dedicated backup machine or should the drives be mirrored, too? Also ECC RAM on this host?
6. Is splitting the rootfs and guest pool into two disk pairs advisable in my use-case? On my current host, it is not split (on consumer SSD and fine for me...but I did not went through a complete reinstall yet.
7. Am I overthinking this in a homelab context?

Spoiler: Available hardware / original considerations

Host 1 & 2:

• Mainboard: Supermicro x11ssl-f (6x SATA, two 1Gbe NIC's)
• CPU: i3 6100
• RAM: 64 GB ECC
• 2x 1TB Datacenter NVMe (Samsung PM9A3) in Supermicro AOC-SL3-2M2 PCIe to M.2 adapter (8x bifuricated to 2/4x)
• (optional: 2x 128GB / 240GB consumer SATA SSDs)

Host 3:

• Mainboard: Supermicro x11ssm-f (8x SATA, 2x 1Gbe NIC's)
• CPU: i3 6100
• RAM: 64 GB ECC
• 2x 128GB NVMe (consumer grade, in 2x sepreate PCIe to M.2 adapters)
• 2x 4TB HDD (Seagate Ironwolf)

Planned Backup Server (Online only on demand):

• Mainboard: Fujitsu D3417-B2 (6x SATA, 1x 1Gbe NIC)
• CPU: i3 7100
• RAM: undecided (ECC?)
• 2x 128GB SSD, consumer grade)
• HDD: undecided, see questions

And a side note: I've also a powerful Proxmox machine (Ryzen 5950x, 64G RAM and 3 GPU's) available that is used for Desktops VM's as a daily driver, but is only powered on demand, so I took it out of the equation early.

I really hope somebody will make it to the end of this post and this is all not too confusing. Please give a note if additinonal infos are required.

Thank you for advice in advance

 

[floh8]

Which option should I go for?

For what u use the truenas scale enviroment? I think its more simple to go with separate baremetal PBS.

When using ZFS replication, could I possibly reduce the NVMe pool to one disk instead of a mirror?

yes, depends on your data safety.

Am I okay with using only one disk for the dedicated backup machine or should the drives be mirrored, too?

your standalone machine for backup is a single point of failure, so its better to go with the mirror.

Also ECC RAM on this host?

At the backup host the ECC function is not necessary.

Is splitting the rootfs and guest pool into two disk pairs advisable in my use-case?

SSD and NVMe differ massivly in speed, so its better to separate them. Possible is to install only the NVMes. Put both the rootfs and the VM-storage on them. Use zfs mirror while setup of PVE.

 

[Cromagnonaut]

Thank you for the reply!

For what u use the truenas scale enviroment? I think its more simple to go with separate baremetal PBS.

Mostly for filesharing of personal files / libraries for some vm's and in the future also steam libraries. So, only lite usage.

yes, depends on your data safety.

Yes, I am weighing if the mirror is really necessary in my case. But I am leaning more towards a mirror of the vm disks and single for the rootfs if i am out of pci slots (3/3/4).

your standalone machine for backup is a single point of failure, so its better to go with the mirror.

Good point, did not think this way. Is noted for the next order of disks, thank you!

SSD and NVMe differ massivly in speed, so its better to separate them. Possible is to install only the NVMes. Put both the rootfs and the VM-storage on them. Use zfs mirror while setup of PVE.

Actually, I think I will never fully utilize the speed of the ordered nvme's. I Just ordered them, because they where about the same price as sata ssd's and I could place them into a pci slot with bifurication. Splitting the rootfs and vm pool seems much more logical. But with clean backups, this advantage seems to be smaller..

I the meantime, I created a testcluster of three nodes (als of them with mirrored boot disks, two of them with mirrored nvme's and one with a hba passed through to the nas vm, the third one just as a dummy node). But I am not too happy with this solution as it is drawing about 80 watts and just seems "too much" for my use case. But ZFS replication works great and it has the potential too use a shared 10GBe link to my switch and also 10GBe between two nodes.