3 tier storage design - Is this valid?

[Proxfan2025]

Hi,

I'm planning to set up PVE with PBS inside it, with some basic workstation hardware.

CPU: Ryzen 5
Memory: 32GB
Storage 1: SATA 256GB*2 SSD / ZFS Mirror (Proxmox system & PBS system)
Storage 2: NVME 256GB*3 ZFS Striping (VM workaround space, except for PVE system or PBS VM)
Storage 3: HDD 1TB*3 ZFS RAIDZ1 (Data storage for PBS - dedicated for backup)

- Storage 1 is solely used for Proxmox system boot & PBS VM (system storage)
- Since NVME disks are not easily crash, I'd set them striped for best performance. (Failover plan is stated below)
- VMs in Storage 2 will be backed up to Partition 3 daily.

By that design, when Storage 2 fails, I suppose I will be able to restore VM data from the Storage 3 diskspace. PBS is still be running on Storage 1&3, even when Storage 2 goes down and being replaced.

Is this setup valid? Pointing out any errors will be welcome.
Thanks.

 

[Dunuin]

Hi,

I'm planning to set up PVE with PBS inside it, with some basic workstation hardware.

CPU: Ryzen 5
Memory: 32GB
Storage 1: SATA 256GB*2 SSD / ZFS Mirror (Proxmox system & PBS system)
Storage 2: NVME 256GB*3 ZFS Striping (VM workaround space, except for PVE system or PBS VM)
Storage 3: HDD 1TB*3 ZFS RAIDZ1 (Data storage for PBS - dedicated for backup)

- Storage 1 is solely used for Proxmox system boot & PBS VM (system storage)
- Since NVME disks are not easily crash, I'd set them striped for best performance. (Failover plan is stated below)
- VMs in Storage 2 will be backed up to Partition 3 daily.

By that design, when Storage 2 fails, I suppose I will be able to restore VM data from the Storage 3 diskspace. PBS is still be running on Storage 1&3, even when Storage 2 goes down and being replaced.

Is this setup valid? Pointing out any errors will be welcome.
Thanks.

PBS datastore shouldn`t be on HDDs. Or if you really need to use HDDs you should atleast setup the HDDs as striped mirror and add SSDs to them as mirrored special devices to store the metadata on fast SSDs for better GC performance.

Your failover plan got a problem. Your storage 2 doesn't got any parity data so it can't repair corrupted data. You won't get bit rot protection that way. So you would need to run scrubs very often and monitor your pools well. Lets say you create daily backups but only run scrub once per month. You think everything is fine but then the monthly scrub runs and will tell you that you got checksum errors meaning you got some corrupted files and ZFS can tell you which files are damaged. Normally ZFS now would repair that corrupted data but it can't do that because of missing parity data. You don't know when exactly your files got corrupted, you just know that everyting was fine when the last scrub ran 1 month ago. So all your daily backups of the past 4 weeks are basically useless because they might contain the already damaged data. If you don't keep your backups for atleast over a month you are screwed because you maybe don't got a single backup with healthy data. But even if you keep long term backups, that is really annoying, as you need to replace your latest but corrupted files with version of those files from a backup that is atleast 1 month old. So you loose all the changes done in the past 4 weeks. So if you don't want to loose a month of work/data you really should use some kind of storage layout with ZFS/ceph/btrfs and parity like a striped mirror, mirror or a raidz1/2/3. But Raidz1/2/3 isn't great as a VM storage because of bad IOPS performance and high blocksize and a striped mirror isn't great if you got 3 disks, as you can only use a multiple of 2 disks (atleast when not using 3-way mirrors).

Would make more sense to just buy two 1TB NVMes and use them as a mirror as VM storage in case you care about your data.

 

[Proxfan2025]

Dunuin,

Thanks, I noticed that focusing on performance is dangerous. I will avoid striping any Proxmox system as long as it is used for business purposes.
I understand that IOPS is not so good for RAIDZ. I wasn't too concerned about block size before. So, mirroring disk is the safest option. I will consider it. Thank you.