To ZFS or not

[dcastro]

Recently my NAS (QNAP TS251) died, so I've been the last two weeks deciding how to replace it. I don't wan't another QNAP or similar (Synology). And also I am taking the opportunity to change other things at home, as for example extract multiple services from home assistant, run home assistant under Proxmox, between other things.

That last thing is what I have clear: I want everything under Proxmox.

One disclaimer: this is not my passion, I am not doing this changes to learn about Proxmox, ZFS, etc. I know I will learn, but that is not the main purpose. I need a series of services in my day to day life and I would like to have them working in a simple and safe way with the less complications. And if some issue arrives, that it will, I wouldn't like to have a very sophisticated thing that is going to force me to investigate a lot of things and complicate the solution.

My plan is to use minipcs, basically because I want low consumptions.

Looking lot of brands of minipcs I've discovered that there is no one with ECC RAM support. And also is difficult to find one with more than 2 nvme SSD (and lots of the ones that accepts 2 nvme SSD, have one limited to 1TB).

I was thinking in having initially 2 minipcs: 1 for Proxmox VE and 1 for Proxmox BS

And here come some questions:

For the PVE I've found one minipc which accepts 3 nvme SSD.

A) Should it be interesting to install the PVE OS in one smaller disk and have the other two with ZFS in mirror?
B) The minipcs which have only 2 disks would also be interesting? Should be better to have the PVE OS in one disk and ZFS in the other disk (without mirroring) or using EXT4 in the other disk? I haven't understood yet the benefits os ZFS in a single disk with regards to EXT4
C) I've read multiple times that ECC RAM is not a must for ZFS. But the last thing that I've read about ZFS is that the disks suffer and consumer disks are not recommend, is this true or is an exaggeration?
D) Let's say I have ZFS mirrored in 2 1TB disks. If I would need to increase the capacity, could I first replace one disk with a 2TB and then when everything is ok, replace the other one? I would say that my current data is less than 1TB, so I would like to start with 1TB disks to save some bucks. I don't event know if I am going to need more than 1TB

For the PBS (although this is not the forum)

D) Should I use the same minipc in order to have ZFS mirrored and the OS in a different disk?
E) Is it worth to have ZFS both in PVE and PBS?
F) Same question as B but for the PBS, should it be interesting to have just one disk with ZFS for storing the backups? I am asking also this because for the PBS I was planning to use Intel N100 to keep consumption as low as possible, and is difficult to find a minipc with that processor with more than 2 disks.
G) My intention is to use RClone to send the backups to the cloud. I guess that I should use some kind of mechanism to ensure that the backup in the cloud is correct right? (I don't know yet if RClone does this kind of checks, but if it doesn't, I should find the way)
H) I've seen some people having TrueNas as a VM in PBS and others having PBS running on TrueNas, what is the purpose of these kind of things? Why not just having PBS? I want to keep things simple
I) I've found the AOOSTAR R1 which uses Intel N100 and has 1 nvme disk and 2x 2'5/3'5" SATA. Could it be a good option? I already have 2 3'5" mechanical disks (Seagate Ironwolf) from my previous QNAP (what has died is the NAS, not the disks) and I am not sure about reusing them

Lot of questions and all opinionated. I am sure that once I've posted it, more questions will come to my mind

Thank you!

 

[Dunuin]

One disclaimer: this is not my passion, I am not doing this changes to learn about Proxmox, ZFS, etc. I know I will learn, but that is not the main purpose. I need a series of services in my day to day life and I would like to have them working in a simple and safe way with the less complications. And if some issue arrives, that it will, I wouldn't like to have a very sophisticated thing that is going to force me to investigate a lot of things and complicate the solution.

ZFS/PVE requires you to fix stuff via CLI where a typo will wipe all data on that pool. Would recommend to use something else like TrueNAS if you don't want to spend some hours/days to learn how to properly administrate it.

Looking lot of brands of minipcs I've discovered that there is no one with ECC RAM support.

There are some old Supermicro MiniPCs that got ECC but you probably don' want to spend that much money on old and slow hardware. ECC isn't required but without it you can never trust the scrub results. So data might be corrupted while ZFS is telling you that data integrity is fine...

A) Should it be interesting to install the PVE OS in one smaller disk and have the other two with ZFS in mirror?

You don't need a dedicated system disk. Raid1 for everything would be fine. But nice to have in case you got enough bays/ports.

B) The minipcs which have only 2 disks would also be interesting? Should be better to have the PVE OS in one disk and ZFS in the other disk (without mirroring) or using EXT4 in the other disk? I haven't understood yet the benefits os ZFS in a single disk with regards to EXT4

Transparent block level compression, replication, checksumming, snapshots, deduplication, ... lots of stuff ext4 can't do. If you simply want a software raid, ZFS is overkill as it is so much more.

But the last thing that I've read about ZFS is that the disks suffer and consumer disks are not recommend, is this true or is an exaggeration?

Depends on your workload. If you write a lot or you do the wrong writes (like random sync writes) you will kill them pretty fast. Could last for years or be killed in a few weeks. Especially make sure not to buy QLC NAND SSDs or SMR HDDs in case you want to stick with consumer SSDs.

D) Let's say I have ZFS mirrored in 2 1TB disks. If I would need to increase the capacity, could I first replace one disk with a 2TB and then when everything is ok, replace the other one?

Yes, that is one way to do it.

I would say that my current data is less than 1TB, so I would like to start with 1TB disks to save some bucks. I don't event know if I am going to need more than 1TB

You shouldn`t fill a ZFS pool too much. Usually you try to keep it below 80%. And in case you want to make use of snapshotting you need additional space. So that's more like 800GB usable.

E) Is it worth to have ZFS both in PVE and PBS?

Not that important with PBS as PBS is already doing stuff like encyption, compression, deduplication, checksumming and so on, on the software and not the filesystem/storage level.

F) Same question as B but for the PBS, should it be interesting to have just one disk with ZFS for storing the backups? I am asking also this because for the PBS I was planning to use Intel N100 to keep consumption as low as possible, and is difficult to find a minipc with that processor with more than 2 disks.

Better to buy another N100 MiniPC to have two PBS so you got anoher one you could host offsite and with sync jobs for ransomware protection and offsite backups.

G) My intention is to use RClone to send the backups to the cloud. I guess that I should use some kind of mechanism to ensure that the backup in the cloud is correct right? (I don't know yet if RClone does this kind of checks, but if it doesn't, I should find the way)

Keep in mind that you can't restore any VMs/LXCs without a working PBS. Thats not like with VZDump where backup archives exist that could be restored directly by PVE.

H) I've seen some people having TrueNas as a VM in PBS and others having PBS running on TrueNas, what is the purpose of these kind of things? Why not just having PBS? I want to keep things simple

PBS can't run VMs unless you install a PVE on top of it. And running PBS as a VM on TrueNAS makes more sense than running PBS on the same computer that is running your single PVE.

I) I've found the AOOSTAR R1 which uses Intel N100 and has 1 nvme disk and 2x 2'5/3'5" SATA. Could it be a good option? I already have 2 3'5" mechanical disks (Seagate Ironwolf) from my previous QNAP (what has died is the NAS, not the disks) and I am not sure about reusing them

PBS needs IOPS performance. Using HDDs is a pain once you are planning to store TBs instead of GBs of backups. Usually when using HDDs you also add some SSDs as special devices to store the metadata on the SSDs.

 

[dcastro]

Thank you so much for your response.

I had also found that supermicro minipcs, but I discarded them.

So you are proposing using TrueNas just to avoid cli for ZFS tasks.

With all that answers I am still without a clear idea of what to do.

Something that I didn't mention is that I do not need 24x7 solution. If something breaks it would be good to have the system down until broken parts are replaced. What I don't want is to loose data (like my whole life pictures).

So with this in mind, I am thinking to have one single disk in PVE and PBS with EXT4, and trust in the periodic backups.

Do you think it is an acceptable solution or is there something that I am missing?

 

[Dunuin]

So you are proposing using TrueNas just to avoid cli for ZFS tasks.

Yes, if you want something that is simply working, offering a GUI for all tasks and will prevent you from doing something stupid, then it might be better to choose an appliance (like TrueNAS/UnRaid) and not a Linux server (like PVE) that got a way steeper learning curve.

Something that I didn't mention is that I do not need 24x7 solution. If something breaks it would be good to have the system down until broken parts are replaced. What I don't want is to loose data (like my whole life pictures).

So with this in mind, I am thinking to have one single disk in PVE and PBS with EXT4, and trust in the periodic backups.

Do you think it is an acceptable solution or is there something that I am missing?

Then you probably want multiple backups on different media and also some of them offsite. Like PBS for automated recent backups and some VZDump backups written on USB HDDs that you store at some friends/family members home. I already lost 10 years of family photos because my backup disk died as well. So don't bet all your money on the same horse and don't cheap out on backups. Otherwise you might learn it the hard way like me...

 

[dcastro]

I will take into account your recommendations. I was planning to have a dedicated minipcs for PBS and sending a copy of these backups to a cloud provider (cyphered).

I will consider an additional PBS (like suggested previously) and also having additional copies in USB SSDs

Thanks again!

 

[esi_y]

There are some old Supermicro MiniPCs that got ECC but you probably don' want to spend that much money on old and slow hardware. ECC isn't required but without it you can never trust the scrub results. So data might be corrupted while ZFS is telling you that data integrity is fine...

I don't have much to add to this discussion that would add value, just ... why not get e.g. an HP Microserver for the NAS specifically?