PBS ZFS disk speed thoughts option

AkUsP said:
Yea, well I dont see ZFS worse than any other filesystem in stealing storage for itself.
Click to expand...
But it is :)

A Hardware RAID5 with 4 drives will give you exactly 75% storage efficiency.

A ZFS RAIDZ1 with 4 drives will give you between 50% and 75%, depending on padding and pool geometry. Which in return depends on your zvol and dataset settings and what file you plan on saving onto it.

IMHO the problem with let's say 66% storage efficency is (this is what you get with the default 16k blocksize) that it is only 16% more than a mirror and because of that not worth it. Certainly not for VMs. For PBS it might be enough, if you really don't care about backup speeds.
 
I think the OP might be aware of the padding issue since they mentioned 3x disk RAIDZ1 and 6x disk RAIDZ2. Not 100% sure.

However, any parity-based RAID is going to perform poorly for the type of I/O that PBS generates (lots of small reads/writes).

A special device on SSD seems critical for PBS regardless.

Or, go for the hardware RAID controller setup due to the write-back cache functionality, and probably EXT4 for the file system.
 
J-Rod said:
A special device on SSD seems critical for PBS regardless.
Click to expand...
This!

J-Rod said:
Or, go for the hardware RAID controller setup due to the write-back cache functionality, and probably EXT4 for the file system.
Click to expand...
That may be a good choice for some situations - however I can't imagine it is a good idea to drop all ZFS goodies. ((You all know what I mean; for interested new-to-ZFS-users: https://forum.proxmox.com/threads/f...y-a-few-disks-should-i-use-zfs-at-all.160037/ ))
 
J-Rod said:
A special device on SSD seems critical for PBS regardless.
Click to expand...
I am not 100% sure on that one.
Sure, I have pretty low backup requirements (around 20 Linux VMs and 8 Windows VMs, 182 snapshots at 1.5TB) but I only see 800MBit/s and 500 IOPS peak. So I am not entirely convinced it would be much slower with HDDs instead of my SSD mirror.

Yes, it is writing random files, but these files are huge (1-4MB) not small (4k).
Another question would be, if you even need fast backups, because the VMs are not shut down during the backup.
Or how fast do you need them to be restored? And read on the other hand than way faster than writes. Maybe some of it is even from ARC (probably not, but it could be true). Or maybe your VMs don't change a lot, so the backup perf. is way more dependant on PVE read, chunk hash performance and your PBS does not matter much.

What I am trying to say, it is probably a lot more nuanced. Just like with the claim "you need enterprise PLP drives". If you paint with a broad brush, that is the best claim you can make. But if you go into nuances, "you are probably fine with a Samsung 990 Pro and a Kingston DC1000 in a mirror" becomes the often better answer.

I

Thread 'PBS testimonial (hardware requirements are a joke)'

Clickbait headline, but I think it is true :)
Maybe others can chime in with their setups and why they need beefy hardware.
Here is my PBS testimonial.

Reading the documentation I was a little bit scared. Especially the storage part seemed brutal.
Turns out it is not that bad and I misunderstood how PBS works.
I wanted to backup 3 destination, each roughly hosting 5-10 VMs.
Mixture of Windows and Linux VMs.

Since some of the PVE are from my clients, I wanted to do it right.
AMD Ryzen 5 8500G, two 120GB SSDs for boot and two 2TB nvme for datastore.
Changed to recordsize to 4M for the...
  • Like
 
Last edited:
J-Rod said:
I have filled my PBS servers with ZFS to 100% many times and did not see any performance difference, most likely due to PBS operations generating a lot of very small reads & writes, as opposed to large, continuous writes. Fragmentation of free space is definitely something that can impact writes if you are trying to write a very large file. The filesystem has to really dig deep to locate enough tiny chunks of space to fulfil the request. I've seen this over and over with Linux file servers and defrag on Linux just isn't particularly elegant, no matter what the file system.

With regard to your PBS datastore RAID setup, it seems you are working on a second server now for secondary backups, as opposed to the original issue of PBS backup performance on the primary server? Are you planning to add SSDs for an sVDEV?
Click to expand...

When I did this post. I had 1 PBS running. It was there I saw the issue. During this thread I started my secondary PBS. And now have 2 PBS running.
Both of them are R740XD with a 6xdisk RAIDZ2.

What happned was that when I added 6TB of new files to my cluster storage. The backup speed from the nodes to PBS1 dropped down to about 10% disk write. This issue got better and better and according to a Proxmox guru (that helped me paralell of this thread), he said that it was the big new files that made the ZFS to use all of my RAM. And the ARC cache got fed up. And I guess he was right. Over days I saw a performance increase. Not back to 100%. But upp from 10% to 50%. That was good enough to keep my enivroment afloat as of now.

But since I just deployed this enviroment, I must think ahead ofc and build a system that is good enogut for the future. So as of now, the performance with 32GB RAM and RAIDZ2 is "good enough". I will over the weeks reconfig both my PBS machines with 64GB RAM and 3x3 RAIDZ1 pool for future proof performance.
 
You must log in or register to reply here.
Top Bottom
Back