PBS ZFS disk speed thoughts option

AkUsP said:
Yea, well I dont see ZFS worse than any other filesystem in stealing storage for itself.
Click to expand...
But it is :)

A Hardware RAID5 with 4 drives will give you exactly 75% storage efficiency.

A ZFS RAIDZ1 with 4 drives will give you between 50% and 75%, depending on padding and pool geometry. Which in return depends on your zvol and dataset settings and what file you plan on saving onto it.

IMHO the problem with let's say 66% storage efficency is (this is what you get with the default 16k blocksize) that it is only 16% more than a mirror and because of that not worth it. Certainly not for VMs. For PBS it might be enough, if you really don't care about backup speeds.
 
I think the OP might be aware of the padding issue since they mentioned 3x disk RAIDZ1 and 6x disk RAIDZ2. Not 100% sure.

However, any parity-based RAID is going to perform poorly for the type of I/O that PBS generates (lots of small reads/writes).

A special device on SSD seems critical for PBS regardless.

Or, go for the hardware RAID controller setup due to the write-back cache functionality, and probably EXT4 for the file system.
 
J-Rod said:
A special device on SSD seems critical for PBS regardless.
Click to expand...
This!

J-Rod said:
Or, go for the hardware RAID controller setup due to the write-back cache functionality, and probably EXT4 for the file system.
Click to expand...
That may be a good choice for some situations - however I can't imagine it is a good idea to drop all ZFS goodies. ((You all know what I mean; for interested new-to-ZFS-users: https://forum.proxmox.com/threads/f...y-a-few-disks-should-i-use-zfs-at-all.160037/ ))
 
J-Rod said:
A special device on SSD seems critical for PBS regardless.
Click to expand...
I am not 100% sure on that one.
Sure, I have pretty low backup requirements (around 20 Linux VMs and 8 Windows VMs, 182 snapshots at 1.5TB) but I only see 800MBit/s and 500 IOPS peak. So I am not entirely convinced it would be much slower with HDDs instead of my SSD mirror.

Yes, it is writing random files, but these files are huge (1-4MB) not small (4k).
Another question would be, if you even need fast backups, because the VMs are not shut down during the backup.
Or how fast do you need them to be restored? And read on the other hand than way faster than writes. Maybe some of it is even from ARC (probably not, but it could be true). Or maybe your VMs don't change a lot, so the backup perf. is way more dependant on PVE read, chunk hash performance and your PBS does not matter much.

What I am trying to say, it is probably a lot more nuanced. Just like with the claim "you need enterprise PLP drives". If you paint with a broad brush, that is the best claim you can make. But if you go into nuances, "you are probably fine with a Samsung 990 Pro and a Kingston DC1000 in a mirror" becomes the often better answer.

I

Thread 'PBS testimonial (hardware requirements are a joke)'

Clickbait headline, but I think it is true :)
Maybe others can chime in with their setups and why they need beefy hardware.
Here is my PBS testimonial.

Reading the documentation I was a little bit scared. Especially the storage part seemed brutal.
Turns out it is not that bad and I misunderstood how PBS works.
I wanted to backup 3 destination, each roughly hosting 5-10 VMs.
Mixture of Windows and Linux VMs.

Since some of the PVE are from my clients, I wanted to do it right.
AMD Ryzen 5 8500G, two 120GB SSDs for boot and two 2TB nvme for datastore.
Changed to recordsize to 4M for the...
  • Like
 
Last edited:
J-Rod said:
I have filled my PBS servers with ZFS to 100% many times and did not see any performance difference, most likely due to PBS operations generating a lot of very small reads & writes, as opposed to large, continuous writes. Fragmentation of free space is definitely something that can impact writes if you are trying to write a very large file. The filesystem has to really dig deep to locate enough tiny chunks of space to fulfil the request. I've seen this over and over with Linux file servers and defrag on Linux just isn't particularly elegant, no matter what the file system.

With regard to your PBS datastore RAID setup, it seems you are working on a second server now for secondary backups, as opposed to the original issue of PBS backup performance on the primary server? Are you planning to add SSDs for an sVDEV?
Click to expand...

When I did this post. I had 1 PBS running. It was there I saw the issue. During this thread I started my secondary PBS. And now have 2 PBS running.
Both of them are R740XD with a 6xdisk RAIDZ2.

What happned was that when I added 6TB of new files to my cluster storage. The backup speed from the nodes to PBS1 dropped down to about 10% disk write. This issue got better and better and according to a Proxmox guru (that helped me paralell of this thread), he said that it was the big new files that made the ZFS to use all of my RAM. And the ARC cache got fed up. And I guess he was right. Over days I saw a performance increase. Not back to 100%. But upp from 10% to 50%. That was good enough to keep my enivroment afloat as of now.

But since I just deployed this enviroment, I must think ahead ofc and build a system that is good enogut for the future. So as of now, the performance with 32GB RAM and RAIDZ2 is "good enough". I will over the weeks reconfig both my PBS machines with 64GB RAM and 3x3 RAIDZ1 pool for future proof performance.
 
To clarify, I mean a special device on SSD seems critical for the OP's stated HDD configuration using parity and the desire for performance.
 
  • Like
Reactions: IsThisThingOn
IsThisThingOn said:
I am not 100% sure on that one.
Sure, I have pretty low backup requirements (around 20 Linux VMs and 8 Windows VMs, 182 snapshots at 1.5TB) but I only see 800MBit/s and 500 IOPS peak. So I am not entirely convinced it would be much slower with HDDs instead of my SSD mirror.

Yes, it is writing random files, but these files are huge (1-4MB) not small (4k).
Another question would be, if you even need fast backups, because the VMs are not shut down during the backup.
Or how fast do you need them to be restored? And read on the other hand than way faster than writes. Maybe some of it is even from ARC (probably not, but it could be true). Or maybe your VMs don't change a lot, so the backup perf. is way more dependant on PVE read, chunk hash performance and your PBS does not matter much.

What I am trying to say, it is probably a lot more nuanced. Just like with the claim "you need enterprise PLP drives". If you paint with a broad brush, that is the best claim you can make. But if you go into nuances, "you are probably fine with a Samsung 990 Pro and a Kingston DC1000 in a mirror" becomes the often better answer.

I

Thread 'PBS testimonial (hardware requirements are a joke)'

Clickbait headline, but I think it is true :)
Maybe others can chime in with their setups and why they need beefy hardware.
Here is my PBS testimonial.

Reading the documentation I was a little bit scared. Especially the storage part seemed brutal.
Turns out it is not that bad and I misunderstood how PBS works.
I wanted to backup 3 destination, each roughly hosting 5-10 VMs.
Mixture of Windows and Linux VMs.

Since some of the PVE are from my clients, I wanted to do it right.
AMD Ryzen 5 8500G, two 120GB SSDs for boot and two 2TB nvme for datastore.
Changed to recordsize to 4M for the...
  • Like
Click to expand...

So I'm up 'n running with my 3x3 HDD RAIDZ in a pool on both PBS1 and PBS2.
So far no worries. But! I used different command lines to create the pool and got 2 "different" pools on "zpool status" check.

PBS1 says:

State: ONLINE
config:

NAME STATE READ WRITE CKSUM
Storagevolume ONLINE 0 0 0
raidz1-0 ONLINE 0 0 0
sda ONLINE 0 0 0
sdb ONLINE 0 0 0
sdc ONLINE 0 0 0
raidz1-1 ONLINE 0 0 0
sdd ONLINE 0 0 0
sde ONLINE 0 0 0
sdf ONLINE 0 0 0
raidz1-2 ONLINE 0 0 0
sdg ONLINE 0 0 0
sdh ONLINE 0 0 0
sdi ONLINE 0 0 0

PBS2 says:

NAME STATE READ WRITE CKSUM
Storagevolume ONLINE 0 0 0
raidz1-0 ONLINE 0 0 0
wwn-0x5000c5011367e0c2 ONLINE 0 0 0
wwn-0x5000c5011350c325 ONLINE 0 0 0
wwn-0x5000c500ebe7515f ONLINE 0 0 0
raidz1-1 ONLINE 0 0 0
wwn-0x5000c500f1c3aaa5 ONLINE 0 0 0
wwn-0x5000c500f2081ce0 ONLINE 0 0 0
wwn-0x5000c500be474eb8 ONLINE 0 0 0
raidz1-2 ONLINE 0 0 0
wwn-0x5000c500c39223dc ONLINE 0 0 0
wwn-0x5000c500bf1de6d9 ONLINE 0 0 0
wwn-0x5000c500be475b60 ONLINE 0 0 0

The command I used, said SDA to SDI but! The difference was that. in the PBS2 I dident put / between the RAIDZ. So somehow it uses the uniq ID och the disks. The PBS1 said
raidz1 sda sdb sdc \
3
raidz1 sdd sde sdf \
4
raidz1 sdg sdh sdi

So it was a / beteween the RAIDZ. This time then the PBS1 made the pool, not with the uniq ID of the disks. But will the linux name SDA and so on...

My question is, after googleing it. Should I avoid using the SDA .. SDB ... SDC. and so on? Because it seams this easily brake, if I do some small change in the system? Or is this running from the false wolf in the woods?
 
You must log in or register to reply here.
Top Bottom
Back