PBS ZFS disk speed thoughts option

A

AkUsP

New Member
Proxmox Subscriber
Aug 26, 2026
15
0
1
Hello. I'm new here. After 20 years VMware addiciton I'm now sober and saved by the Holy Ch... Proxmox.

After migrating about 20 VM's from Vware my PBS ran backups @ about 200-300mb/s.
After migrating my last 5 VM', that are file servers and have very many many small files. The PBS runns @ about 10 - 20mb/s.
This is my machine:
Server: Dell Ryzen Epic Gen1
Raidcontroller: PERC 740P in mixed HBA mode.
PBS version: Latest
PBS installed on HW RAID1 240GB SATA SSD
PBS Storage 6x 5400rpm SATA disks on RAIDZ2 via HBA bypass

I have 3 possible ways forward. Can you please help me what each scenario would give.

Scenario 1:
I expand from 6x to 10x 5400rpm disks in my RAIDZ2. According to the "internet" you should keep the amout of disk even + parity.
Meaning: 4 + 2 or 8 + 2 och 12 + 2. This accroding to the "internet" is profitable for the ZFS file system.
I have 4 disk extra so a 8 + 2 scenario is possible

Scenario 2:
I break my RAIDZ2 and go for RAID10 in HBA mode, so the ZFS does the 5 + 5 RAID

Scenario 3:
I reboot the PERC 740P back in RAID mode and go for RAID10 in HW mode. Not using the ZFS more than a filesystem.


Side Question. When working with RAID in hardware mode, there is something called span, that makes raid5 go 50 and raid6 go 60.
So using 24 slow disk in raid 5, you can go for 4 spans and get 6 disk in each spam, getting a 6 disk raid 5 with the speed of 4x.
I cannot find anything on this in Proxmox VE or PBS. Cant the ZFS benefit from this trick also?

//Many thanks in advance, and I hope to be aroud with you for many years now that I found home...!
 
Personally, I would suggest scenario 3, assuming your RAID controller has a battery-backed cache and you set the controller to write-back (not the disks). That will probably be the fastest setup of the three scenarios listed, although perhaps ZFS in RAID10 with an SSD special device might be faster for some PBS operations. This would probably require some decent testing protocols to be sure.

Have you run any PBS benchmarks? For example, from the PBS server: proxmox-backup-client benchmark --repository yourdatastorenamehere
You can also run it from a PVE host: proxmox-backup-client benchmark --server yourpbsserverhere --repository yourdatastorenamehere
 
J-Rod said:
Personally, I would suggest scenario 3, assuming your RAID controller has a battery-backed cache and you set the controller to write-back (not the disks). That will probably be the fastest setup of the three scenarios listed, although perhaps ZFS in RAID10 with an SSD special device might be faster for some PBS operations. This would probably require some decent testing protocols to be sure.

Have you run any PBS benchmarks? For example, from the PBS server: proxmox-backup-client benchmark --repository yourdatastorenamehere
You can also run it from a PVE host: proxmox-backup-client benchmark --server yourpbsserverhere --repository yourdatastorenamehere
Click to expand...

Yes. SSD would be faster. But I currently have the disks no additional costs.
No I havernt run any benchmark.
Is there any test avalible where someone has pitted RAID10 ZFS software in HBA mode vis a HW RAID10 ?
 
IsThisThingOn said:
First of all, what are we talking about?
What do you mean by "migrating"?
Is it the backup from PVE to PBS speed you are unhappy with?
Click to expand...

What I meen is that:
I have a Proxmox enviroment where i migrated my 20vms som VMware. They've been running for about 6 weeks.
The PBS Server speed when running backup on them has been while watching it do it, about 200 mbs/s +
These machines where only "empty" servers, running different programs, but no "file" servers for the employees.
We are talking about network keays, firewalls, printservers, script servers. Very little data changes on them day after day.
For the PBS, it was a walk in the park.

Still having my VMware enviroment up and running the last fileservers.

Then after 6 - 7 weeks I choose a weekend where the employes where @ home and moved the last 5 VM's. All of them file servers, ranging from 600gb to 1600gb in size. Total about 6000gb. When the PBS takes backup on them every day the speed starts first second to about 200-350mb/s. Then you se how it quick decline to about 20mb/s, somethimes bumping up to about 80mb/s @ best. But avering abouty 25-30mb/s during the entire backup procedure.
Right now im doing backup 1/once day. I would in the future, be able to do maby every 2 houres or more often during work day. As it looks now, it wont manage.

So I'm looking to optimize the backup speed to be able to do backup maby every 60 min in the future.
 
What you call "span" is "VDEV" for ZFS and their number does matter a lot, especially when dealing with very slow drives.
5400 rpm SATA is basically as slow as you possibly could be.

6 drives in RAIDZ2 are only one VDEV and give write performance of 1 single drive. I guess what you would call "span 1".

Basically what you would call RAID10 "span3" with 6 disks can be done with ZFS as well.
Define your pool with 3 VDEVs of 2 disks each as MIRROR. You will get 300% write performance.
 
  • Like
Reactions: Johannes S and news
woma said:
What you call "span" is "VDEV" for ZFS and their number does matter a lot, especially when dealing with very slow drives.
5400 rpm SATA is basically as slow as you possibly could be.

6 drives in RAIDZ2 are only one VDEV and give write performance of 1 single drive. I guess what you would call "span 1".

Basically what you would call RAID10 "span3" with 6 disks can be done with ZFS as well.
Define your pool with 3 VDEVs of 2 disks each as MIRROR. You will get 300% write performance.
Click to expand...

As I said. I have 4 more disk. So we could play with 10 disks.
But 1 VDEV with 2 disks gives the space of 1 disk correct ? Same as normal RAID10 HW?
In this case. I have 10 disks. So It would end up a VDEV och 5x MIRROR with 500% write performance?
And I do all this is ZFS layer, with disks as HBA?

According to "the internet" the proxmox can do all these raids, but not stack the VDEV's toughter. This must be done in CLI ?
 
Last edited:
There is a long discussion here: https://forum.proxmox.com/threads/yet-another-zfs-on-hw-raid-thread-with-benchmarks.138947/
The problem is that ZFS is a sensitive topic, so to speak, so conversations seem to turn in to religious discussions with plenty of speculation.
You have the specific hardware in front of you, so the right answer is to set up the various configurations and test the speed.
But of course that does require a bit of work and perhaps some risk to your backups that maybe you don't want to deal with.

Looking at your configuration, I think you have the trifecta of performance killers: a lack of a special vdev, dual parity RAID and 5400RPM drives.
 
  • Like
Reactions: Johannes S
J-Rod said:
There is a long discussion here: https://forum.proxmox.com/threads/yet-another-zfs-on-hw-raid-thread-with-benchmarks.138947/
The problem is that ZFS is a sensitive topic, so to speak, so conversations seem to turn in to religious discussions with plenty of speculation.
You have the specific hardware in front of you, so the right answer is to set up the various configurations and test the speed.
But of course that does require a bit of work and perhaps some risk to your backups that maybe you don't want to deal with.

Looking at your configuration, I think you have the trifecta of performance killers: a lack of a special vdev, dual parity RAID and 5400RPM drives.
Click to expand...

Yea I get your point. Im a cheapskate :-). Yea dual parity with already slow drives doesent help. It was another Proxmox "specialist" who said against RAID10 to me. Opting that ZFS in software layer doesent depend on the RAID controllers short-commings. And that RAIDZ2 will be more easy on the disk wear than RAID10. That I dont realy understand why it should. "So it's always optimal to go for RAIDZ rather than HW RAID10." Was the ideá
I gave it a shot, but I realized form the begining it wont work. Since its a rotating disk with about 90-150mb/s PEAK, the RAID controllers speed will never be and Issue, the spindle movment will be. Since I'm a Vmware/SAN guy I havent done anything than RAID10 for the last 10 years, dealing with rotating disks.

The quickest easyes way for me is to go 10disk RAID10 in HW mode. The PERC 740P is 8x PCI-E 3.0, so we are talking about 6500mb/s as the limit. It will never be close to that, even with 24 rotating disks in front. But if I can do the RAID10 in ZFS software mode, I can try that.
 
Can you run the benchmark command on your PBS host?

My setup is some pretty old hardware with six disk RAID10 with 7200RPM WD4002FYYZ:
(Forgot to mention, this is with mirrored SSDs as a special device.)
Code: 
┌───────────────────────────────────┬────────────────────┐
│ Name                              │ Value              │
╞═══════════════════════════════════╪════════════════════╡
│ TLS (maximal backup upload speed) │ 410.33 MB/s (33%)  │
├───────────────────────────────────┼────────────────────┤
│ SHA256 checksum computation speed │ 340.66 MB/s (17%)  │
├───────────────────────────────────┼────────────────────┤
│ ZStd level 1 compression speed    │ 360.63 MB/s (48%)  │
├───────────────────────────────────┼────────────────────┤
│ ZStd level 1 decompression speed  │ 489.07 MB/s (41%)  │
├───────────────────────────────────┼────────────────────┤
│ Chunk verification speed          │ 205.79 MB/s (27%)  │
├───────────────────────────────────┼────────────────────┤
│ AES256 GCM encryption speed       │ 1182.49 MB/s (32%) │
└───────────────────────────────────┴────────────────────┘
 
Last edited:
  • Like
Reactions: Johannes S and IsThisThingOn
Here's a PBS server running as VM in a Synology DS1821+ with mixed-use SSDs in RAID6:
Code: 
┌───────────────────────────────────┬────────────────────┐
│ Name                              │ Value              │
╞═══════════════════════════════════╪════════════════════╡
│ TLS (maximal backup upload speed) │ 485.80 MB/s (39%)  │
├───────────────────────────────────┼────────────────────┤
│ SHA256 checksum computation speed │ 283.89 MB/s (14%)  │
├───────────────────────────────────┼────────────────────┤
│ ZStd level 1 compression speed    │ 260.83 MB/s (35%)  │
├───────────────────────────────────┼────────────────────┤
│ ZStd level 1 decompression speed  │ 406.37 MB/s (34%)  │
├───────────────────────────────────┼────────────────────┤
│ Chunk verification speed          │ 161.91 MB/s (21%)  │
├───────────────────────────────────┼────────────────────┤
│ AES256 GCM encryption speed       │ 2098.17 MB/s (58%) │
└───────────────────────────────────┴────────────────────┘
 
SHA256 speed: 478.94 MB/s
Compression speed: 431.37 MB/s
Decompress speed: 638.40 MB/s
AES256/GCM speed: 3445.89 MB/s
Verify speed: 272.01 MB/s
┌───────────────────────────────────┬────────────────────┐
│ Name │ Value │
╞═══════════════════════════════════╪════════════════════╡
│ TLS (maximal backup upload speed) │ not tested │
├───────────────────────────────────┼────────────────────┤
│ SHA256 checksum computation speed │ 478.94 MB/s (24%) │
├───────────────────────────────────┼────────────────────┤
│ ZStd level 1 compression speed │ 431.37 MB/s (57%) │
├───────────────────────────────────┼────────────────────┤
│ ZStd level 1 decompression speed │ 638.40 MB/s (53%) │
├───────────────────────────────────┼────────────────────┤
│ Chunk verification speed │ 272.01 MB/s (36%) │
├───────────────────────────────────┼────────────────────┤
│ AES256 GCM encryption speed │ 3445.89 MB/s (95%) │
└───────────────────────────────────┴────────────────────┘
 
OK, from the PVE host:
Code: 
┌───────────────────────────────────┬────────────────────┐
│ Name                              │ Value              │
╞═══════════════════════════════════╪════════════════════╡
│ TLS (maximal backup upload speed) │ not tested         │
├───────────────────────────────────┼────────────────────┤
│ SHA256 checksum computation speed │ 446.35 MB/s (22%)  │
├───────────────────────────────────┼────────────────────┤
│ ZStd level 1 compression speed    │ 377.82 MB/s (50%)  │
├───────────────────────────────────┼────────────────────┤
│ ZStd level 1 decompression speed  │ 548.12 MB/s (46%)  │
├───────────────────────────────────┼────────────────────┤
│ Chunk verification speed          │ 242.57 MB/s (32%)  │
├───────────────────────────────────┼────────────────────┤
│ AES256 GCM encryption speed       │ 3348.91 MB/s (92%) │
└───────────────────────────────────┴────────────────────┘
 
J-Rod said:
From those numbers, I suppose the result is that the disk I/O is the problem...
Click to expand...
Yea, this was acctualy my new Backup 2 server, that is not in work yet. But both machines are similar.
I'm certain its a disk I/O problem. The hosts are running Samsung 1725 disc so its not that.

But OK, I have read the cons an pros about the ZFS bypass, why not use HW RAID.
If I then stand between RAID10 HW and a 5VDEV pool with 2+2+2+2+2 mirrors.
How will the performance stand, you think I will get the I/O up enough for me?
 
J-Rod said:
From those numbers, I suppose the result is that the disk I/O is the problem...
Click to expand...

Exactly why you optimize those ZFS VDEV numbers when using spinning rust. Hard drives lack IOPS and improvement directly reduces waiting time for tasks to finish.

SSDs have so much IOPS that it mostly does not matter. Maybe that is why younger guys tell you "that is religion" and older guys that it is very important.
 
woma said:
Exactly why you optimize those ZFS VDEV numbers when using spinning rust. Hard drives lack IOPS and improvement directly reduces waiting time for tasks to finish.

SSDs have so much IOPS that it mostly does not matter. Maybe that is why younger guys tell you "that is religion" and older guys that it is very important.
Click to expand...

Spinning rust or not. The theory about using even numbers on disk for data (exept parity 2 + 2 or 4 + 2 or 6 + 2 or 8 + 2 and so on....), was this an old rumor that ZFS incarnations has worked out or not?

Google AI says this is not an issue anymore, I qoute below:
"
No, even or odd numbers of drives do not matter significantly for RAIDZ performance or functionality in modern ZFS.

Why Even/Odd Doesn't Matter
    • No Power-of-2 Rule: Older hardware RAID controllers required an even number of drives (like powers of 2) to avoid padding waste.
    • Variable Stripe Width: ZFS uses dynamic striping and built-in compression, meaning stripe sizes shift based on data size.
    • Minimal Padding Impact: While specific drive counts relative to parity (data + parity) can minorly impact space efficiency for uncompressed small blocks, modern compression entirely nullifies the old "power of two" disk rule. [1]

Better Guidelines for RAIDZ Sizing
Instead of worrying about even numbers, focus on recommended vdev (virtual device) widths:


    • RAIDZ1: 3 to 9 drives is standard (often recommended around 3 to 5 for efficiency/rebuild times).
    • RAIDZ2: 6 to 10 drives is the sweet spot.
    • RAIDZ3: 7 to 15 drives for large setups.
    • Performance Rule: Performance scales better by adding more vdevs to a pool rather than stacking too many disks into a single RAIDZ vdev.
 
So confusing now. The benchmark is from a different server instead of the one we've been discussing?

Anyway, in my opinion ZFS is going to be particularly slow on hard drives no matter what you do since latencies are a lot higher and there is no write caching. The PERC H740P has 8GB of RAM and will acknowledge writes once they are written to the cache, so is going to be faster.

You could also experiment with this: https://forum.proxmox.com/threads/pbs-client-change-detection-mode.150538/
 
AkUsP said:
Then after 6 - 7 weeks I choose a weekend where the employes where @ home and moved the last 5 VM's. All of them file servers, ranging from 600gb to 1600gb in size. Total about 6000gb.
Click to expand...
The Proxmox BS is not for your file server and data!
It Backup your LXC and VM.
 
J-Rod said:
So confusing now. The benchmark is from a different server instead of the one we've been discussing?

Anyway, in my opinion ZFS is going to be particularly slow on hard drives no matter what you do since latencies are a lot higher and there is no write caching. The PERC H740P has 8GB of RAM and will acknowledge writes once they are written to the cache, so is going to be faster.

You could also experiment with this: https://forum.proxmox.com/threads/pbs-client-change-detection-mode.150538/
Click to expand...
The speed is the same. My PBS1 och PBS2 are sister machines.
 
You must log in or register to reply here.
Top Bottom
Back