Low ZFS read performance Disk->Tape

I just applied the latest patch as well, thank you @Lephisto for sharing the .deb file, I was able to install it without any issues.

My datastore is made up of two three-way HDD mirrors plus an SSD special device. Previously I was only getting 70MB/s, I just started a tape backup job with four threads and I am getting a pretty stable 300MB/s so far. I'll update if it changes. Makes an incredible difference in the time it takes to write a tape, plus no more worries about shoe-shining. This patch is really a must-have for anyone who wants to use tape with a HDD datastore.

1743077097538.png
 
It has now slowed to 200-250MB/s. I'm not sure why, but it's possible the data that it's currently reading is more fragmented than what it read to begin with. Still very, very happy with the 3x speed improvement compared to what it was on a single thread.

1743094912833.png
 
Well for me it also doesn't totally sustain 300MB/s all the time, but it's okay'ish.

@pibber what cpu and hdd/controller type do you run?
 
I'm using old repurposed hardware for this, CPU is an Intel Xeon E3-1230 v3, HDDs are 6TB WD Red (mix of CMR and SMR) connected via an LSI 9300-8i.
 
Last edited:
Okay, I have a more streamlined Hardare setup I guess.. AMD Epyc 7313 16-Core / 128GB ECC / Broadcom/LSI Tri-Band Controller / Exos 20TB Drives / 4-way mirror for special device on NVME (2x Koixia, 2x Samsung) and the LTO9HH drive in a Superloader 3.

Workers set to 16, and even if no other stuff is running, I can't max out the 300MB/s reliably. Interesstingly: If i sync a datastore to another System it runs faster by multiple times, I suspect it has somehow to do with the Order in which the chunks are to be read and subsequently written to tape, which is not required for datastoresync.

Still, without the patch I get 2-digits performance, so it's alot better now, but still far away from what the Hardware specs should be good.

@dcsapak do you see any other room for improvement? Readahead buffers?
 
Lephisto said:
Is there something significantly different?
Click to expand...
no, the latest iteration just moved the place where to configure the amount of threads, but the underlying logic stayed the same
Lephisto said:
@dcsapak do you see any other room for improvement? Readahead buffers?
Click to expand...
not sure if that would help, for that we'd have to determine where the bottleneck comes from in that scenario (iow. are your disks now saturated with this?, cpu load?)

there is still one relatively obvious (but not easy to implement) performance improvement possible, when you have many small snapshots (e.g. from incremental backups) we often switch between writing chunks and snapshots.
We could batch those chunks into larger archives and write the snapshots after that.
 
  • Like
Reactions: rybakowskijakub and Johannes S
Any idea why we all seem to top out at the same maximum of 300MB/s despite having significantly different hardware? What is the factor limiting it from going higher? At least for @Lephisto with LTO-9 it shouldn't be his drive speed, and my LTO-8 drive would also support 350MB/s theoretically.

I'm referencing this from a test a while ago:
1743180136819.png
 
Last edited:
So, as for the moment, the patch seems to work fine, even with verification jobs running in parallel it maxes out over large stretches of the time.

#worksforme
 
Lephisto said:
So, as for the moment, the patch seems to work fine, even with verification jobs running in parallel it maxes out over large stretches of the time.

#worksforme
Click to expand...
nice to hear :)

would it be possible for you to post a short answer mail on the devel list with that info to this message: https://lore.proxmox.com/pbs-devel/20250221150631.3791658-1-d.csapak@proxmox.com/#t ?
some basic info that it works and maybe a line like (if you're ok with that being in the commit):

'Tested-by: Your Name <youremail>'

This could help getting it in faster, since not all devs are reading all threads in the forum ;)

No big deal if not though
 
  • Like
Reactions: Johannes S
You must log in or register to reply here.
Top Bottom