Low ZFS read performance Disk->Tape

I just applied the latest patch as well, thank you @Lephisto for sharing the .deb file, I was able to install it without any issues.

My datastore is made up of two three-way HDD mirrors plus an SSD special device. Previously I was only getting 70MB/s, I just started a tape backup job with four threads and I am getting a pretty stable 300MB/s so far. I'll update if it changes. Makes an incredible difference in the time it takes to write a tape, plus no more worries about shoe-shining. This patch is really a must-have for anyone who wants to use tape with a HDD datastore.

1743077097538.png
 
It has now slowed to 200-250MB/s. I'm not sure why, but it's possible the data that it's currently reading is more fragmented than what it read to begin with. Still very, very happy with the 3x speed improvement compared to what it was on a single thread.

1743094912833.png
 
Well for me it also doesn't totally sustain 300MB/s all the time, but it's okay'ish.

@pibber what cpu and hdd/controller type do you run?
 
I'm using old repurposed hardware for this, CPU is an Intel Xeon E3-1230 v3, HDDs are 6TB WD Red (mix of CMR and SMR) connected via an LSI 9300-8i.
 
Last edited:
Okay, I have a more streamlined Hardare setup I guess.. AMD Epyc 7313 16-Core / 128GB ECC / Broadcom/LSI Tri-Band Controller / Exos 20TB Drives / 4-way mirror for special device on NVME (2x Koixia, 2x Samsung) and the LTO9HH drive in a Superloader 3.

Workers set to 16, and even if no other stuff is running, I can't max out the 300MB/s reliably. Interesstingly: If i sync a datastore to another System it runs faster by multiple times, I suspect it has somehow to do with the Order in which the chunks are to be read and subsequently written to tape, which is not required for datastoresync.

Still, without the patch I get 2-digits performance, so it's alot better now, but still far away from what the Hardware specs should be good.

@dcsapak do you see any other room for improvement? Readahead buffers?
 
Lephisto said:
Is there something significantly different?
Click to expand...
no, the latest iteration just moved the place where to configure the amount of threads, but the underlying logic stayed the same
Lephisto said:
@dcsapak do you see any other room for improvement? Readahead buffers?
Click to expand...
not sure if that would help, for that we'd have to determine where the bottleneck comes from in that scenario (iow. are your disks now saturated with this?, cpu load?)

there is still one relatively obvious (but not easy to implement) performance improvement possible, when you have many small snapshots (e.g. from incremental backups) we often switch between writing chunks and snapshots.
We could batch those chunks into larger archives and write the snapshots after that.
 
  • Like
Reactions: rybakowskijakub and Johannes S
Any idea why we all seem to top out at the same maximum of 300MB/s despite having significantly different hardware? What is the factor limiting it from going higher? At least for @Lephisto with LTO-9 it shouldn't be his drive speed, and my LTO-8 drive would also support 350MB/s theoretically.

I'm referencing this from a test a while ago:
1743180136819.png
 
Last edited:
So, as for the moment, the patch seems to work fine, even with verification jobs running in parallel it maxes out over large stretches of the time.

#worksforme
 
Lephisto said:
So, as for the moment, the patch seems to work fine, even with verification jobs running in parallel it maxes out over large stretches of the time.

#worksforme
Click to expand...
nice to hear :)

would it be possible for you to post a short answer mail on the devel list with that info to this message: https://lore.proxmox.com/pbs-devel/20250221150631.3791658-1-d.csapak@proxmox.com/#t ?
some basic info that it works and maybe a line like (if you're ok with that being in the commit):

'Tested-by: Your Name <youremail>'

This could help getting it in faster, since not all devs are reading all threads in the forum ;)

No big deal if not though
 
  • Like
Reactions: Johannes S
Hello, I see that there are already threads available in 3.4.0. I use Proxmox at home on weak hardware r730xd with 5x 2.5 segate 5TB HDD CMR in raidz2 for data. The performance is terrible but for my home use it is acceptable. This is probably the worst setup proxmox backup server but "Write once read many". So it's ok.

Previously on LTO-6 with TL4000 I achieved unstable 15-50MB/s which was very problematic.

Now I set 4 threads and I get 70-90MB/s (Maximum speed for LTO-6 is 160MB/s) Which is already a great result for me. Tomorrow I will see with 8 threads. If I achieve more than 100MB/s I will be happy. I will test and share the results in a while.

Now I have over 7TB of data that I save to tapes. I have no problem with rewinding and the speed is acceptable. Thanks for the work and the great ecosystem that is proxmox!


//EDIT

Proxmox is going crazy and peaking at over 160MB/s now!

2025-04-12T23:03:28+02:00: wrote 1194 chunks (4296.54 MB at 96.55 MB/s)
2025-04-12T23:04:13+02:00: wrote 1227 chunks (4295.23 MB at 93.52 MB/s)
2025-04-12T23:04:58+02:00: wrote 1187 chunks (4297.33 MB at 96.31 MB/s)
2025-04-12T23:05:49+02:00: wrote 1149 chunks (4296.80 MB at 84.55 MB/s)
2025-04-12T23:06:35+02:00: wrote 1210 chunks (4296.54 MB at 93.06 MB/s)
2025-04-12T23:07:25+02:00: wrote 1203 chunks (4305.98 MB at 85.48 MB/s)
2025-04-12T23:08:10+02:00: wrote 1148 chunks (4301.26 MB at 96.21 MB/s)
2025-04-12T23:08:58+02:00: wrote 1205 chunks (4295.49 MB at 89.24 MB/s)
2025-04-12T23:09:48+02:00: wrote 1181 chunks (4296.02 MB at 87.02 MB/s)
2025-04-12T23:10:14+02:00: wrote 1144 chunks (4296.02 MB at 165.59 MB/s)
2025-04-12T23:10:40+02:00: wrote 1095 chunks (4300.73 MB at 165.58 MB/s)
2025-04-12T23:11:06+02:00: wrote 1136 chunks (4310.17 MB at 165.59 MB/s)
2025-04-12T23:11:37+02:00: wrote 1170 chunks (4297.85 MB at 137.62 MB/s)
2025-04-12T23:12:03+02:00: wrote 1089 chunks (4296.02 MB at 162.29 MB/s)
2025-04-12T23:12:30+02:00: wrote 1101 chunks (4301.26 MB at 164.52 MB/s)
2025-04-12T23:12:56+02:00: wrote 1130 chunks (4301.00 MB at 164.83 MB/s)
2025-04-12T23:13:26+02:00: wrote 1155 chunks (4295.49 MB at 140.93 MB/s)
2025-04-12T23:14:13+02:00: wrote 1148 chunks (4299.69 MB at 92.45 MB/s)
2025-04-12T23:14:56+02:00: wrote 1135 chunks (4296.54 MB at 98.99 MB/s)
2025-04-12T23:15:22+02:00: wrote 1118 chunks (4295.75 MB at 165.65 MB/s)
2025-04-12T23:15:48+02:00: wrote 1119 chunks (4296.02 MB at 165.52 MB/s)
2025-04-12T23:16:20+02:00: wrote 1092 chunks (4296.54 MB at 133.32 MB/s)
2025-04-12T23:16:54+02:00: wrote 1115 chunks (4295.49 MB at 134.19 MB/s)
2025-04-12T23:17:20+02:00: wrote 1162 chunks (4296.54 MB at 163.72 MB/s)

Zrzut ekranu 2025-04-12 o 23.31.58.png
 
Last edited:
  • Like
Reactions: _gabriel, UdoB and pibber
You must log in or register to reply here.
Top Bottom