Problems about PBS verify workers

robt · Sep 4, 2025

Hi I got a PBS server for manage the backup from PVE.
The size of the each vm backup file is about 48TB.
I want do the verify jobs after the new backup has created. But it has to take about 3 days to complete.
It really take too long. Is there any way to make's it faster?

The averge speed during the backup is about 200MB/s and it's stable. In my opinion it was too slow for my setup maybe?
Resourse comsume is about 3.11% of 64 CPU(s) and IO delay 1-2%

The datastore I used is ZFS which 3XRaidz2(each vdev has 6 spining disks with 6TB volume) with special mirror vdev(2X3.2T nvme ssd),
and using default 128K configuration for 'special_small_blocks'. The pool is enable compression using default level zstd.

BKP 53.4T 47.7T 5.56K 97 190M 6.53M
raidz2-0 18.6T 14.1T 1.94K 3 66.5M 50.0K
scsi-35000c50083e2c623 - - 332 0 11.1M 8.33K
scsi-35000c50083e2ccd3 - - 332 0 11.1M 8.33K
scsi-35000c50083e2ee07 - - 331 0 11.1M 8.33K
scsi-35000c50083e2f0e3 - - 330 0 11.0M 8.33K
scsi-35000c50083e2f2cf - - 330 0 11.0M 8.33K
scsi-35000c50083e315fb - - 331 0 11.1M 8.33K
raidz2-1 18.7T 14.0T 1.95K 3 66.9M 46.8K
scsi-35000c500843e110b - - 333 0 11.2M 7.80K
scsi-35000c50084413443 - - 333 0 11.2M 7.80K
scsi-35000c5008444e6a3 - - 332 0 11.1M 7.80K
scsi-35000c500845e6857 - - 331 0 11.1M 7.80K
scsi-35000c500845e95c3 - - 332 0 11.1M 7.80K
scsi-35000c500845ea863 - - 332 0 11.2M 7.80K
raidz2-2 15.5T 17.2T 1.61K 3 55.9M 29.9K
scsi-35000c5008472803f - - 274 0 9.30M 4.98K
scsi-35000c50084a653bb - - 273 0 9.31M 4.98K
scsi-35000c50084a6fea3 - - 274 0 9.33M 4.98K
scsi-35000c50084aa7ac3 - - 274 0 9.33M 4.98K
scsi-35000c50084b378c3 - - 274 0 9.32M 4.98K
scsi-35000c50084b3834f - - 274 0 9.31M 4.98K
special - - - - - -
mirror-3 586G 2.33T 65 86 539K 6.41M
nvme-MEMBLAZE_P6536CH0320M00_SH222503665 - - 32 43 270K 3.21M
nvme-MEMBLAZE_P6536CH0320M00_SH222902758 - - 32 43 269K 3.21M
CPU is AMD EPYC 7D12 64 core, with 128GB RAM
Kernel Version Linux 6.8.12-13-pve (2025-07-22T10:00Z)

I noticed that even I set special_small_blocks=128K, there still a lot 32K block reading from the spining disks.Is that a problem? (below is zpool iostats -rv 1 output)
scsi-35000c50084a6fea3 sync_read sync_write async_read async_write scrub trim rebuild
req_size ind agg ind agg ind agg ind agg ind agg ind agg ind agg
-------------------------------------------- ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- -----
512 0 0 0 0 0 0 0 0 0 0 0 0 0 0
1K 0 0 0 0 0 0 0 0 0 0 0 0 0 0
2K 0 0 0 0 0 0 0 0 0 0 0 0 0 0
4K 0 0 0 0 0 0 0 0 0 0 0 0 0 0
8K 0 0 0 0 0 0 0 0 0 0 0 0 0 0
16K 0 0 0 0 0 0 0 0 0 0 0 0 0 0
32K 183 0 0 0 90 0 0 0 0 0 0 0 0 0
64K 0 0 0 0 0 1 0 0 0 0 0 0 0 0
128K 0 0 0 0 0 0 0 0 0 0 0 0 0 0
256K 0 0 0 0 0 0 0 0 0 0 0 0 0 0
512K 0 0 0 0 0 0 0 0 0 0 0 0 0 0
1M 0 0 0 0 0 0 0 0 0 0 0 0 0 0
2M 0 0 0 0 0 0 0 0 0 0 0 0 0 0
4M 0 0 0 0 0 0 0 0 0 0 0 0 0 0
8M 0 0 0 0 0 0 0 0 0 0 0 0 0 0
16M 0 0 0 0 0 0 0 0 0 0 0 0 0 0
----------------------------------------------------------------------------------------------------------------------------------------------

UdoB · Sep 4, 2025

robt said:
I want do the verify jobs after the new backup has created. But it has to take about 3 days to complete.

The averge speed during the backup is about 200MB/s and it's stable. In my opinion it was too slow for my setup maybe?

During backup (=writing new data) or verify = reading chunks? Assuming verify for now, so:

You have three vdevs --> the IOPS of three spindles. You need to read a lot (approx. 12 million) of 4 MiB chunks, several will be smaller. 66 MB/s per spindle does not look bad!

I have a PBS with a pool with three mirrors = with IOPS of three spindles with old (=slower than yours) 3 TB WD Red, plus a mirrored Special Device with cheap (=non Enterprise) SSDs and I can get

Code:

# fio --name=randrw --ioengine=libaio --direct=1  --rw=randrw --bs=4M --numjobs=1 --iodepth=16 --size=22G --runtime=60 --time_based --rwmixread=99

  read: IOPS=10, BW=43.8MiB/s (45.9MB/s)(2628MiB/60003msec)
  write: IOPS=0, BW=751KiB/s (769kB/s)(44.0MiB/60003msec); 0 zone resets

  READ: bw=43.8MiB/s (45.9MB/s), 43.8MiB/s-43.8MiB/s (45.9MB/s-45.9MB/s), io=2628MiB (2756MB), run=60003-60003msec
  WRITE: bw=751KiB/s (769kB/s), 751KiB/s-751KiB/s (769kB/s-769kB/s), io=44.0MiB (46.1MB), run=60003-60003msec

This is an Intel i3 with 16 GiB Ram, 12 GiB ARC.

I am happy to read 43.8 MB per second with a cold cache. (A second run gives me 62MB/s, but that's irrelevant.) Given that the head movement of your newer disks can not be much faster than my old disks I am under the impression that your 66.6 MB/s is... acceptable.

robt · Sep 4, 2025

UdoB said:
During backup (=writing new data) or verify = reading chunks? Assuming verify for now, so:

You have three vdevs --> the IOPS of three spindles. You need to read a lot (approx. 12 million) of 4 MiB chunks, several will be smaller. 66 MB/s per spindle does not look bad!

I have a PBS with a pool with three mirrors = with IOPS of three spindles with old (=slower than yours) 3 TB WD Red, plus a mirrored Special Device with cheap (=non Enterprise) SSDs and I can get

Code:

# fio --name=randrw --ioengine=libaio --direct=1 --rw=randrw --bs=4M --numjobs=1 --iodepth=16 --size=22G --runtime=60 --time_based --rwmixread=99 read: IOPS=10, BW=43.8MiB/s (45.9MB/s)(2628MiB/60003msec) write: IOPS=0, BW=751KiB/s (769kB/s)(44.0MiB/60003msec); 0 zone resets READ: bw=43.8MiB/s (45.9MB/s), 43.8MiB/s-43.8MiB/s (45.9MB/s-45.9MB/s), io=2628MiB (2756MB), run=60003-60003msec WRITE: bw=751KiB/s (769kB/s), 751KiB/s-751KiB/s (769kB/s-769kB/s), io=44.0MiB (46.1MB), run=60003-60003msec

This is an Intel i3 with 16 GiB Ram, 12 GiB ARC.

I am happy to read 43.8 MB per second with a cold cache. (A second run gives me 62MB/s, but that's irrelevant.) Given that the head movement of your newer disks can not be much faster than my old disks I am under the impression that your 66.6 MB/s is... acceptable.

Yes, 200MB/s is for only verify,so it's better to get a large number of vdev? I'll try your scripit on my pool, let's see what we will got.

robt · Sep 4, 2025

robt said:
Yes, 200MB/s is for only verify,so it's better to get a large number of vdev? I'll try your scripit on my pool, let's see what we will got.

emmm......both good news and bad news......here's my result:fio --name=randrw --ioengine=libaio --direct=1 --rw=randrw --bs=4M --numjobs=1 --iodepth=16 --size=22G --runtime=60 --time_based --rwmixread=99

randrw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=libaio, iodepth=16
fio-3.33
Starting 1 process
randrw: Laying out IO file (1 file / 22528MiB)
Jobs: 1 (f=1): [m(1)][100.0%][r=3660MiB/s,w=40.0MiB/s][r=915,w=10 IOPS][eta 00m:00s]
randrw: (groupid=0, jobs=1): err= 0: pid=315665: Thu Sep 4 17:42:10 2025
read: IOPS=834, BW=3337MiB/s (3499MB/s)(196GiB/60001msec)
slat (usec): min=545, max=8512, avg=1186.94, stdev=359.96
clat (usec): min=3, max=66967, avg=17788.86, stdev=3785.87
lat (usec): min=1071, max=68320, avg=18975.79, stdev=4026.56
clat percentiles (usec):
| 1.00th=[15270], 5.00th=[15533], 10.00th=[15664], 20.00th=[15795],
| 30.00th=[15926], 40.00th=[16057], 50.00th=[16057], 60.00th=[16319],
| 70.00th=[16909], 80.00th=[19268], 90.00th=[23462], 95.00th=[25560],
| 99.00th=[31327], 99.50th=[39060], 99.90th=[41157], 99.95th=[45876],
| 99.99th=[61604]
bw ( MiB/s): min= 1520, max= 3816, per=99.94%, avg=3335.33, stdev=538.80, samples=119
iops : min= 380, max= 954, avg=833.83, stdev=134.70, samples=119
write: IOPS=8, BW=35.1MiB/s (36.8MB/s)(2108MiB/60001msec); 0 zone resets
slat (usec): min=688, max=1731, avg=862.46, stdev=76.69
clat (usec): min=14782, max=41586, avg=17878.19, stdev=3606.58
lat (usec): min=15629, max=42505, avg=18740.65, stdev=3614.74
clat percentiles (usec):
| 1.00th=[15270], 5.00th=[15664], 10.00th=[15795], 20.00th=[15795],
| 30.00th=[15926], 40.00th=[16057], 50.00th=[16188], 60.00th=[16712],
| 70.00th=[17171], 80.00th=[19268], 90.00th=[23725], 95.00th=[25035],
| 99.00th=[30016], 99.50th=[38011], 99.90th=[41681], 99.95th=[41681],
| 99.99th=[41681]
bw ( KiB/s): min= 8192, max=90112, per=99.88%, avg=35934.66, stdev=18165.39, samples=119
iops : min= 2, max= 22, avg= 8.77, stdev= 4.43, samples=119
lat (usec) : 4=0.01%
lat (msec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=82.21%, 50=17.75%
lat (msec) : 100=0.03%
cpu : usr=0.29%, sys=91.80%, ctx=20293, majf=0, minf=13
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=50059,527,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
READ: bw=3337MiB/s (3499MB/s), 3337MiB/s-3337MiB/s (3499MB/s-3499MB/s), io=196GiB (210GB), run=60001-60001msec
WRITE: bw=35.1MiB/s (36.8MB/s), 35.1MiB/s-35.1MiB/s (36.8MB/s-36.8MB/s), io=2108MiB (2210MB), run=60001-60001msec

According to this, I don't thinks the problem is from the raw device

UdoB · Sep 4, 2025

robt said:
read: IOPS=834, BW=3337MiB/s (3499MB/s)(196GiB/60001msec)

Yeah, that's great!

robt said:
According to this, I don't thinks the problem is from the raw device

Maybe. Maybe not.

How much Ram does the system have? For ARC? When you started that command for the first time the very first thing it does is to create the testfile by writing to it. After doing so the whole content may be in your ARC.

Some unverified source on "the net" tells me that I can clear ARC this way:

Code:

echo 0 > /sys/module/zfs/parameters/zfs_arc_shrinker_limit
echo 3 > /proc/sys/vm/drop_caches

Please do so and repeat the very same "fio".

floh8 · Sep 4, 2025

And of course, disable zfs compression. Pbs client brings its own compression. And your one can change to a higher recordsize.

Johannes S · Sep 4, 2025

floh8 said:
And of course, disable zfs compression. Pbs client brings its own compression. And your one can change to a higher recordsize.

ZFS compression doesn't hurt performance that bad if you use lzo ( which is defsult anyway) and might still be a benefit. The biggest performance-hurting factor are the RAIDZ with HDDs imho. That might be reduced ( although not fully) with striped mirror pools instead of the RAIDz and experimenting with changed zfs properies e.g. recordsize=1m

robt · Sep 5, 2025

UdoB said:
Yeah, that's great!

Maybe. Maybe not.

How much Ram does the system have? For ARC? When you started that command for the first time the very first thing it does is to create the testfile by writing to it. After doing so the whole content may be in your ARC.

Some unverified source on "the net" tells me that I can clear ARC this way:

Code:

echo 0 > /sys/module/zfs/parameters/zfs_arc_shrinker_limit echo 3 > /proc/sys/vm/drop_caches

Please do so and repeat the very same "fio".

Yeah, have tried to disable the arc memery,(actually reboot could always remove the stored arc data)here's value without arc cache:

randrw: (groupid=0, jobs=1): err= 0: pid=6171: Fri Sep 5 09:08:15 2025
read: IOPS=404, BW=1618MiB/s (1696MB/s)(27.6GiB/17446msec)
slat (usec): min=561, max=9714, avg=2457.67, stdev=771.70
clat (usec): min=2, max=74542, avg=36574.98, stdev=10901.24
lat (usec): min=1051, max=77397, avg=39032.65, stdev=11616.74
clat percentiles (usec):
| 1.00th=[15664], 5.00th=[15795], 10.00th=[15926], 20.00th=[16450],
| 30.00th=[40109], 40.00th=[41157], 50.00th=[41681], 60.00th=[42206],
| 70.00th=[42206], 80.00th=[42730], 90.00th=[43254], 95.00th=[43779],
| 99.00th=[47973], 99.50th=[58983], 99.90th=[71828], 99.95th=[73925],
| 99.99th=[74974]
bw ( MiB/s): min= 1168, max= 3736, per=96.53%, avg=1561.65, stdev=555.78, samples=34
iops : min= 292, max= 934, avg=390.41, stdev=138.95, samples=34
write: IOPS=4, BW=19.7MiB/s (20.7MB/s)(344MiB/17446msec); 0 zone resets
slat (usec): min=839, max=1504, avg=940.07, stdev=141.27
clat (usec): min=15848, max=50051, avg=37532.18, stdev=10341.43
lat (usec): min=16714, max=51555, avg=38472.25, stdev=10379.82
clat percentiles (usec):
| 1.00th=[15795], 5.00th=[16057], 10.00th=[16188], 20.00th=[37487],
| 30.00th=[41157], 40.00th=[41681], 50.00th=[42206], 60.00th=[42206],
| 70.00th=[42730], 80.00th=[43254], 90.00th=[44303], 95.00th=[45351],
| 99.00th=[50070], 99.50th=[50070], 99.90th=[50070], 99.95th=[50070],
| 99.99th=[50070]
bw ( KiB/s): min= 8192, max=40960, per=97.13%, avg=19611.15, stdev=8674.29, samples=33
iops : min= 2, max= 10, avg= 4.79, stdev= 2.12, samples=33
lat (usec) : 4=0.01%
lat (msec) : 2=0.01%, 4=0.03%, 10=0.08%, 20=20.65%, 50=78.41%
lat (msec) : 100=0.80%
cpu : usr=0.22%, sys=56.27%, ctx=22216, majf=0, minf=11
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=99.8%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=7056,86,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
READ: bw=1618MiB/s (1696MB/s), 1618MiB/s-1618MiB/s (1696MB/s-1696MB/s), io=27.6GiB (29.6GB), run=17446-17446msec
WRITE: bw=19.7MiB/s (20.7MB/s), 19.7MiB/s-19.7MiB/s (20.7MB/s-20.7MB/s), io=344MiB (361MB), run=17446-17446msec

which I think still fast?

And also to make things more obvious, here is the same fio test I do for the raw 6T apining disks:
fio --name=/dev/sdaj --ioengine=libaio --direct=1 --rw=randrw --bs=4M --numjobs=1 --iodepth=16 --size=22G --runtime=60 --time_based --rwmixread=99
/dev/sdaj: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=libaio, iodepth=16
fio-3.33
Starting 1 process
Jobs: 1 (f=1): [m(1)][100.0%][r=160MiB/s][r=40 IOPS][eta 00m:00s]
/dev/sdaj: (groupid=0, jobs=1): err= 0: pid=6316: Fri Sep 5 08:52:54 2025
read: IOPS=38, BW=153MiB/s (161MB/s)(9232MiB/60202msec)
slat (usec): min=294, max=354193, avg=25688.49, stdev=7888.27
clat (msec): min=194, max=1323, avg=382.74, stdev=45.57
lat (msec): min=224, max=1347, avg=408.43, stdev=45.81
clat percentiles (msec):
| 1.00th=[ 309], 5.00th=[ 330], 10.00th=[ 342], 20.00th=[ 359],
| 30.00th=[ 368], 40.00th=[ 376], 50.00th=[ 384], 60.00th=[ 388],
| 70.00th=[ 393], 80.00th=[ 397], 90.00th=[ 409], 95.00th=[ 439],
| 99.00th=[ 567], 99.50th=[ 634], 99.90th=[ 701], 99.95th=[ 718],
| 99.99th=[ 1318]
bw ( KiB/s): min=131072, max=172032, per=100.00%, avg=157850.89, stdev=7518.50, samples=119
iops : min= 32, max= 42, avg=38.54, stdev= 1.84, samples=119
write: IOPS=0, BW=1905KiB/s (1951kB/s)(112MiB/60202msec); 0 zone resets
slat (usec): min=398, max=34726, avg=25148.53, stdev=6003.62
clat (msec): min=325, max=2714, avg=577.41, stdev=604.47
lat (msec): min=355, max=2738, avg=602.55, stdev=604.13
clat percentiles (msec):
| 1.00th=[ 326], 5.00th=[ 363], 10.00th=[ 372], 20.00th=[ 372],
| 30.00th=[ 384], 40.00th=[ 393], 50.00th=[ 401], 60.00th=[ 409],
| 70.00th=[ 418], 80.00th=[ 439], 90.00th=[ 793], 95.00th=[ 2668],
| 99.00th=[ 2702], 99.50th=[ 2702], 99.90th=[ 2702], 99.95th=[ 2702],
| 99.99th=[ 2702]
bw ( KiB/s): min= 8192, max=16384, per=100.00%, avg=9972.87, stdev=3454.90, samples=23
iops : min= 2, max= 4, avg= 2.43, stdev= 0.84, samples=23
lat (msec) : 250=0.13%, 500=97.82%, 750=1.88%, 1000=0.04%, 2000=0.04%
lat (msec) : >=2000=0.09%
cpu : usr=0.03%, sys=1.99%, ctx=75046, majf=0, minf=12
IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=2308,28,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
READ: bw=153MiB/s (161MB/s), 153MiB/s-153MiB/s (161MB/s-161MB/s), io=9232MiB (9680MB), run=60202-60202msec
WRITE: bw=1905KiB/s (1951kB/s), 1905KiB/s-1905KiB/s (1951kB/s-1951kB/s), io=112MiB (117MB), run=60202-60202msec

Disk stats (read/write):
sdaj: ios=74200/899, merge=0/0, ticks=15010918/312405, in_queue=15323323, util=100.00%

UdoB · Sep 5, 2025

robt said:
which I think still fast?

Yes

At least for all system I have access to.

Unfortunately I do not have another idea; maybe some other forum user will chime in...

robt · Sep 5, 2025

I was confused about some part of the code of the verify........(https://github.com/proxmox/proxmox-backup/blob/master/src/backup/verify.rs)

Code:

let decoder_pool = ParallelHandler::new(
        "verify chunk decoder",
        4,
        move |(chunk, digest, size): (DataBlob, [u8; 32], u64)| {
            let chunk_crypt_mode = match chunk.crypt_mode() {
                Err(err) => {
                    corrupt_chunks2.lock().unwrap().insert(digest);
                    task_log!(worker2, "can't verify chunk, unknown CryptMode - {}", err);
                    errors2.fetch_add(1, Ordering::SeqCst);
                    return Ok(());
                }
                Ok(mode) => mode,
            };

            if chunk_crypt_mode != crypt_mode {
                task_log!(
                    worker2,
                    "chunk CryptMode {:?} does not match index CryptMode {:?}",
                    chunk_crypt_mode,
                    crypt_mode
                );
                errors2.fetch_add(1, Ordering::SeqCst);
            }

Here in my opinion, this part start 4 verify chunk decoder to check blocks.
Is that enough to use the full performance of the disk? After testing the pool performance, I really can't think others

Is there anyone can help me with this?

robt · Sep 9, 2025

Johannes S said:
ZFS compression doesn't hurt performance that bad if you use lzo ( which is defsult anyway) and might still be a benefit. The biggest performance-hurting factor are the RAIDZ with HDDs imho. That might be reduced ( although not fully) with striped mirror pools instead of the RAIDz and experimenting with changed zfs properies e.g. recordsize=1m

New updates! But even worse.
To keep the old pool working, I used another 18 spining disk to create a new pool. And using zfs send | zfs recieve -F to clone data to the new pool
Here's the new pool:

Code:

pool: TEST
 state: ONLINE
config:

        NAME                        STATE     READ WRITE CKSUM
        TEST                        ONLINE       0     0     0
          raidz1-0                  ONLINE       0     0     0
            scsi-35000c5006363806b  ONLINE       0     0     0
            scsi-35000c500636399eb  ONLINE       0     0     0
            scsi-35000c500833964d3  ONLINE       0     0     0
          raidz1-1                  ONLINE       0     0     0
            scsi-35000c50083e2f01b  ONLINE       0     0     0
            scsi-35000c5008442cb57  ONLINE       0     0     0
            scsi-35000c500845eb98b  ONLINE       0     0     0
          raidz1-2                  ONLINE       0     0     0
            scsi-35000c50097289623  ONLINE       0     0     0
            scsi-35000c500972916df  ONLINE       0     0     0
            scsi-35000c50083b93f93  ONLINE       0     0     0
          raidz1-3                  ONLINE       0     0     0
            scsi-35000c50084991b0f  ONLINE       0     0     0
            scsi-35000c5008461f88f  ONLINE       0     0     0
            scsi-35000c50083d3ab9f  ONLINE       0     0     0
          raidz1-4                  ONLINE       0     0     0
            scsi-35000c500849657eb  ONLINE       0     0     0
            scsi-35000c50098defeb3  ONLINE       0     0     0
            scsi-35000c50098ded32f  ONLINE       0     0     0
          raidz1-5                  ONLINE       0     0     0
            scsi-35000c500845f19c7  ONLINE       0     0     0
            scsi-35000c50084eaacc7  ONLINE       0     0     0
            scsi-35000c50084ec959b  ONLINE       0     0     0

errors: No known data errors

Something that I've tried:
Try1.Stay the old 128K recordsize, just increase number of vdevs by using raidz1.
Try2.Destory the new pool and create with new 1M recordsize. Then copy the data to the new pool again.

Here's some results:
When I do Try1, the speed of verify is close to 150MB/s , which is slower than the old one. But it's quite fair because it doesn't got special devices.
But when I set the recordsize to 1M (which is Try2), things become interesting. The speed of verify becomes even slower to around 100-120MB/s.

Is there anything wrong with my settings? Or there's some tricky things in ZFS send and recieve command?

floh8 · Sep 9, 2025

use striped mirror zfs pool!

robt · Sep 10, 2025

floh8 said:
使用条带镜像 zfs 池！

哦,I will test using 6 raid 1 vdev.

robt · Sep 13, 2025

floh8 said:
use striped mirror zfs pool!

New,updates now here's new pool

Code:

pool: TEST
 state: ONLINE
config:

        NAME                      STATE     READ WRITE CKSUM
        TEST                      ONLINE       0     0     0
          scsi-35000c5006363806b  ONLINE       0     0     0
          scsi-35000c500636399eb  ONLINE       0     0     0
          scsi-35000c500833964d3  ONLINE       0     0     0
          scsi-35000c50083e2f01b  ONLINE       0     0     0
          scsi-35000c5008442cb57  ONLINE       0     0     0
          scsi-35000c500845eb98b  ONLINE       0     0     0
          scsi-35000c50097289623  ONLINE       0     0     0
          scsi-35000c500972916df  ONLINE       0     0     0
          scsi-35000c50083b93f93  ONLINE       0     0     0
          scsi-35000c50084991b0f  ONLINE       0     0     0
          scsi-35000c5008461f88f  ONLINE       0     0     0
          scsi-35000c50083d3ab9f  ONLINE       0     0     0

To be clear, I don't have enough disks to do the mirror, so I used 12 striped pool.
But the result are almost same as the raidz2 in both 128K and 1M record size. Which is around 100-120MB/s......
The problem may comes from the verify code it self.........

Search

Search

Problems about PBS verify workers

robt

New Member

UdoB

Distinguished Member

robt

New Member

robt

New Member

UdoB

Distinguished Member

floh8

Renowned Member

Johannes S

Famous Member

robt

New Member

UdoB

Distinguished Member

robt

New Member

robt

New Member

floh8

Renowned Member

robt

New Member

robt

New Member

We value your privacy