ZFS quite slow on 10Gb/s uas-devices

[gunpie]

I'm currently move my PVE to new hosts which have no SATA-Interfaces, thus i decided using USB-boxes (uas) with 4 bays as storage.
Initial test shows that the boxes provided reasonable performance:

root@mini02:~# dd bs=8k of=/dev/null if=/dev/sda&
root@mini02:~# dd bs=8k of=/dev/null if=/dev/sdb&
root@mini02:~# dd bs=8k of=/dev/null if=/dev/sdc&
root@mini02:~# dd bs=8k of=/dev/null if=/dev/sdd&

and the performance looked good:

root@mini02:~# S_COLORS=never S_TIME_FORMAT=ISO iostat -xczNtm 5
...
2025-10-27T00:14:10+0100
avg-cpu: %user %nice %system %iowait %steal %idle
0.03 0.00 0.65 11.65 0.00 87.67

Device r/s rMB/s rrqm/s %rrqm r_await rareq-sz w/s wMB/s wrqm/s %wrqm w_await wareq-sz d/s dMB/s drqm/s %drqm d_await dareq-sz f/s f_await aqu-sz %util
nvme0n1 0.00 0.00 0.00 0.00 0.00 0.00 13.60 0.15 0.00 0.00 1.22 11.47 0.00 0.00 0.00 0.00 0.00 0.00 0.40 0.50 0.02 0.82
nvme1n1 0.00 0.00 0.00 0.00 0.00 0.00 12.20 0.15 0.00 0.00 1.05 12.79 0.00 0.00 0.00 0.00 0.00 0.00 0.40 1.00 0.01 0.66
sda 2019.00 252.38 0.00 0.00 0.97 128.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 1.96 100.00
sdb 2033.40 254.17 0.00 0.00 0.97 128.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 1.96 100.00
sdc 2024.60 253.07 0.00 0.00 0.97 128.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 1.96 100.00
sdd 2010.40 251.30 0.00 0.00 0.98 128.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 1.96 100.00

However after initializing the devices for ZFS, the performance of "zfs send" (on a subvolume) is quite low:

2025-10-27T00:07:32+0100
avg-cpu: %user %nice %system %iowait %steal %idle
0.41 0.00 1.35 0.03 0.00 98.21

Device r/s rMB/s rrqm/s %rrqm r_await rareq-sz w/s wMB/s wrqm/s %wrqm w_await wareq-sz d/s dMB/s drqm/s %drqm d_await dareq-sz f/s f_await aqu-sz %util
nvme0n1 0.00 0.00 0.00 0.00 0.00 0.00 12.00 0.12 0.00 0.00 1.38 10.27 0.00 0.00 0.00 0.00 0.00 0.00 0.40 1.00 0.02 0.86
nvme1n1 0.00 0.00 0.00 0.00 0.00 0.00 12.60 0.12 0.00 0.00 1.00 9.78 0.00 0.00 0.00 0.00 0.00 0.00 0.40 0.50 0.01 0.68
sda 81.00 19.10 0.00 0.00 35.03 241.51 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 2.84 82.88
sdb 81.60 19.73 0.00 0.00 33.93 247.54 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 2.77 83.08
sdc 103.60 20.32 0.00 0.00 8.05 200.88 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.83 26.30
sdd 102.40 19.57 0.00 0.00 4.58 195.75 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.47 14.96

Doing the same operation on the original host with SATA-Interfaces on disks of the same type and identical initialization looks like this:

root@adam:~# S_COLORS=never S_TIME_FORMAT=ISO iostat -xcmzNt 5
...
2025-10-30T22:56:59+0100
avg-cpu: %user %nice %system %iowait %steal %idle
9.14 0.00 9.57 0.91 0.00 80.38

Device r/s rMB/s rrqm/s %rrqm r_await rareq-sz w/s wMB/s wrqm/s %wrqm w_await wareq-sz d/s dMB/s drqm/s %drqm d_await dareq-sz f/s f_await aqu-sz %util
nvme0n1 0.00 0.00 0.00 0.00 0.00 0.00 13.20 0.10 0.00 0.00 0.47 7.39 0.00 0.00 0.00 0.00 0.00 0.00 0.40 2.00 0.01 0.36
nvme1n1 0.00 0.00 0.00 0.00 0.00 0.00 13.20 0.10 0.00 0.00 0.24 7.39 0.00 0.00 0.00 0.00 0.00 0.00 0.40 2.00 0.00 0.22
sda 409.60 115.64 0.00 0.00 5.57 289.11 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 2.28 79.60
sdb 406.20 115.12 0.00 0.00 6.06 290.21 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 2.46 83.86
sdc 374.80 130.14 0.00 0.00 5.98 355.56 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 2.24 77.52
sdd 377.80 130.61 0.00 0.00 6.56 354.01 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 2.48 85.30

Any idea to explain this discrepancies in performance?

 

[alexskysilk]

yes. your USB enclosure doesnt do what you think it does.

Just because the marketing says its "USB 3.2 10gbit blah blah blah" doesnt mean that your host connection can, that the cable can, that the bridge chip can, nor the sata multiplexer.

you're getting 400MB/S to the storage. I'd call that a win.

 

[gunpie]

Sorry, but the initial test shows clearly, the the setup functions well at 10Gb/s USB-Speed with impressive IOP/s and fast parallel responses from the disks.
The USB "zfs send" looks as if it process just one request at a time instead and thust delivers very poor performance, but the above proves that the devices can do more.

The stats from the SATA-host are also poor, compared to the stream test, but show acceptable performance.