zfs send over 10Gbe horribly slow

O

oppersnokker

New Member
Sep 18, 2025
4
0
1
I am trying to use zfs send to duplicate and entire pool between 2 nodes and I am getting very slow results. Any help would be appreciated.

Hardware (both nodes identical):
HPZ840, Dual Xeon 36 core, 256GB ram
Asus M4 with 4x 2TB WD SSN7100 in RaidZ
NIC Intel X550T2 10Gbe

Testing pool speeds using:
dd if=/dev/zero of=/nvmepool/testfile bs=1G count=4 oflag=direct
Node 1: 4294967296 bytes (4.3 GB, 4.0 GiB) copied, 3.13623 s, 1.4 GB/s
Node 2: 294967296 bytes (4.3 GB, 4.0 GiB) copied, 1.89471 s, 2.3 GB/s

Network (both nodes):
ethtool ens5f1 | grep Speed
Speed: 10000Mb/s

First, testing network speeds I am only getting around 250MB between both nodes.
dd if=/dev/zero bs=1G count=10 | ssh root@10.0.0.2 "cat > /tmp/testfile"
10737418240 bytes (11 GB, 10 GiB) copied, 43.268 s, 248 MB/s

MTU is set to 1500 and also I tried different cables, Cat5e, Cat6, Cat6e. Nodes sit right next to each other, 50cm distance.
ens5f1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000

Using a zfs send, I am only getting around 15MB/s and in verbose mode it keeps sending 43.3K chunks if I am correct.
15:10:29 43.3K nvmepool@migration

Anyone an idea how to improve this? I am at a loss. :(
 
please use /dev/random for your tests.
dd if=/dev/random of=test.raw bs=1M count=10240 status=progress
 
for the raw network speed.
check ethtool <device>
and after that use:
iperf3 as client/ server tool
 
oppersnokker said:
First, testing network speeds I am only getting around 250MB between both nodes.
Click to expand...
You would be at 100% cpu usage from ssh process this way - see in "top" !!
When you install nfs-server eg temporary you could use nfs mounts (1->2 && 2->1) while were able to reach 1200MB/s with nvme's.
 
Last edited:
waltar said:
You would be at 100% cpu usage from ssh process this way - see in "top" !!
When you install nfs-server eg temporary you could use nfs mounts (1->2 && 2->1) while were able to reach 1200MB/s with nvme's.
Click to expand...
Just checked, CPU load is 5% max. I'll try the nfs mounts!
 
/dev/random is slow until stops while out of enthropie, so /dev/urandom is usefull but even not as endless fast as /dev/zero (unuseable for fs with compression) and /dev/null. ethtool shows connected interface capability (out of possibilities) without testing performance.
iperf(3) is a tool to check bandwidth but nfs is more real as it transfers real data while even you can check a possible bottleneck easy if any until interface capability is reached. PS: zfs send/recv cannot use nfs, I just mentioned nfs for checking your I/O, cpu or network observed bottleneck.
 
waltar said:
/dev/random is slow until stops while out of enthropie, so /dev/urandom is usefull
Click to expand...
Yes, that's burned into my own mind too.

But "something" has changed. I can read 5 GB from both devices with the same speed:
Code: 
~# dd if=/dev/urandom of=/dev/zero bs=1M count=5000
5000+0 records in
5000+0 records out
5242880000 bytes (5.2 GB, 4.9 GiB) copied, 8.25407 s, 635 MB/s

~# dd if=/dev/random of=/dev/zero bs=1M count=5000
5000+0 records in
5000+0 records out
5242880000 bytes (5.2 GB, 4.9 GiB) copied, 8.28133 s, 633 MB/s

(This is a AMD Ryzen 7 7700X with uptodate PVE 9 / Trixie)
 
  • Like
Reactions: news
of=/dev/zero ?? better be /dev/null otherwise get the sunglasses
:cool: Interesting random as fast as urandom ... what I saw too on newer cpus you can do same as fast as having cores which was previously only one max and with more processes only ever smaller parts of the max.
 
  • Like
Reactions: UdoB
waltar said:
of=/dev/zero ?? better be /dev/null otherwise get the sunglasses
Click to expand...
Yes. Maybe. No. On the same machine as above:

Code: 
~# dd if=/dev/random of=/dev/null bs=1M count=5000
5000+0 records in
5000+0 records out
5242880000 bytes (5.2 GB, 4.9 GiB) copied, 8.26672 s, 634 MB/s

~# dd if=/dev/urandom of=/dev/null bs=1M count=5000
5000+0 records in
5000+0 records out
5242880000 bytes (5.2 GB, 4.9 GiB) copied, 8.25938 s, 635 MB/s

zero or null seems to behave exactly the same (for this test).
 
  • Like
Reactions: waltar
You must log in or register to reply here.
Top Bottom