zfs send over 10Gbe horribly slow

[oppersnokker]

I am trying to use zfs send to duplicate and entire pool between 2 nodes and I am getting very slow results. Any help would be appreciated.

Hardware (both nodes identical):
HPZ840, Dual Xeon 36 core, 256GB ram
Asus M4 with 4x 2TB WD SSN7100 in RaidZ
NIC Intel X550T2 10Gbe

Testing pool speeds using:
dd if=/dev/zero of=/nvmepool/testfile bs=1G count=4 oflag=direct
Node 1: 4294967296 bytes (4.3 GB, 4.0 GiB) copied, 3.13623 s, 1.4 GB/s
Node 2: 294967296 bytes (4.3 GB, 4.0 GiB) copied, 1.89471 s, 2.3 GB/s

Network (both nodes):
ethtool ens5f1 | grep Speed
Speed: 10000Mb/s

First, testing network speeds I am only getting around 250MB between both nodes.
dd if=/dev/zero bs=1G count=10 | ssh root@10.0.0.2 "cat > /tmp/testfile"
10737418240 bytes (11 GB, 10 GiB) copied, 43.268 s, 248 MB/s

MTU is set to 1500 and also I tried different cables, Cat5e, Cat6, Cat6e. Nodes sit right next to each other, 50cm distance.
ens5f1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000

Using a zfs send, I am only getting around 15MB/s and in verbose mode it keeps sending 43.3K chunks if I am correct.
15:10:29 43.3K nvmepool@migration

Anyone an idea how to improve this? I am at a loss.

 

[news]

please use /dev/random for your tests.
dd if=/dev/random of=test.raw bs=1M count=10240 status=progress

 

[news]

for the raw network speed.
check ethtool <device>
and after that use:
iperf3 as client/ server tool

 

[waltar]

First, testing network speeds I am only getting around 250MB between both nodes.

You would be at 100% cpu usage from ssh process this way - see in "top" !!
When you install nfs-server eg temporary you could use nfs mounts (1->2 && 2->1) while were able to reach 1200MB/s with nvme's.