[SOLVED] Mellanox ConnectX-2 (MT26448)

[K3N8]

Hi,

I bought two 10GbE network cards and a SFP+ DAC cable on Ebay. I connected two Dell PowerEdge R720 servers and assigned them both a statis IP address on a different subnet. Set MTU at 9000. Everything works fine. Here is the output from iPerf:
Connecting to host 192.168.0.2, port 5201 [ 5] local 192.168.0.1 port 49726 connected to 192.168.0.2 port 5201 [ ID] Interval Transfer Bitrate Retr Cwnd [ 5] 0.00-1.00 sec 1.14 GBytes 9.77 Gbits/sec 0 1.45 MBytes [ 5] 1.00-2.00 sec 1.15 GBytes 9.89 Gbits/sec 0 1.45 MBytes [ 5] 2.00-3.00 sec 1.15 GBytes 9.89 Gbits/sec 0 1.45 MBytes [ 5] 3.00-4.00 sec 1.12 GBytes 9.64 Gbits/sec 0 1.45 MBytes [ 5] 4.00-5.00 sec 1.15 GBytes 9.89 Gbits/sec 0 1.45 MBytes [ 5] 5.00-6.00 sec 1.15 GBytes 9.90 Gbits/sec 0 1.45 MBytes [ 5] 6.00-7.00 sec 1.15 GBytes 9.89 Gbits/sec 0 1.45 MBytes [ 5] 7.00-8.00 sec 1.15 GBytes 9.89 Gbits/sec 0 1.45 MBytes [ 5] 8.00-9.00 sec 1.15 GBytes 9.89 Gbits/sec 0 1.45 MBytes [ 5] 9.00-10.00 sec 1.15 GBytes 9.89 Gbits/sec 0 1.45 MBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-10.00 sec 11.5 GBytes 9.85 Gbits/sec 0 sender [ 5] 0.00-10.00 sec 11.5 GBytes 9.85 Gbits/sec receiver iperf Done.

When I copy my ZFS snapshots from one server to the other I max out at 200MB/s. I use a SSH tunnel to copy the data. Anyone have any ideas?

 

[spirit]

ssh tunnel can limit speed, as it's not multithreaded. (around 2-3gbit/s max on fast cpu).

maybe look at ssh process, and see if you have 1core 100%.

 

[K3N8]

That would be an explanation. I also tried Netcat. Same issue. Any ideas how to make it faster?

 

[Dunuin]

In case you need no encryption and just want to copy stuff on the LAN you could use SSH with disabled encryption + netcat. I think that way you can copy multithreaded.

 

[K3N8]

I'll try that.