Dolphin adapter DXH510 on 2.6.18

Oct 22, 2009
92
1
26
Hi,

I have a problem with Dolphin DXH510 latency on the 2.6.18 kernel. DRBD write latency is around 300us. Dolphinics has identified some issues around DMA and interrupt handling - I don't know if it's particularly on my system or if it's a general issue with this kernel. Anybody else with similar problems?
If I change to 2.6.24 or .32 I'm getting write latency around 100-110us which is quite good.
I have tested a Myricom adapter also which gives around 130us. This is still good, and works fine in all three kernels.

Thanks in advance,
Bo
 
I've now tested a Centos installation (2.6.18-194.11.1.el5). This does not suffer the same latency problem. What is the basis of pve-2.6.18 at the moment?

ovzkernel-2.6.18-194.3.1.el5.028stab069.5.src.rpm

newer versions have serious problems with kvm, so we can't update.
 
DRBD latency measurement (dd if=/dev/zero of=dev/drbdvg1/drbdlvm bs=512 count=1000 oflag=direct)
disconnected DRBD (adaptec 5445Z) 54us
1GbE 425us
10GbE (myricom) 120-130us
Dolphin 100-110us (2.6.32 kernel)

Throughput on my test system was 600MB/s when synchronizing using Myricom. Dolphin I got up to 550MB/s, but as this post mentions I have a latency problem using 2.6.18 kernel regarding the Dolphin adapter.
Also note I am following a few tweaks which means hyperthreading, power saving etc. has been disabled. Test setup is 20 sata disks in raid0 on each server.
 
DRBD latency measurement (dd if=/dev/zero of=dev/drbdvg1/drbdlvm bs=512 count=1000 oflag=direct)
disconnected DRBD (adaptec 5445Z) 54us
1GbE 425us
10GbE (myricom) 120-130us
Dolphin 100-110us (2.6.32 kernel)

Throughput on my test system was 600MB/s when synchronizing using Myricom. Dolphin I got up to 550MB/s, but as this post mentions I have a latency problem using 2.6.18 kernel regarding the Dolphin adapter.
Also note I am following a few tweaks which means hyperthreading, power saving etc. has been disabled. Test setup is 20 sata disks in raid0 on each server.
Hi,
thanks for the info - impressive values! With this speed DRBD makes sense not only for things with low-io-usage.

Udo
 
Hi,
a little bit OT: with which "trick" do you compile the dolphin-driver? I just get two cards, but the install-programm don't run (don't find .config and include-files).
I open a ticket, but perhaps you can give me a faster hint.

Udo
 
Hi Udo,

I extracted the source tarball using "DIS_.......sh --get-tarball". Extracting that tarball you'll find a "build_deb"-script located in "adm/bin/Linux_pkgs/". You will need the kernel headers to compile. Otherwise I think it was quite straight forward to sort out the other dependencies.
You need to run "build_deb --dx" to compile for the DX adapter. See "build_deb --help" for more options.

Bo
 
Hi Udo,

I extracted the source tarball using "DIS_.......sh --get-tarball". Extracting that tarball you'll find a "build_deb"-script located in "adm/bin/Linux_pkgs/". You will need the kernel headers to compile. Otherwise I think it was quite straight forward to sort out the other dependencies.
You need to run "build_deb --dx" to compile for the DX adapter. See "build_deb --help" for more options.

Bo
Hi Bo,
thanks for the explanation. I'm now able to compile the software... very good. Now i must understand how the dolphin-nics are configured.

Udo
 
Hi Udo,

I extracted the source tarball using "DIS_.......sh --get-tarball". Extracting that tarball you'll find a "build_deb"-script located in "adm/bin/Linux_pkgs/". You will need the kernel headers to compile. Otherwise I think it was quite straight forward to sort out the other dependencies.
You need to run "build_deb --dx" to compile for the DX adapter. See "build_deb --help" for more options.

Bo
Hi Bo,
all packets are build sucessfull, only the dolphin-supersockets-dx don't build :-(
Which version do you use? I try the 3.6.1.5 an get only the warning/error:
Code:
The order of dolphin-sisci dolphin-sisci-devel dolphin-networkhosts dolphin-networkmanager dolphin-networkadmin DIS_DX SuperSockets is not optional. /tmp/DISDPKGROOT_MrHTWQ/SOURCES
...
cp: cannot stat `/tmp/DISDPKGROOT_gFgoeH/BUILD/dolphin-dx-3.6.1.5-2.6.32-4-pve/opt/DIS/lib64/librmlib*': No such file or directory
mv: cannot stat `/tmp/DISDPKGROOT_gFgoeH/BUILD/dolphin-dx-3.6.1.5-2.6.32-4-pve/opt/DIS/bin/reflective_memory': No such file or directory
mv: cannot stat `/tmp/DISDPKGROOT_gFgoeH/BUILD/dolphin-dx-3.6.1.5-2.6.32-4-pve/opt/DIS/lib64/librmlib*': No such file or directory
Are there other dependencies? Have you changed the package-order?

Can you put the dolphin-supersockets-dx*.deb on a ftp-server, please? (for first testing - of course i need to compile all successful)

Without the supersockets, the cards are useless for drbd...


Regards

Udo
 
I went for the Myricom NICs so I don't remember much about the Dolphin adapters, sorry :(. Neither did I keep the binaries - they went away with a clean install of proxmox after my testing :(.
What I have is a "DIS_DX_install_DIS_RELEASE_3_6_1_2_AUGUST_18_2010.sh" which probably was the last one I tried. Don't know if that is still downloadable from Dolphinics? Otherwise you
could try to contact Dolphinics. They were very interested in helping when I tested.