Dolphin adapter DXH510 on 2.6.18

Oct 22, 2009
92
1
26
Hi,

I have a problem with Dolphin DXH510 latency on the 2.6.18 kernel. DRBD write latency is around 300us. Dolphinics has identified some issues around DMA and interrupt handling - I don't know if it's particularly on my system or if it's a general issue with this kernel. Anybody else with similar problems?
If I change to 2.6.24 or .32 I'm getting write latency around 100-110us which is quite good.
I have tested a Myricom adapter also which gives around 130us. This is still good, and works fine in all three kernels.

Thanks in advance,
Bo
 
I've now tested a Centos installation (2.6.18-194.11.1.el5). This does not suffer the same latency problem. What is the basis of pve-2.6.18 at the moment?
 
I've now tested a Centos installation (2.6.18-194.11.1.el5). This does not suffer the same latency problem. What is the basis of pve-2.6.18 at the moment?

ovzkernel-2.6.18-194.3.1.el5.028stab069.5.src.rpm

newer versions have serious problems with kvm, so we can't update.
 
DRBD latency measurement (dd if=/dev/zero of=dev/drbdvg1/drbdlvm bs=512 count=1000 oflag=direct)
disconnected DRBD (adaptec 5445Z) 54us
1GbE 425us
10GbE (myricom) 120-130us
Dolphin 100-110us (2.6.32 kernel)

Throughput on my test system was 600MB/s when synchronizing using Myricom. Dolphin I got up to 550MB/s, but as this post mentions I have a latency problem using 2.6.18 kernel regarding the Dolphin adapter.
Also note I am following a few tweaks which means hyperthreading, power saving etc. has been disabled. Test setup is 20 sata disks in raid0 on each server.
 
DRBD latency measurement (dd if=/dev/zero of=dev/drbdvg1/drbdlvm bs=512 count=1000 oflag=direct)
disconnected DRBD (adaptec 5445Z) 54us
1GbE 425us
10GbE (myricom) 120-130us
Dolphin 100-110us (2.6.32 kernel)

Throughput on my test system was 600MB/s when synchronizing using Myricom. Dolphin I got up to 550MB/s, but as this post mentions I have a latency problem using 2.6.18 kernel regarding the Dolphin adapter.
Also note I am following a few tweaks which means hyperthreading, power saving etc. has been disabled. Test setup is 20 sata disks in raid0 on each server.
Hi,
thanks for the info - impressive values! With this speed DRBD makes sense not only for things with low-io-usage.

Udo
 
Hi,
a little bit OT: with which "trick" do you compile the dolphin-driver? I just get two cards, but the install-programm don't run (don't find .config and include-files).
I open a ticket, but perhaps you can give me a faster hint.

Udo
 
Hi Udo,

I extracted the source tarball using "DIS_.......sh --get-tarball". Extracting that tarball you'll find a "build_deb"-script located in "adm/bin/Linux_pkgs/". You will need the kernel headers to compile. Otherwise I think it was quite straight forward to sort out the other dependencies.
You need to run "build_deb --dx" to compile for the DX adapter. See "build_deb --help" for more options.

Bo
 
Hi Udo,

I extracted the source tarball using "DIS_.......sh --get-tarball". Extracting that tarball you'll find a "build_deb"-script located in "adm/bin/Linux_pkgs/". You will need the kernel headers to compile. Otherwise I think it was quite straight forward to sort out the other dependencies.
You need to run "build_deb --dx" to compile for the DX adapter. See "build_deb --help" for more options.

Bo
Hi Bo,
thanks for the explanation. I'm now able to compile the software... very good. Now i must understand how the dolphin-nics are configured.

Udo
 
Hi Udo,

I extracted the source tarball using "DIS_.......sh --get-tarball". Extracting that tarball you'll find a "build_deb"-script located in "adm/bin/Linux_pkgs/". You will need the kernel headers to compile. Otherwise I think it was quite straight forward to sort out the other dependencies.
You need to run "build_deb --dx" to compile for the DX adapter. See "build_deb --help" for more options.

Bo
Hi Bo,
all packets are build sucessfull, only the dolphin-supersockets-dx don't build :-(
Which version do you use? I try the 3.6.1.5 an get only the warning/error:
Code:
The order of dolphin-sisci dolphin-sisci-devel dolphin-networkhosts dolphin-networkmanager dolphin-networkadmin DIS_DX SuperSockets is not optional. /tmp/DISDPKGROOT_MrHTWQ/SOURCES
...
cp: cannot stat `/tmp/DISDPKGROOT_gFgoeH/BUILD/dolphin-dx-3.6.1.5-2.6.32-4-pve/opt/DIS/lib64/librmlib*': No such file or directory
mv: cannot stat `/tmp/DISDPKGROOT_gFgoeH/BUILD/dolphin-dx-3.6.1.5-2.6.32-4-pve/opt/DIS/bin/reflective_memory': No such file or directory
mv: cannot stat `/tmp/DISDPKGROOT_gFgoeH/BUILD/dolphin-dx-3.6.1.5-2.6.32-4-pve/opt/DIS/lib64/librmlib*': No such file or directory
Are there other dependencies? Have you changed the package-order?

Can you put the dolphin-supersockets-dx*.deb on a ftp-server, please? (for first testing - of course i need to compile all successful)

Without the supersockets, the cards are useless for drbd...


Regards

Udo
 
I went for the Myricom NICs so I don't remember much about the Dolphin adapters, sorry :(. Neither did I keep the binaries - they went away with a clean install of proxmox after my testing :(.
What I have is a "DIS_DX_install_DIS_RELEASE_3_6_1_2_AUGUST_18_2010.sh" which probably was the last one I tried. Don't know if that is still downloadable from Dolphinics? Otherwise you
could try to contact Dolphinics. They were very interested in helping when I tested.
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!