rsync very slow

[tekwize]

Hi All
We are running proxmox 1.7 on linux kernel 2.6.32 on a system with 4x4 core CPU's
all is fine, but running rsync to copy files from any disk to any disk (including just a file copy to the same partition) causes the transfer rate to slow down in the first 30 to 40 seconds, and settle to around 1 to 2 Mbytes per second, even dropping down to a few hundred Kbytes per sec at times ??
Even more bizzare, we do not see this problem on any of the virtual systems installed on this server ?
We have tried installing the same version of proxmox on another single core single cpu system and no problems at all with rsync ?
As we use rsync as part of our backup procedures, this is causing the backups to run forever, and at times need to be killed.
doing a cp or dd of the same backup files completes in about 10 seconds ??

rsync is version 3.0.7 protocol version 30.

pveperf
CPU BOGOMIPS: 76606.35
REGEX/SECOND: 829986
HD SIZE: 29.53 GB (/dev/mapper/pve-root)
BUFFERED READS: 252.06 MB/sec
AVERAGE SEEK TIME: 4.47 ms
FSYNCS/SECOND: 2635.50
DNS EXT: 282.21 ms
DNS INT: 0.88 ms

Any assistance, advice etc really appreciated.
Thanks
John.

Update - Managed to test rsync on similar hardware as above but without a raid controller card and its fine !
Now we suspect problem is related to the raid controller.
Will need to do more tests, but anyone had a similar experience ?

 

[mikero]

I'm extremely curious about your problem - we've seen something very similar. Were you by any chance using a 3ware 9750-4i raid controller?

 

[tekwize]

No, we have an Adaptec 5405, had a look at the specs for both cards and looks like different chip sets ?

 

[mikero]

yeah - entirely different. Interesting. In that case the only other thing i am interested in if you dont mind is the kernel you are on. We're on a lucid ubuntu box, but with a maverick kernel 2.6.35-25-server.

Thanks for your help.

 

[tekwize]

Were running debian 2.6.32-4-pve

 

[mdo]

In addition to tekwize's details (my colleague), I have done more testing on this issue (which seems to be an rsync only problem) over the weekend and added details to another performance related post which seems to indicate similar issues:
http://forum.proxmox.com/threads/5539-Big-performance-problem?p=34233#post34233

I am thinking it's hardware related, either the (existence of) Adaptec controller maybe in combination with the E5620 Xaons? Although this sounds very strange I admit.