rsync (host) performance issues with kernel 2.6.32 on Supermicro platform

M

mdo

Renowned Member
Proxmox Subscriber
Dec 5, 2010
50
9
73
New Zealand
Running latest Proxmox 1.8 (we started with 1.7 on that system) we find unbelievable bad performance when using rsync at the host.

We are running kernel 2.6.32-4.

pveversion -v
pve-manager: 1.8-15 (pve-manager/1.8/5754)
running kernel: 2.6.32-4-pve
proxmox-ve-2.6.32: 1.8-32
pve-kernel-2.6.32-4-pve: 2.6.32-32
qemu-server: 1.1-30
pve-firmware: 1.0-11
libpve-storage-perl: 1.0-16
vncterm: 0.9-2
vzctl: 3.0.24-1pve4
vzdump: 1.2-11
vzprocps: 2.0.11-1dso2
vzquota: 3.0.11-1
pve-qemu-kvm: 0.14.0-2
ksm-control-daemon: 1.0-5

The system in use is a Supermicro "X8DTI-F" (or X8DT3?) board (very popular I think), 2 x Intel E5620, 12 GB RAM + Adaptec 5405 hardware RAID controller, 2 x 300GB SAS plus 2 x 1TB Sata as RAID1 each.

We have actually 3 of these systems running (one still Proxmox 1.7) and the problem is consistent. On the other hand, the performance for rsync is GOOD on an alternative testing system which is a lower end board, running an Intel Xeon X3430 processor and no hardware RAID controller, just Sata disks using the onboard controller.

For testing on the 'problem' platform, we swapped RAM and even removed the Adaptec controller but all that made no real difference so we have to believe it is a motherboard/kernel 2.6.32 related problem. Help from Super Micro through our hardware supplier is basically Zero. I suspect this is as soon as they hear Debian rather a commercial product.

We have tested one of the 'problem systems' with kernels 2.6.18 and 2.6.35 and the performance with 2.6.18 kernel was GOOD (!) 2.6.35 bad, only slightly better than 2.6.32 (it did not drop below 2MB throughput - but still very poor)

Below are details of both systems, performance data and testing results.

A) The 'problem' system:

pveversion -v
pve-manager: 1.8-15 (pve-manager/1.8/5754)
running kernel: 2.6.32-4-pve
proxmox-ve-2.6.32: 1.8-32
pve-kernel-2.6.32-4-pve: 2.6.32-32
qemu-server: 1.1-30
pve-firmware: 1.0-11
libpve-storage-perl: 1.0-16
vncterm: 0.9-2
vzctl: 3.0.24-1pve4
vzdump: 1.2-11
vzprocps: 2.0.11-1dso2
vzquota: 3.0.11-1
pve-qemu-kvm: 0.14.0-2
ksm-control-daemon: 1.0-5

pveperf /var/lib/vz
CPU BOGOMIPS: 76606.83
REGEX/SECOND: 867968
HD SIZE: 367.63 GB (/dev/mapper/pve-data)
BUFFERED READS: 226.38 MB/sec
AVERAGE SEEK TIME: 4.80 ms
FSYNCS/SECOND: 2912.92
DNS EXT: 257.44 ms
DNS INT: 0.91 ms (x.y.z)

Create an 8 GB test file (on the SAS disk):
dd if=/dev/zero of=/var/lib/vz/testfile_MDO bs=1024k count=8192 conv=fdatasync
8192+0 records in
8192+0 records out
8589934592 bytes (8.6 GB) copied, 71.4297 s, 120 MB/s

Rsync on SAS drives (aborted after several minutes due to slow speed):
rsync --progress /var/lib/vz/testfile_MDO /var/lib/vz/testfile_MDO2
testfile_MDO
592379904 6% 949.34kB/s 2:20:24 ^C
rsync error: received SIGINT, SIGTERM, or SIGHUP (code 20) at rsync.c(543) [sender=3.0.7]

Copy file on the SAS disk:
twt250sv:/var/lib/vz# date; cp testfile_MDO testfile_MDO2 ; date
Sun Apr 17 09:34:45 NZST 2011
Sun Apr 17 09:36:28 NZST 2011 = 103 secs

Copy file from SAS to Sata:
twt250sv:/backup# date; cp /var/lib/vz/testfile_MDO /backup/testfile_MDO ; date
Sun Apr 17 09:42:13 NZST 2011
Sun Apr 17 09:43:54 NZST 2011 = 101 secs

Copy file on Sata disk:
twt250sv:/backup# date; cp testfile_MDO testfile_MDO2 ; date
Sun Apr 17 09:45:30 NZST 2011
Sun Apr 17 09:48:58 NZST 2011 = 208 secs

Performance on the Sata disks for this system:

pveperf /backup/
CPU BOGOMIPS: 76606.83
REGEX/SECOND: 907656
HD SIZE: 458.02 GB (/dev/sdb2)
BUFFERED READS: 160.46 MB/sec
AVERAGE SEEK TIME: 9.71 ms
FSYNCS/SECOND: 3294.10
DNS EXT: 257.54 ms
DNS INT: 0.88 ms (x.y.nz)


Create 8 GB file; 81MB/s on Sata as compared to 120MB on SAS:
dd if=/dev/zero of=/backup/testfile_MDO bs=1024k count=8192 conv=fdatasync
8192+0 records in
8192+0 records out
8589934592 bytes (8.6 GB) copied, 105.088 s, 81.7 MB/s

Rsync on Sata (as bad as SAS), aborted.
rsync --progress /backup/testfile_MDO /backup/testfile_MDO2
testfile_MDO
2370535424 27% 556.12kB/s 3:06:23 ^C
rsync error: received SIGINT, SIGTERM, or SIGHUP (code 20) at rsync.c(543) [sender=3.0.7]


B) The low specd test system, Sata only:

pveversion -v
pve-manager: 1.8-15 (pve-manager/1.8/5754)
running kernel: 2.6.32-4-pve
proxmox-ve-2.6.32: 1.8-32
pve-kernel-2.6.32-4-pve: 2.6.32-32
qemu-server: 1.1-30
pve-firmware: 1.0-11
libpve-storage-perl: 1.0-16
vncterm: 0.9-2
vzctl: 3.0.24-1pve4
vzdump: 1.2-11
vzprocps: 2.0.11-1dso2
vzquota: 3.0.11-1
pve-qemu-kvm: 0.14.0-2
ksm-control-daemon: 1.0-5


pveperf /var/lib/vz
CPU BOGOMIPS: 19153.26
REGEX/SECOND: 770841
HD SIZE: 879.00 GB (/dev/mapper/pve-data)
BUFFERED READS: 118.36 MB/sec
AVERAGE SEEK TIME: 14.04 ms
FSYNCS/SECOND: 1375.59
DNS EXT: 277.88 ms
DNS INT: 0.90 ms (tw.co.nz)

Create 8 MB file - speed slightly better than Sata on higher end system!?

dd if=/dev/zero of=/var/lib/vz/testfile_MDO bs=1024k count=8192 conv=fdatasync
8192+0 records in
8192+0 records out
8589934592 bytes (8.6 GB) copied, 93.151 s, 92.2 MB/s

Successful rsync on test system - speed considered good compared with above.

rsync --progress /var/lib/vz/testfile_MDO /var/lib/vz/testfile_MDO2
testfile_MDO
8589934592 100% 47.32MB/s 0:02:53 (xfer#1, to-check=0/1)

sent 8590983242 bytes received 31 bytes 49515753.73 bytes/sec
total size is 8589934592 speedup is 1.00

Copy on test system Sata disk is faster than Sata copy on better system?

twt251sv:/var/lib/vz# date; cp testfile_MDO2 testfile_MDO3 ; date
Sun Apr 17 09:33:50 NZST 2011
Sun Apr 17 09:36:39 NZST 2011 = 169 secs

C) Tests inside a VM (CentOS 5.5) on the better system - rsync performance OK.

[root@tga1 tmp]# dd if=/dev/zero of=/tmp/testfile_MDO bs=1024k count=8192 conv=fdatasync
8192+0 records in
8192+0 records out
8589934592 bytes (8.6 GB) copied, 101.55 seconds, 84.6 MB/s
[root@tga1 tmp]#

[root@tga1 tmp]# rsync --progress testfile_MDO testfile_MDO2
testfile_MDO
8589934592 100% 36.98MB/s 0:03:41 (xfer#1, to-check=0/1)

Sorry for the long post. I would like some help otherwise it might help someone else.

At this stage I have to say (at least for us) to avoid that Super Micro board.

Michael
 
Hi,
i have also supermicro boards (but meanly amd-based) and the performance is ok (some other things in relation to supermicro not).
Is it right, that at all your meassurings the sas/sata drives connected on the adaptec raidcontroller?
You dd- and pveperf-values looks ok for me. raid-1 bring no real performance boost against one disk.
To measure the time for copy is not very usefull, because you meassure buffering.

What happens, if you use two disk which are directly connected to the mainboard?

The rsyncs are all local? Or do you have tried another NIC?

Have you tried to change bios-settings (ECC-Ram...)?

What shows top during slow rsync (idle/waid/system)? And what shows "iostat -dm 5 sda" - or sdb or whatelse? During rsync and after that. Perhaps other processes use the disk also.
You get iostat with "apt-get install sysstat".

Udo
 
udo said:
Hi,
Is it right, that at all your meassurings the sas/sata drives connected on the adaptec raidcontroller?
Click to expand...
Yes, all the details in that posting were with SAS and Sata using the controller but we had even done tests at some stage with a Sata disk connected straight to the board and the Adaptec controller completely removed for that time. Sadly the performance was as bad as before.

raid-1 bring no real performance boost against one disk.
Click to expand...
I know, its not for performance but safety.

What happens, if you use two disk which are directly connected to the mainboard?
Click to expand...
We have connected one disk at some stage and ran rsync on that single disk with similar, bad performance.

The rsyncs are all local? Or do you have tried another NIC?
Click to expand...
All rsync results are local. I did want to take any potential network issue out of the picture.

Have you tried to change bios-settings (ECC-Ram...)?
Click to expand...
Yes, tried many changes, one after each other. We even tested with Non-ECC RAM - still no real difference.

What shows top during slow rsync (idle/waid/system)? And what shows "iostat -dm 5 sda" - or sdb or whatelse? During rsync and after that. Perhaps other processes use the disk also.
Click to expand...
Thanks, will try and report back.

Many Thanks for replying. I will come back with more details.
 
Thanks again to Udo for trying to help.

FTR - we have switched to the "Redhat kernel" 2.6.18 and rsync speed (~ 80MB/s) is as expected :

Create an 8 GB test file (on the SAS disk):
dd if=/dev/zero of=/var/lib/vz/testfile_MDO bs=1024k count=8192 conv=fdatasync

rsync --progress /var/lib/vz/testfile_MDO /backup/testfile_MDO2
testfile_MDO
8589934592 100% 80.51MB/s 0:01:41 (xfer#1, to-check=0/1)

sent 8590983242 bytes received 31 bytes 83814470.96 bytes/sec
total size is 8589934592 speedup is 1.00

I have no idea why the combination of "Debian kernel" 2.6.32 (or Ubuntu kernel 2.6.35) and this hardware causes that trouble but Supermicro were hot helpful (could not help) so we had to try our own workaround.
 
FTR:

The problems are solved with (further) changes on the original BIOS settings (CPU related).

Interesting (actually, confusing for me): The combination of "Redhat" kernel + original BIOS settings showed good rsync performance as reported before but it turned out that the VMs created a higher system load than before with the Debian 2.6.32 kernel and we ran into performance issues on the entire server.

At the end the (CPU related) BIOS changes made all the difference and now the "Debian" 2.6.32 kernel (also tested with the "Ubuntu" 2.6.35 kernel) show good rsync performance AND good performance for all VMs. The 8GB testfile rsync takes around 2:10 (rather 1:41 under the Redhat kernel) but that is 'good enough' for the moment.

An 'interesting' experience ....

Michael
 
mdo said:
FTR:

The problems are solved with (further) changes on the original BIOS settings (CPU related).

Interesting (actually, confusing for me): The combination of "Redhat" kernel + original BIOS settings showed good rsync performance as reported before but it turned out that the VMs created a higher system load than before with the Debian 2.6.32 kernel and we ran into performance issues on the entire server.

At the end the (CPU related) BIOS changes made all the difference and now the "Debian" 2.6.32 kernel (also tested with the "Ubuntu" 2.6.35 kernel) show good rsync performance AND good performance for all VMs. The 8GB testfile rsync takes around 2:10 (rather 1:41 under the Redhat kernel) but that is 'good enough' for the moment.

An 'interesting' experience ....

Michael
Click to expand...
Hi,
strange... i'm waiting for more coreboot-support for server-mainboards to avoid such things!

Udo
 
mdo said:
FTR:

The problems are solved with (further) changes on the original BIOS settings (CPU related).

Interesting (actually, confusing for me): The combination of "Redhat" kernel + original BIOS settings showed good rsync performance as reported before but it turned out that the VMs created a higher system load than before with the Debian 2.6.32 kernel and we ran into performance issues on the entire server.

At the end the (CPU related) BIOS changes made all the difference and now the "Debian" 2.6.32 kernel (also tested with the "Ubuntu" 2.6.35 kernel) show good rsync performance AND good performance for all VMs. The 8GB testfile rsync takes around 2:10 (rather 1:41 under the Redhat kernel) but that is 'good enough' for the moment.

An 'interesting' experience ....

Michael
Click to expand...

can you give more details about the needed changes in the BIOS?
 
We have two more, identical servers at customer sites with the same symptoms (slow rsync perf.). I will repeat the changes there with a step by step approach to isolate the fix. Will take a while to get access and will report back then.
Michael
 
thanks, I am very interested to get this.
 
After testing on different systems, it seems to be down to one of the following two BIOS settings:
- "Clock speed spectrum" is now set to 'enabled'
- "NUMA support" is now set to 'disabled'.

I suspect it to be the "NUMA" support setting. There is no help in the BIOS itself about "NUMA" but here is the section from the user guide:

"Select Enabled to enable Non-Uniform Memory Access support for an "NUMAAware"
OS to improve CPU performance. Select Disabled to provide better memory
access for an "non-NUMA" OS. Select NUMA for SLES 11 for better CPU performance
on a SUSE Linux Enterprise Server 11. The options are Enabled, Disabled
and NUMA for SLES11. "

So when using the Debian and Ubuntu kernels with NUMA support previously set to "enabled" made rsync extremely slow but the Redhat kernel (closer to Suse's SLES I suspect) did not have that problem - Debian seems not yet to be a "NUMAaware OS"?

Michael
 
Our kernels are compiled with NUMA support.

# grep NUMA /boot/config-2.6.32-4-pve
CONFIG_NUMA_IRQ_DESC=y
CONFIG_NUMA=y
CONFIG_K8_NUMA=y
CONFIG_X86_64_ACPI_NUMA=y
CONFIG_NUMA_EMU=y
CONFIG_ACPI_NUMA=y
 
pveversion -v
pve-manager: 1.8-18 (pve-manager/1.8/6070)
running kernel: 2.6.32-4-pve
proxmox-ve-2.6.32: 1.8-33
pve-kernel-2.6.32-4-pve: 2.6.32-33
qemu-server: 1.1-30
pve-firmware: 1.0-11
libpve-storage-perl: 1.0-17
vncterm: 0.9-2
vzctl: 3.0.27-1pve1
vzdump: 1.2-13
vzprocps: 2.0.11-2
vzquota: 3.0.11-1
pve-qemu-kvm: 0.14.1-1
ksm-control-daemon: 1.0-6

Testing results below with 8GB file, rsync times, all VMs stopped.

Advanced ACPI Config:
1.) "NUMA support" = Disabled; rsync = 2:13 minutes
2.) "NUMA support" = Enabled; rsync interrupted after about 8 minutes at 87%, still > 4 min estimated to finish at 3.9MB/sec at that time
3.) "NUMA support" set back to Disabled; rsync = 2:09 minutes

This is consistent on three (pretty identical) Supermicro mainboards. Maybe Supermicro has got Disabled/Enabled wrong in their BIOS GUI logic?

Michael
 
You must log in or register to reply here.
Top Bottom