Disappointing performance for DB in KVM with BBU RAID10 over SAS 15K disks and Virtio

Y

yatesco

Well-Known Member
Sep 25, 2009
211
5
58
Hi all,

I have a HP GL380 G6 with RAID10 (including the battery addition) with write caching enabled over 4x300 15K SAS disks.

I have a KVM machine running Windows 7/SQLServer2005 with 2GB RAM and 1 CPU using the latest virtio drivers on a .18 kernel (also tried the .32 kernel to no effect).

The machine absolutely flies when doing long reads or long writes, but for lots of small reads/writes the performance in the KVM machine is terrible.

To put it into context, I have a java app which reads a couple of rows from a DB, does some stuff and then writes it to another DB - i.e. lots and lots of tiny reads and writes. If I run this on my Snow Leopard laptop running XP on VMWare desktop it runs over twice as fast on the writes then it does on the KVM virtual machine (windows 7). Both VMWare and the KVM box have 2G RAM, 1 vCPU and the drivers (Vmware tools and VIRTIO) installed. Both machines also have 'write cache' enabled on the guest (Win XP calls this 'configured for performance' I think).

The KVM machine absolutely flies through the reads but then lags way behind the Vmware on my crappy little laptop on the writes.

I even tried using tmpfs and creating a RAW file on that, but I see the same speed difference, which obviously means this is nothing to do with the RAID card, but I just out of options. I tried cache=none, and also tried the .32 and the .18 kernel but that didn't have any efect.

The only discrepency is that sdparm reports that WCE isn't enabled, but it clearly is - the FSYNCs indicate that as does the HP raid utilities....

Any pointers gratefully received.....

(the following is done whilst the server is running two KVM machines - the disk IO is usually better)

Code: 
pveperf
CPU BOGOMIPS:      72537.63
REGEX/SECOND:      560094
HD SIZE:           94.49 GB (/dev/pve/root)
BUFFERED READS:    240.90 MB/sec
AVERAGE SEEK TIME: 5.17 ms
FSYNCS/SECOND:     1419.94
DNS EXT:           36.65 ms
DNS INT:           34.14 ms

Code: 
hdparm -tT /dev/cciss/c0d0

/dev/cciss/c0d0:
 Timing cached reads:   8008 MB in  2.00 seconds = 4008.15 MB/sec
 Timing buffered disk reads:  650 MB in  3.00 seconds = 216.34 MB/sec

Code: 
sdparm /dev/cciss/c0d0
    /dev/cciss/c0d0: HP        LOGICAL VOLUME    2.50
Read write error recovery mode page:
  AWRE        1  [cha: n, def:  1]
  ARRE        1  [cha: n, def:  1]
  PER         0  [cha: n, def:  0]
Caching (SBC) mode page:
  WCE         0  [cha: n, def:  0]
  RCD         0  [cha: n, def:  0]
Control mode page:
  SWP         0  [cha: n, def:  0]

Trying to set WCE fails, but it is definitely enabled (according to the HP utilities).

Code: 
sdparm --set=WCE /dev/cciss/c0d0
    /dev/cciss/c0d0: HP        LOGICAL VOLUME    2.50
mode select (10): transport: Host_status=0x0a [DID_PASSTHROUGH]
Driver_status=0x00 [DRIVER_OK, SUGGEST_OK]

change_mode_page: failed setting page: Caching (SBC)

Code: 
=> controller slot=0 show config detail

Smart Array P410i in Slot 0 (Embedded)
   Bus Interface: PCI
   Slot: 0
   Serial Number: 50123456789ABCDE
   Cache Serial Number: PAAVP9SYIFWF
   RAID 6 (ADG) Status: Disabled
   Controller Status: OK
   Chassis Slot: 
   Hardware Revision: Rev C
   Firmware Version: 2.50
   Rebuild Priority: Medium
   Expand Priority: Medium
   Surface Scan Delay: 3 secs
   Queue Depth: Automatic
   Monitor and Performance Delay: 60 min
   Elevator Sort: Enabled
   Degraded Performance Optimization: Disabled
   Inconsistency Repair Policy: Disabled
   Post Prompt Timeout: 15 secs
   Cache Board Present: True
   Cache Status: OK
   Accelerator Ratio: 25% Read / 75% Write
   Drive Write Cache: Enabled
   Total Cache Size: 512 MB
   No-Battery Write Cache: Enabled
   Cache Backup Power Source: Batteries
   Battery/Capacitor Count: 1
   Battery/Capacitor Status: OK
   SATA NCQ Supported: True

   Array: A
      Interface Type: SAS
      Unused Space: 0 MB
      Status: OK

      Logical Drive: 1
         Size: 558.7 GB
         Fault Tolerance: RAID 1+0
         Heads: 255
         Sectors Per Track: 32
         Cylinders: 65535
         Stripe Size: 128 KB
         Status: OK
         Array Accelerator: Enabled
         Unique Identifier: 600508B1001037383941424344450400
         Disk Name: /dev/cciss/c0d0
         Mount Points: /boot 512 MB
         Logical Drive Label: AB3DA22250123456789ABCDE2B86
         Mirror Group 0:
            physicaldrive 1I:1:1 (port 1I:box 1:bay 1, SAS, 300 GB, OK)
            physicaldrive 1I:1:2 (port 1I:box 1:bay 2, SAS, 300 GB, OK)
         Mirror Group 1:
            physicaldrive 1I:1:3 (port 1I:box 1:bay 3, SAS, 300 GB, OK)
            physicaldrive 1I:1:4 (port 1I:box 1:bay 4, SAS, 300 GB, OK)

      physicaldrive 1I:1:1
         Port: 1I
         Box: 1
         Bay: 1
         Status: OK
         Drive Type: Data Drive
         Interface Type: SAS
         Size: 300 GB
         Rotational Speed: 10000
         Firmware Revision: HPDC
         Serial Number: 3SE199ED00009021NAZM
         Model: HP      DG0300FAMWN     
         PHY Count: 2
         PHY Transfer Rate: 6.0GBPS, Unknown
      physicaldrive 1I:1:2
         Port: 1I
         Box: 1
         Bay: 2
         Status: OK
         Drive Type: Data Drive
         Interface Type: SAS
         Size: 300 GB
         Rotational Speed: 10000
         Firmware Revision: HPDC
         Serial Number: 3SE182N800009020YPF2
         Model: HP      DG0300FAMWN     
         PHY Count: 2
         PHY Transfer Rate: 6.0GBPS, Unknown
      physicaldrive 1I:1:3
         Port: 1I
         Box: 1
         Bay: 3
         Status: OK
         Drive Type: Data Drive
         Interface Type: SAS
         Size: 300 GB
         Rotational Speed: 10000
         Firmware Revision: HPD5
         Serial Number: D001P9B055WU0945
         Model: HP      EG0300FARTT     
         PHY Count: 2
         PHY Transfer Rate: 6.0GBPS, Unknown
      physicaldrive 1I:1:4
         Port: 1I
         Box: 1
         Bay: 4
         Status: OK
         Drive Type: Data Drive
         Interface Type: SAS
         Size: 300 GB
         Rotational Speed: 10000
         Firmware Revision: HPD5
         Serial Number: D001P9B053ET0945
         Model: HP      EG0300FARTT     
         PHY Count: 2
         PHY Transfer Rate: 6.0GBPS, Unknown
 
Re: Disappointing performance for DB in KVM with BBU RAID10 over SAS 15K disks and Vi

yatesco said:
The machine absolutely flies when doing long reads or long writes, but for lots of small reads/writes the performance in the KVM machine is terrible.
Click to expand...

kvm has too much CPU overhead when doing IO, so I guess performance is limited by the CPU (one CORE is used 100% for IO?)
 
Re: Disappointing performance for DB in KVM with BBU RAID10 over SAS 15K disks and Vi

Excellent point - yes it does have 100%. Hmm, how can I solve that though - giving it multiple cores won't help will it?

This actually has 16 cores because of hyper-threading - will turning off hyper-threading speed things up?
 
Re: Disappointing performance for DB in KVM with BBU RAID10 over SAS 15K disks and Vi

yatesco said:
Excellent point - yes it does have 100%. Hmm, how can I solve that though - giving it multiple cores won't help will it?
Click to expand...

I guess the kvm team is already aware of that, and this will get better with newer kvm releases. giving it multiple cores does not help, since IO only use one thread.

yatesco said:
This actually has 16 cores because of hyper-threading - will turning off hyper-threading speed things up?
Click to expand...

maybe - just try.
 
Re: Disappointing performance for DB in KVM with BBU RAID10 over SAS 15K disks and Vi

Which version of KVM is included in the 2.6.18 kernel? Upgrading to .32 doesn't seem to help any.

Would running against the proxmox testing help?
 
Re: Disappointing performance for DB in KVM with BBU RAID10 over SAS 15K disks and Vi

yatesco said:
Which version of KVM is included in the 2.6.18 kernel?
Click to expand...

kvm-83 with rh patches.

yatesco said:
Upgrading to .32 doesn't seem to help any.

Would running against the proxmox testing help?
Click to expand...

no.
 
Re: Disappointing performance for DB in KVM with BBU RAID10 over SAS 15K disks and Vi

Hello Guys,

i had the same problem with VMware Server 2.0 based on CentOS and XenServer from Citrix.

Change your Hardware to a Dell or Fujitsu and everything will be fine. The only problem is the Raid-Controller from HP. It has the badest perfomance I've ever seen. The company I recognised it, changed the Raid-Controller to LSI and the performance was very good.

Cheers
macday
 
You must log in or register to reply here.
Top Bottom