Mysql performance / pveperf / lvm / fsyncs per second

[jamesd256]

Hi,

Our production server seems to have some Mysql performance issues since migrating to Proxmox 1.8, Kernel 2.6.32. Mysql is version 5, and we have a database with millions of records in innodb with file per table.

As you can see here, our raid 5 (SAS) array is apparently not performing as it should do, particularly for fsyncs/second:

CPU BOGOMIPS: 85119.26
REGEX/SECOND: 986372
HD SIZE: 66.93 GB (/dev/mapper/pve-root)
BUFFERED READS: 49.66 MB/sec
AVERAGE SEEK TIME: 10.79 ms
FSYNCS/SECOND: 4.92
DNS EXT: 140.23 ms

I read somewhere that pveperf does not support LVM, yet the default install of Proxmox uses LVM on the root partition, which is what we have, so this doesn't make too much sense to me.

Can pveperf be relied upon when the root partition is LVM? If it is a reiliable measure, then has anyone seen such a remarkably low value for fsyncs per seconds, and if so, how did you fix it?

Thanks,

James

 

[dietmar]

Can pveperf be relied upon when the root partition is LVM?

yes

If it is a reiliable measure, then has anyone seen such a remarkably low value for fsyncs per seconds, and if so, how did you fix it?

Most likely your RAID controller cache settings are wrong. Please check that you run with 'write back' mode. You need a BBU for that, and the battery status needs to be OK.

 

[jamesd256]

Thanks for the reply Dietmar.

The write policy is write back, and the read policy is adaptive read ahead. The battery status is 'ok'.

hdparm tells us we're getting 300Mb's read (timing buffered disk reads)

Strange, huh? It's a vanilla Dell Poweredge i710 with a standard Proxmox install. Any ideas or suggestions gratefully received.

 

[dietmar]

What filesystem do you use? This looks like a standard installation, so I assume it is ext3?

 

[dietmar]

I assume there is no load on the server when you run pveperf?

 

[jamesd256]

What filesystem do you use? This looks like a standard installation, so I assume it is ext3?

Yes, it's ext3. We went over to reiser on our failover environment, and the performance was similarly bad.

 

[jamesd256]

I assume there is no load on the server when you run pveperf?

There is load, because it's a production server. The load average is only 1.4 though, and hdparm can still achieve 300mb/s, so I would be surprised if load caused our fsync performance to be orders of magnitude below what I would expect.

Another update, we have a data partition on another lvm partition, and running pveperf against that gave us 5000 fsyncs a second, but only 5mb/s buffered reads.

I understand load will skew the results, but these numbers seem wild.

 

[dietmar]

There is load, because it's a production server. The load average is only 1.4 though, and hdparm can still achieve 300mb/s, so I would be surprised if load caused our fsync performance to be orders of magnitude below what I would expect.

pveperf does not provide accurate results when other things use IO.

Another update, we have a data partition on another lvm partition, and running pveperf against that gave us 5000 fsyncs a second, but only 5mb/s buffered reads.

I understand load will skew the results, but these numbers seem wild.

Again, pveperf only gives accurate result when there is no load on the server.