Poor fsyncs/second with mdadm raid

[blargling]

I'm noticing extremely slow write speeds within my VMs (virtio in on) and pveperf is show poor fsyncs/second. All my VM logical volumes reside on a mdadm software raid 5 consisting of three disks. hdparm is showing good read speeds from each disk and from the raid as a whole. Using dd as a write test to a LV on the raid is showing good write speeds. Any ideas why my fsyncs/second are so low? I certainly expect software raid to not be as fast as hardware of course, but this seems out of the ordinary. Please see below for details:

picard:~# cat /proc/mdstat
Personalities : [raid6] [raid5] [raid4]
md0 : active raid5 sda1[0] sdc1[2] sdb1[1]
1953524736 blocks level 5, 128k chunk, algorithm 2 [3/3] [UUU]

unused devices: <none>

picard:~# pveperf /mnt/test/
CPU BOGOMIPS: 38571.88
REGEX/SECOND: 1165217
HD SIZE: 3.94 GB (/dev/mapper/datavg-perftest)
BUFFERED READS: 114.84 MB/sec
AVERAGE SEEK TIME: 10.40 ms
FSYNCS/SECOND: 32.92
DNS EXT: 202.08 ms
DNS INT: 0.87 ms (snip)

picard:~# dd if=/dev/zero of=/mnt/test/blah bs=4k count=400000
400000+0 records in
400000+0 records out
1638400000 bytes (1.6 GB) copied, 12.8695 s, 127 MB/s

picard:~# hdparm -tT /dev/sdb

/dev/sdb:
Timing cached reads: 6208 MB in 2.00 seconds = 3104.91 MB/sec
Timing buffered disk reads: 270 MB in 3.01 seconds = 89.74 MB/sec
picard:~# hdparm -tT /dev/sdc

/dev/sdc:
Timing cached reads: 7432 MB in 2.00 seconds = 3717.09 MB/sec
Timing buffered disk reads: 268 MB in 3.05 seconds = 87.75 MB/sec

picard:~# hdparm -tT /dev/sdd

/dev/sdd:
Timing cached reads: 6918 MB in 2.00 seconds = 3460.67 MB/sec
Timing buffered disk reads: 170 MB in 3.03 seconds = 56.19 MB/sec

picard:~# hdparm -tT /dev/md0

/dev/md0:
Timing cached reads: 6872 MB in 2.00 seconds = 3437.10 MB/sec
Timing buffered disk reads: 478 MB in 3.02 seconds = 158.29 MB/sec


Thanks,
Kyle

 

[udo]

... Using dd as a write test to a LV on the raid is showing good write speeds. ...
picard:~# dd if=/dev/zero of=/mnt/test/blah bs=4k count=400000
400000+0 records in
400000+0 records out
1638400000 bytes (1.6 GB) copied, 12.8695 s, 127 MB/s

...

Hi,
you measure with internal buffering!
Try following:

dd if=/dev/zero of=/mnt/test/blah bs=4k count=400000 conv=fdatasync

(you can also do a "time (dd if=/dev/zero of=/mnt/test/blah bs=4k count=400000; sync)" but this measure also other buffer flashes...

Udo

 

[blargling]

Here are the results:


picard:~# dd if=/dev/zero of=/mnt/test/blah bs=4k count=400000 conv=fdatasync
400000+0 records in
400000+0 records out
1638400000 bytes (1.6 GB) copied, 19.4839 s, 84.1 MB/s

picard:~# time (dd if=/dev/zero of=/mnt/test/blah bs=4k count=400000; sync)
400000+0 records in
400000+0 records out
1638400000 bytes (1.6 GB) copied, 12.0501 s, 136 MB/s

real 0m18.083s
user 0m0.030s
sys 0m2.010s

 

[vmanz]

Hi blargling,

I know about your fsync/s-problematics.

Using 2.6.24-kernel is fast at mdraid. Have a look at this threat and my tests.

http://forum.proxmox.com/threads/4068-Installguide-proxmox-on-Software-Raid-(Hetzner-EQ6-server)

Espacially my post:
http://forum.proxmox.com/threads/40...e-Raid-(Hetzner-EQ6-server)?p=26412#post26412

I don`t know what has changed in mdraid from 2.6.24 to 2.6.32 and haven`t found any answer. After aptitude install pve-kernel-2.6.24-12-pve don`t forget to remove your /etc/udev/rules.d/70-persistent-net.rules and eventually change your /boot/grub/menu.lst.

Then reboot and please post your results of pveperf (and pveversion -v) . Maybe someone else can find an answer to this behaviour and this issue can be solved for 2.6.3x-kernel.

Good luck,

vmanz

 

[blargling]

vmanz,
Thank you for the suggestion. I tried going down to the 2.6.24 kernel, but unfortunately i'm still getting poor fsyncs/s. I also tried different i/o schedulers for each disk in the raid, but they made little difference.

picard:~# pveversion -v
pve-manager: 1.7-10 (pve-manager/1.7/5323)
running kernel: 2.6.24-12-pve
proxmox-ve-2.6.24: 1.6-26
pve-kernel-2.6.32-4-pve: 2.6.32-30
pve-kernel-2.6.24-12-pve: 2.6.24-25
qemu-server: 1.1-28
pve-firmware: 1.0-10
libpve-storage-perl: 1.0-16
vncterm: 0.9-2
vzctl: 3.0.24-1pve4
vzdump: 1.2-10
vzprocps: 2.0.11-1dso2
vzquota: 3.0.11-1
pve-qemu-kvm: 0.13.0-3
picard:~# pveperf /mnt/test
CPU BOGOMIPS: 38572.27
REGEX/SECOND: 679031
HD SIZE: 3.94 GB (/dev/mapper/datavg-perftest)
BUFFERED READS: 96.92 MB/sec
AVERAGE SEEK TIME: 8.12 ms
FSYNCS/SECOND: 217.55

 

[udo]

vmanz,
Thank you for the suggestion. I tried going down to the 2.6.24 kernel, but unfortunately i'm still getting poor fsyncs/s. I also tried different i/o schedulers for each disk in the raid, but they made little difference.

picard:~# pveversion -v
pve-manager: 1.7-10 (pve-manager/1.7/5323)
running kernel: 2.6.24-12-pve
proxmox-ve-2.6.24: 1.6-26
pve-kernel-2.6.32-4-pve: 2.6.32-30
pve-kernel-2.6.24-12-pve: 2.6.24-25
qemu-server: 1.1-28
pve-firmware: 1.0-10
libpve-storage-perl: 1.0-16
vncterm: 0.9-2
vzctl: 3.0.24-1pve4
vzdump: 1.2-10
vzprocps: 2.0.11-1dso2
vzquota: 3.0.11-1
pve-qemu-kvm: 0.13.0-3
picard:~# pveperf /mnt/test
CPU BOGOMIPS: 38572.27
REGEX/SECOND: 679031
HD SIZE: 3.94 GB (/dev/mapper/datavg-perftest)
BUFFERED READS: 96.92 MB/sec
AVERAGE SEEK TIME: 8.12 ms
FSYNCS/SECOND: 217.55

Hi,
217 fsyncs/s are not too bad (i guess). How much do you expect with softwareraid?
You can't compare this with an hardware raidcontroller.

Udo

 

[vmanz]

Hmm,

I have no experience with other md-raid-levels other than raid1. But I think you should get more out of it.

Did you check your drives with a vendor-tool? When I got weak fsyncs/s (with 2.6.24-12) there also was trouble with a disk.

Two weeks before I bought a Controller (HP P212 with cache and enabled), but the performance is lower than mdraid-1 on 2.6.24.
I think it`s an effect of getting data from 2 disks at same time on software raid

My Example is a HP ML110 G6, Xeon 3430 each with two Seagate 750NS:

Controller-RAID1:
vhost01:~# pveversion -v
pve-manager: 1.7-10 (pve-manager/1.7/5323)
running kernel: 2.6.32-4-pve
proxmox-ve-2.6.32: 1.7-30
pve-kernel-2.6.32-4-pve: 2.6.32-30
qemu-server: 1.1-28
pve-firmware: 1.0-10
libpve-storage-perl: 1.0-16
vncterm: 0.9-2
vzctl: 3.0.24-1pve4
vzdump: 1.2-10
vzprocps: 2.0.11-1dso2
vzquota: 3.0.11-1
pve-qemu-kvm: 0.13.0-3
ksm-control-daemon: 1.0-4

vhost01:~# pveperf
CPU BOGOMIPS: 19151.29
REGEX/SECOND: 930443
HD SIZE: 19.69 GB (/dev/mapper/pve-root)
BUFFERED READS: 65.32 MB/sec
AVERAGE SEEK TIME: 8.05 ms
FSYNCS/SECOND: 2277.22
DNS EXT: 122.31 ms
DNS INT: 48.95 ms


MD-RAID:
vhost01:~# pveversion -v
pve-manager: 1.7-10 (pve-manager/1.7/5323)
running kernel: 2.6.24-12-pve
proxmox-ve-2.6.24: 1.6-26
pve-kernel-2.6.24-12-pve: 2.6.24-25
qemu-server: 1.1-28
pve-firmware: 1.0-10
libpve-storage-perl: 1.0-16
vncterm: 0.9-2
vzctl: 3.0.24-1pve4
vzdump: 1.2-10
vzprocps: 2.0.11-1dso2
vzquota: 3.0.11-1
pve-qemu-kvm: 0.13.0-3

vhost01:~# pveperf
CPU BOGOMIPS: 19151.19
REGEX/SECOND: 847970
HD SIZE: 93.08 GB (/dev/mapper/pve-root)
BUFFERED READS: 51.41 MB/sec
AVERAGE SEEK TIME: 11.07 ms
FSYNCS/SECOND: 3694.63
DNS EXT: 51.34 ms
DNS INT: 31.79 ms

hope it helps,

vmanz

 

[blargling]

I wanted to follow up with my quasi-resolution in case any else comes across this. I could never get mdadm raid 5 to behave with kvm and write at a decent speed. Outside kvm on the host the raid 5 gave great write speeds, but within a guest it was always poor. I have since switched to a mdadm raid10 setup and it is giving spectacular write performance within the guests.