Very low disk performance

[Mordred]

Hello,
We have a supermicro server with motherboard X9SRI-F and installed two SATA drivers, 4Tb each (WD4000FYYZ). All HDD plug in SATA 3.0 ports. CPU: E5-2609v2. Version of Proxmox is 4.2-17/e1400248. At host installed 5 virtual machines ( 2 Unix, 3 WinSRV 2012R2). Only 3 are started.
IO delay almost all time is 80%.
Results of pvepref:
CPU BOGOMIPS: 20001.76
REGEX/SECOND: 1256293
HD SIZE: 94.37 GB (/dev/dm-0)
BUFFERED READS: 16.86 MB/sec
AVERAGE SEEK TIME: 34.98 ms
FSYNCS/SECOND: 6.79
DNS EXT: 81.30 ms
DNS INT: 69.74 ms
Could you please help with such issue?

Very appreciate for your help!

 

[wolfgang]

Hi,

your disk are to slow.

HD SIZE: 94.37 GB (/dev/dm-0)
BUFFERED READS: 16.86 MB/sec
AVERAGE SEEK TIME: 34.98 ms
FSYNCS/SECOND: 6.79

 

[Mordred]

Hi,

your disk are to slow.

Hello. It's not slow disk, as I understand. I don't know why, but it's proxmox think that it's slow. If i will use them at another server - everything fine.
As you can see at description of such HDD, it's 7200 rpm, cache 64 Mb. At another Proxmox server, I have the same disk - IO delay not high than 3-4% at maximum.

 

[wolfgang]

What does your VM do?
What io load they produce?

you can kill such disk very easy with one SQLDB.
And you have 3 VM running.

 

[Mordred]

What does your VM do?
What io load they produce?

you can kill such disk very easy with one SQLDB.
And you have 3 VM running.

One VM - EXIM, another - Windows Server 2012R2 with AD, DHCP, DNS, and another - Exchange 2016. At Exchange created 10 mailboxes.

 

[e100]

You should install some tools to help see whats going on.
apt-get install systat

You can then run "iostat 1 -g all"
It will output the IO to each block device every second and the number you are most concerned with is tps as this represents IOPS.
Pay attention to the all line.

Your disks can only do 130-200 IOPS and thats doing only all read or all write. If you are doing a mix of read write expect less.

If you run "iostat 1 -g all | grep all" for a minute or so and post the entire output here I can give you my opinion on what it means.
Just make sure you do this while the IO wait is really high.

 

[Mordred]

You should install some tools to help see whats going on.
apt-get install systat

You can then run "iostat 1 -g all"
It will output the IO to each block device every second and the number you are most concerned with is tps as this represents IOPS.
Pay attention to the all line.

Your disks can only do 130-200 IOPS and thats doing only all read or all write. If you are doing a mix of read write expect less.

If you run "iostat 1 -g all | grep all" for a minute or so and post the entire output here I can give you my opinion on what it means.
Just make sure you do this while the IO wait is really high.

Ok. When it's appear again high delay I'll make. Thank you!

 

[Mordred]

You should install some tools to help see whats going on.
apt-get install systat

You can then run "iostat 1 -g all"
It will output the IO to each block device every second and the number you are most concerned with is tps as this represents IOPS.
Pay attention to the all line.

Your disks can only do 130-200 IOPS and thats doing only all read or all write. If you are doing a mix of read write expect less.

If you run "iostat 1 -g all | grep all" for a minute or so and post the entire output here I can give you my opinion on what it means.
Just make sure you do this while the IO wait is really high.

Hi!
Here is result of iostat:
iostat 1 -g all | grep all
all 1535.06 41604.59 3388.42 20972043 1708033
all 2257.00 63520.00 9048.00 63520 9048
all 1018.00 48271.00 1464.00 48271 1464
all 2021.00 44232.00 6504.00 44232 6504
all 2896.00 34910.50 10468.00 34910 10468
all 2742.00 59415.00 10096.00 59415 10096
all 1529.00 48564.00 4988.00 48564 4988
all 3846.00 51994.50 16688.00 51994 16688
all 1817.00 47524.00 5552.00 47524 5552
all 811.00 60352.00 224.00 60352 224
all 3728.00 36163.50 15796.00 36163 15796
all 1947.00 40802.00 5676.00 40802 5676
all 4757.00 33287.00 20948.00 33287 20948
all 1331.00 3396.00 4100.00 3396 4100
all 1431.00 24196.00 4152.00 24196 4152
all 3505.00 28548.00 14976.00 28548 14976
all 1216.00 14020.00 3356.00 14020 3356
all 1638.00 20246.50 4916.00 20246 4916
all 1876.00 29472.00 6648.00 29472 6648
all 6480.00 27640.00 31144.00 27640 31144
all 1640.00 18829.50 6072.00 18829 6072
all 962.00 20896.00 712.00 20896 712
all 1542.00 18823.00 4760.00 18823 4760
all 2079.00 31728.00 7080.00 31728 7080
all 926.00 69068.00 272.00 69068 272
all 1889.00 38656.00 7208.00 38656 7208
all 1725.00 23956.00 5872.00 23956 5872
all 1252.00 28212.50 2928.00 28212 2928
all 1858.00 19540.00 6440.00 19540 6440
all 1909.00 7166.50 6940.00 7166 6940
all 830.00 10764.00 1524.00 10764 1524
all 1429.00 13048.00 3840.00 13048 3840
all 2531.00 16268.00 11696.00 16268 11696
all 1176.00 23292.00 3344.00 23292 3344
all 1111.00 35092.00 832.00 35092 832
all 1932.00 17888.00 6744.00 17888 6744
all 1737.00 19304.00 5024.00 19304 5024
all 1849.00 17854.50 5696.00 17854 5696
all 1644.00 15387.00 5872.00 15387 5872
all 2288.00 18521.50 7936.00 18521 7936
all 11158.00 12768.00 57088.00 12768 57088
all 2687.00 16388.00 10816.00 16388 10816
all 1306.00 14100.00 3888.00 14100 3888
all 1376.00 9436.00 4168.00 9436 4168
all 1790.00 29488.00 4384.00 29488 4384
all 7898.00 16920.00 36888.00 16920 36888
all 3804.00 11890.50 18904.00 11890 18904
all 2406.00 13548.50 8360.00 13548 8360
all 847.00 2848.00 1176.00 2848 1176
all 643.00 2800.00 272.00 2800 272
all 4383.00 30164.00 15680.00 30164 15680
all 20668.00 16768.00 89488.00 16768 89488
all 29973.00 9416.00 154568.00 9416 154568
all 8959.00 12793.00 62432.00 12793 62432
all 9773.00 4080.00 50352.00 4080 50352
all 12376.00 8880.00 63560.00 8880 63560
all 18310.00 3578.50 95720.00 3578 95720
all 6694.00 8192.50 31928.00 8192 31928
all 15035.00 6980.00 78512.00 6980 78512
all 8508.00 5628.00 43776.00 5628 43776
all 20190.00 9312.00 103880.00 9312 103880
all 84891.00 6848.00 389132.00 6848 389132
all 126178.00 0.00 671824.00 0 671824
all 17468.00 308.00 154260.00 308 154260
all 28666.00 1716.00 150440.00 1716 150440
all 86369.00 10176.00 431032.00 10176 431032
all 1460.00 4354.50 32904.00 4354 32904
all 3357.00 25512.50 13400.00 25512 13400
all 555.00 28640.00 992.00 28640 992
all 719.00 8360.00 1760.00 8360 1760
all 319.00 7072.00 640.00 7072 640
all 243.00 7264.00 432.00 7264 432
all 880.00 13832.00 3392.00 13832 3392
all 147.00 8260.00 64.00 8260 64
all 191.00 7044.00 464.00 7044 464
all 232.00 13760.00 312.00 13760 312
all 588.00 7435.00 640.00 7435 640
all 113.00 7040.00 200.00 7040 200
all 175.00 8608.00 360.00 8608 360
all 218.00 7288.00 288.00 7288 288
all 256.00 10864.00 560.00 10864 560
all 110.00 10880.00 16.00 10880 16
all 136.00 7056.00 304.00 7056 304
all 136.00 5952.00 320.00 5952 320
all 259.00 11680.00 672.00 11680 672
all 923.00 8512.00 4232.00 8512 4232
all 314.00 5280.00 1024.00 5280 1024
all 1114.00 11403.00 3080.00 11403 3080
all 1168.00 33636.00 176.00 33636 176
all 369.00 8724.00 768.00 8724 768
all 221.00 10912.00 480.00 10912 480
all 689.00 5776.00 3200.00 5776 3200
all 145.00 10288.00 136.00 10288 136
all 289.00 9632.00 792.00 9632 792
all 176.00 7840.00 288.00 7840 288
all 246.00 6496.00 664.00 6496 664
all 227.00 9142.50 32.00 9142 32
all 2281.00 11844.50 8624.00 11844 8624
all 169.00 7640.00 304.00 7640 304
all 362.00 12024.00 880.00 12024 880
all 151.00 6960.00 208.00 6960 208
all 284.00 9888.00 816.00 9888 816
all 695.00 11072.00 2976.00 11072 2976
all 154.00 9680.00 112.00 9680 112
all 152.00 9568.00 224.00 9568 224
all 209.00 10848.00 408.00 10848 408
all 198.00 5872.00 576.00 5872 576
all 434.00 2059.00 56.00 2059 56
all 368.00 2656.00 840.00 2656 840
all 477.00 3296.00 1760.00 3296 1760
all 117.00 16.00 480.00 16 480
all 133.00 224.00 384.00 224 384
all 125.00 0.00 496.00 0 496
all 375.00 16.00 1824.00 16 1824
all 149.00 96.00 576.00 96 576
all 144.00 2576.00 344.00 2576 344
all 227.00 40.00 896.00 40 896
all 1253.00 2275.00 3752.00 2275 3752
all 267.00 16.00 832.00 16 832
all 333.00 174.00 742.00 174 742
all 1015.00 8416.00 20651.00 8416 20651
all 2889.00 75119.00 76326.00 75119 76326
all 1241.00 20352.00 63077.00 20352 63077
all 355.00 97.00 1672.00 97 1672
all 309.00 32.00 1280.00 32 1280
all 355.00 40.00 2536.00 40 2536
all 611.00 848.00 10272.00 848 10272
all 543.00 547.00 16760.00 547 16760
all 612.00 16.00 123664.00 16 123664
all 289.00 32.00 15296.00 32 15296
all 122.00 0.00 12608.00 0 12608
all 1224.00 32.00 141248.00 32 141248
all 1411.00 29464.00 14052.00 29464 14052
all 1098.00 85760.00 3212.00 85760 3212
all 803.00 82176.00 1168.00 82176 1168

 

[RobFantini]

Hello, while waiting for e100 [ who knows a lot more then I about iostat and other advanced disk topics ] , you may want to try using smartmontools . I have used that [ and zabbix ] to find disks with issues. here are rough notes on usage.

apt-get install smartmontools

#  good for getting serial numbers :
smartctl -a /dev/sda | less

note the serial number is used in  /dev/disk/by-id/

 == smartd ==
http://www.techrepublic.com/blog/linux-and-open-source/using-smartctl-to-get-smart-status-information-on-your-hard-drives/
Finally, smartctl can be used to initiate long and short tests for the drive. These should be run periodically to do quick, or full, self-tests of the drive:
# smartctl --test=short /dev/sda
# smartctl --test=long /dev/sda
# smartctl -a /dev/sda

The above will first perform a short test of the /dev/sda device. This usually takes about a minute to perform, and the smartctl output will tell you when you can check the results. Next, the long test: this one can take quite a bit longer (about two hours here on a 320GB SATA drive). Finally, use the -a option to view the results.

 

[e100]

What cache mode are you using for the virtual disks?

Also please re-run the test but change all to sd so we can see whats going on with each individual disk.
iostat 1 | grep sd"

Do you only have these two 4TB disks in the server?
If you have others please let me know what disks are the 4TB disks such as sda/sdb/sdc/etc

 

[remark]

Here is topic discussed today:
https://forum.proxmox.com/threads/slow-performance-and-low-fsyncs-rate.28635/
Maybe it can help.

Also remember to switch SATA controller to AHCI mode.