High IO delay on SSD and Low fsync rate

[tytanick]

Hi,
I have one cluster with really low fsync rate and i have some problems with high IO while converting qcow to raw on it. And also with different operations.

Specs:
Block size is 4K , Ext4, problem is also on LVM
05:00.0 RAID bus controller: Hewlett-Packard Company Smart Array G6 controllers (rev 01)
2 x 250GB SSD Crucial MX200
root@node2:~# pveversion
pve-manager/4.2-15/6669ad2c (running kernel: 4.4.10-1-pve)
Intel(R) Xeon(R) CPU E5630 @ 2.53GHz x 16 cores

root@node2:~# pveperf
CPU BOGOMIPS: 81045.44
REGEX/SECOND: 1097411
HD SIZE: 7.01 GB (/dev/dm-0)
BUFFERED READS: 293.48 MB/sec
AVERAGE SEEK TIME: 0.21 ms
FSYNCS/SECOND: 294.96
DNS EXT: 67.27 ms

 

[tytanick]

More screenshots.
Anybodo know why is this happening ?
As i see, 10MB/s makes ssd used at 100%.
Why ?

On node2 there is MOVE DISK from NAS to internal DRBD on ssd.

Node4 wchich is connected with node2 with DRBD

 

[hodo]

Hello,
the mx200 SSDs are consumer grade drives optimized for read workloads. I have a similar fsync performance with 500gb mx100 and mx200 SSDs on my machines.

For better fsync performance you should use datacenter disks like intel dc 3700 or samsung sm863 drives.

 

[tytanick]

its not the case.
i have like 10x better performance on other nodes in 3.2 pve also on mx200 crucial (those disks are enterprise level

 

[hodo]

proxmox 3.2 is using ext3 with barriers=0 mount option.
Proxmox 4 uses Ext4 which defaults to barriers=1 mount option.
With barriers=1 the system has to wait for the sync and the sync performance differs greatly between consumer and datacenter SSDs.

 

[tytanick]

Yeah but this is /dev/sdb1 LVM with no filesystem so it doesnt mount anything right ?
Still high IO.

 

[hodo]

Alright, are using a raid configuration for these drives? You wrote 2x250GB. Can you give more information about the HP Smart array G6 Controller?
In my opinion the buffered reads are low for a mx200 SSD. More like sata2 performance instead of sata3.

 

[tytanick]

Sure.
HP ProLiant DL380 G7
Smart Array P410i ( 05:00.0 RAID bus controller: Hewlett-Packard Company Smart Array G6 controllers (rev 01))
2x 250GB ssd Crucial MX200 in Raid1
Created 2 virtual disks /dev/sda for proxmox and /dev/sdb for drbd

Other servers (but all others are Dell) but with proxmox ve 3.2 also on Crucial mx200 shows this:
root@node1:~# pveperf
CPU BOGOMIPS: 183862.80
REGEX/SECOND: 1257319
HD SIZE: 7.01 GB (/dev/mapper/pve-root)
BUFFERED READS: 412.26 MB/sec
AVERAGE SEEK TIME: 0.13 ms
FSYNCS/SECOND: 4410.59
DNS EXT: 10.21 ms

 

[tom]

Crucial mx200 are cheap (pro)consumer SSDs, not suited for high load and DRBD.

if you want performance, go for Intel Datacenter SSDs, best with NMVe.

 

[sumsum]

The Smart Array P410i is known as a low Performance on Board Controller. Not recommended for SSD as far as my experiences goes with this Controller. Try to get a 420 or higher version.

 

[hodo]

Okay, i think the 4410 fsync/s performance you are seeing on the dell machines are the result of some kind of caching from the raid controller.
In my opinion the p410i lacks this caching-options.
I have mx100 and mx200 drives on native intel ahci controllers and iam getting 4xx fsyncs/s.

In this article (https://www.sebastien-han.fr/blog/2...-if-your-ssd-is-suitable-as-a-journal-device/) you can see, that pro/consumer SSDs are not optimized for random write io.

Regarding this article (http://www.webhostingtalk.com/showthread.php?t=1374545) instead of the p410 the p420 is the go to card for ssd io performance.

 

[tytanick]

So it might be an Array. ok then.