High IO delay on SSD and Low fsync rate

tytanick

Member
Feb 25, 2013
96
3
8
Hi,
I have one cluster with really low fsync rate and i have some problems with high IO while converting qcow to raw on it. And also with different operations.

Specs:
Block size is 4K , Ext4, problem is also on LVM
05:00.0 RAID bus controller: Hewlett-Packard Company Smart Array G6 controllers (rev 01)
2 x 250GB SSD Crucial MX200
root@node2:~# pveversion
pve-manager/4.2-15/6669ad2c (running kernel: 4.4.10-1-pve)
Intel(R) Xeon(R) CPU E5630 @ 2.53GHz x 16 cores

root@node2:~# pveperf
CPU BOGOMIPS: 81045.44
REGEX/SECOND: 1097411
HD SIZE: 7.01 GB (/dev/dm-0)
BUFFERED READS: 293.48 MB/sec
AVERAGE SEEK TIME: 0.21 ms
FSYNCS/SECOND: 294.96
DNS EXT: 67.27 ms

oCIfKT4.png
 
More screenshots.
Anybodo know why is this happening ?
As i see, 10MB/s makes ssd used at 100%.
Why ?

On node2 there is MOVE DISK from NAS to internal DRBD on ssd.
ubYzRkK.png


Ph3aeUM.png





Node4 wchich is connected with node2 with DRBD
nfBEbPP.png
 
Hello,
the mx200 SSDs are consumer grade drives optimized for read workloads. I have a similar fsync performance with 500gb mx100 and mx200 SSDs on my machines.

For better fsync performance you should use datacenter disks like intel dc 3700 or samsung sm863 drives.
 
its not the case.
i have like 10x better performance on other nodes in 3.2 pve also on mx200 crucial (those disks are enterprise level :)
 
proxmox 3.2 is using ext3 with barriers=0 mount option.
Proxmox 4 uses Ext4 which defaults to barriers=1 mount option.
With barriers=1 the system has to wait for the sync and the sync performance differs greatly between consumer and datacenter SSDs.
 
Yeah but this is /dev/sdb1 LVM with no filesystem so it doesnt mount anything right ?
Still high IO.
 
Alright, are using a raid configuration for these drives? You wrote 2x250GB. Can you give more information about the HP Smart array G6 Controller?
In my opinion the buffered reads are low for a mx200 SSD. More like sata2 performance instead of sata3.
 
Sure.
HP ProLiant DL380 G7
Smart Array P410i ( 05:00.0 RAID bus controller: Hewlett-Packard Company Smart Array G6 controllers (rev 01))
2x 250GB ssd Crucial MX200 in Raid1
Created 2 virtual disks /dev/sda for proxmox and /dev/sdb for drbd

Other servers (but all others are Dell) but with proxmox ve 3.2 also on Crucial mx200 shows this:
root@node1:~# pveperf
CPU BOGOMIPS: 183862.80
REGEX/SECOND: 1257319
HD SIZE: 7.01 GB (/dev/mapper/pve-root)
BUFFERED READS: 412.26 MB/sec
AVERAGE SEEK TIME: 0.13 ms
FSYNCS/SECOND: 4410.59
DNS EXT: 10.21 ms
 
Crucial mx200 are cheap (pro)consumer SSDs, not suited for high load and DRBD.

if you want performance, go for Intel Datacenter SSDs, best with NMVe.
 
The Smart Array P410i is known as a low Performance on Board Controller. Not recommended for SSD as far as my experiences goes with this Controller. Try to get a 420 or higher version.
 
Okay, i think the 4410 fsync/s performance you are seeing on the dell machines are the result of some kind of caching from the raid controller.
In my opinion the p410i lacks this caching-options.
I have mx100 and mx200 drives on native intel ahci controllers and iam getting 4xx fsyncs/s.

In this article (https://www.sebastien-han.fr/blog/2...-if-your-ssd-is-suitable-as-a-journal-device/) you can see, that pro/consumer SSDs are not optimized for random write io.

Regarding this article (http://www.webhostingtalk.com/showthread.php?t=1374545) instead of the p410 the p420 is the go to card for ssd io performance.
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!