IO Performance Tuning

e100

Renowned Member
Nov 6, 2010
1,268
45
88
Columbus, Ohio
ulbuilder.wordpress.com
Not sure how helpful this will be to everyone since your system may be quite different than mine.

Most of my servers have an Areca RAID card with 4GB of cache ram.
Nearly all of our vms are KVM and stored on LVM over DRBD, we always use no-cache for VM disks.
We backup our VMs to a SATA disk plugged into the motherboard using LUKS encryption.

Because we use no-cache, the tuning here mostly only applies to the SATA backup disk.

All of our servers were 16GB RAM, now we have some with 24GB, 64GB and 128GB.
We have noticed that load has increased when performing backups, so much that sometimes it blocks IO to the VMs.

The problem is with the disk buffers in linux. The defaults are really bad for systems with large RAM.
/proc/sys/vm/dirty_ratio defaults to 20% of RAM
/proc/sys/vm/dirty_background_ratio defaults to 10%of RAM

The easiest way to describe what those settings do is to think of dirty_ratio being a giant bucket with a little hole in the bottom and water flowing out that hole is data written to disk.
Above the dirty_ratio bucket is a smaller bucket, dirty_background_ratio, with a hole in the side.
Disk IO, water, is poured into the smaller bucket with the hole in the side. When the water level rises above the hole, water flows into the giant bucket and drains out the hole in the bottom of the giant bucket.
If the giant bucket gets full, the water stops flowing into the smaller bucket, all IO stalls until the water has drained.
dirty_ratio changes the size of the giant bucket
dirty_background_ratio controls the height of the hole on the side of the smaller bucket.


With just 16GB of RAM the dirty_ratio (giant bucket) is 3.2GB and the dirty_background_ratio (smaller bucket) is 1.6GB.
My SATA backup disk that I am writing to can write at about 100MB/sec
So 1.6GB is written BEFORE anything is written to the SATA disk.
If the write buffer hits 3.2GB all other IO will get blocked until it is flushed 3.2GB/100MB/s = 32seconds.
Having all disk IO stalled for 32seconds is not the sort of performance anyone would want to see.

linux allows us to set either the ratio or set specific bytes, just replace _ratio with _bytes. eg
/proc/sys/vm/dirty_bytes

I choose to tune using bytes since the write speed of my SATA backup disks are the same in all of my systems.
Setting /proc/sys/vm/dirty_background_bytes VERY low resulted in the best performance and smallest load during backups.
For a system with 16GB Ram the default is 1600MB, I set it to 64MB, I think even 32MB might be better but have not tested yet.
I also decided to use 4GB for /proc/sys/vm/dirty_bytes I think this is large enough that I should never hit it causing IO to stall but if I do it will not be stalled for a super long time.

So far these settings are working really well, especially on the servers with 64GB and 128GB RAM.
My backups are slightly faster, load is lower, no more IO blocking during backups, life is good.

If you have some time and are a little daring I encourage you to adjust these settings and post a description of your system and what settings worked best for you.
 
Thanks for the tip!

Am I correct, these values will be returned to default (when are settled via echo)? Is here any way to make these changes permanent?
Thanks in advance,
 
/etc/sysctl.conf

Code:
[COLOR=#000000][FONT=Arial]vm.dirty_ratio = 5[/FONT][/COLOR]
[COLOR=#000000][FONT=Arial]vm.dirty_background_ratio = 10[/FONT][/COLOR]


or

Code:
vm.dirty_bytes = 5555555
vm.dirty_background_bytes = 100000000

Rather than using echo, you can just set them in sysctl.conf and then run:
Code:
sysctl -p
 
bro could you add me on skype ? i have realy bad life with io delay :( myskype:johnfaiz2009
 
I've not used skype for so long I do not remember my login, I would prefer to help you here in the forum so others can learn from the problem and solution.
Start a new thread describing your problem and I will try to help.

okay iwill start it now thanks alot
 
I've not used skype for so long I do not remember my login, I would prefer to help you here in the forum so others can learn from the problem and solution.
Start a new thread describing your problem and I will try to help.

I also went away from skype to jabber...much less n.s.a. style ;)
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!