Not sure how helpful this will be to everyone since your system may be quite different than mine.
Most of my servers have an Areca RAID card with 4GB of cache ram.
Nearly all of our vms are KVM and stored on LVM over DRBD, we always use no-cache for VM disks.
We backup our VMs to a SATA disk plugged into the motherboard using LUKS encryption.
Because we use no-cache, the tuning here mostly only applies to the SATA backup disk.
All of our servers were 16GB RAM, now we have some with 24GB, 64GB and 128GB.
We have noticed that load has increased when performing backups, so much that sometimes it blocks IO to the VMs.
The problem is with the disk buffers in linux. The defaults are really bad for systems with large RAM.
/proc/sys/vm/dirty_ratio defaults to 20% of RAM
/proc/sys/vm/dirty_background_ratio defaults to 10%of RAM
The easiest way to describe what those settings do is to think of dirty_ratio being a giant bucket with a little hole in the bottom and water flowing out that hole is data written to disk.
Above the dirty_ratio bucket is a smaller bucket, dirty_background_ratio, with a hole in the side.
Disk IO, water, is poured into the smaller bucket with the hole in the side. When the water level rises above the hole, water flows into the giant bucket and drains out the hole in the bottom of the giant bucket.
If the giant bucket gets full, the water stops flowing into the smaller bucket, all IO stalls until the water has drained.
dirty_ratio changes the size of the giant bucket
dirty_background_ratio controls the height of the hole on the side of the smaller bucket.
With just 16GB of RAM the dirty_ratio (giant bucket) is 3.2GB and the dirty_background_ratio (smaller bucket) is 1.6GB.
My SATA backup disk that I am writing to can write at about 100MB/sec
So 1.6GB is written BEFORE anything is written to the SATA disk.
If the write buffer hits 3.2GB all other IO will get blocked until it is flushed 3.2GB/100MB/s = 32seconds.
Having all disk IO stalled for 32seconds is not the sort of performance anyone would want to see.
linux allows us to set either the ratio or set specific bytes, just replace _ratio with _bytes. eg
/proc/sys/vm/dirty_bytes
I choose to tune using bytes since the write speed of my SATA backup disks are the same in all of my systems.
Setting /proc/sys/vm/dirty_background_bytes VERY low resulted in the best performance and smallest load during backups.
For a system with 16GB Ram the default is 1600MB, I set it to 64MB, I think even 32MB might be better but have not tested yet.
I also decided to use 4GB for /proc/sys/vm/dirty_bytes I think this is large enough that I should never hit it causing IO to stall but if I do it will not be stalled for a super long time.
So far these settings are working really well, especially on the servers with 64GB and 128GB RAM.
My backups are slightly faster, load is lower, no more IO blocking during backups, life is good.
If you have some time and are a little daring I encourage you to adjust these settings and post a description of your system and what settings worked best for you.
Most of my servers have an Areca RAID card with 4GB of cache ram.
Nearly all of our vms are KVM and stored on LVM over DRBD, we always use no-cache for VM disks.
We backup our VMs to a SATA disk plugged into the motherboard using LUKS encryption.
Because we use no-cache, the tuning here mostly only applies to the SATA backup disk.
All of our servers were 16GB RAM, now we have some with 24GB, 64GB and 128GB.
We have noticed that load has increased when performing backups, so much that sometimes it blocks IO to the VMs.
The problem is with the disk buffers in linux. The defaults are really bad for systems with large RAM.
/proc/sys/vm/dirty_ratio defaults to 20% of RAM
/proc/sys/vm/dirty_background_ratio defaults to 10%of RAM
The easiest way to describe what those settings do is to think of dirty_ratio being a giant bucket with a little hole in the bottom and water flowing out that hole is data written to disk.
Above the dirty_ratio bucket is a smaller bucket, dirty_background_ratio, with a hole in the side.
Disk IO, water, is poured into the smaller bucket with the hole in the side. When the water level rises above the hole, water flows into the giant bucket and drains out the hole in the bottom of the giant bucket.
If the giant bucket gets full, the water stops flowing into the smaller bucket, all IO stalls until the water has drained.
dirty_ratio changes the size of the giant bucket
dirty_background_ratio controls the height of the hole on the side of the smaller bucket.
With just 16GB of RAM the dirty_ratio (giant bucket) is 3.2GB and the dirty_background_ratio (smaller bucket) is 1.6GB.
My SATA backup disk that I am writing to can write at about 100MB/sec
So 1.6GB is written BEFORE anything is written to the SATA disk.
If the write buffer hits 3.2GB all other IO will get blocked until it is flushed 3.2GB/100MB/s = 32seconds.
Having all disk IO stalled for 32seconds is not the sort of performance anyone would want to see.
linux allows us to set either the ratio or set specific bytes, just replace _ratio with _bytes. eg
/proc/sys/vm/dirty_bytes
I choose to tune using bytes since the write speed of my SATA backup disks are the same in all of my systems.
Setting /proc/sys/vm/dirty_background_bytes VERY low resulted in the best performance and smallest load during backups.
For a system with 16GB Ram the default is 1600MB, I set it to 64MB, I think even 32MB might be better but have not tested yet.
I also decided to use 4GB for /proc/sys/vm/dirty_bytes I think this is large enough that I should never hit it causing IO to stall but if I do it will not be stalled for a super long time.
So far these settings are working really well, especially on the servers with 64GB and 128GB RAM.
My backups are slightly faster, load is lower, no more IO blocking during backups, life is good.
If you have some time and are a little daring I encourage you to adjust these settings and post a description of your system and what settings worked best for you.
