IO Scheduler with SSD and HWRaid

[sahostking]

Hi

For cpanel kvm what do you guys use as scheduler. We have RAID 10 Hardware raid with BBU writeback and 6 x SSD disks

I see it sais proxmox uses deadline
https://pve.proxmox.com/wiki/IO_Scheduler

Does this mean we should leave scheduler to "none" on the KVM Guests? or should we set to deadline or noop

 

[manu]

Your KVM guests will have their own scheduler, set up in the VM indenpently from the OS.

I recommend you to leave the default on both hosts and guests.

 

[gkovacs]

Your KVM guests will have their own scheduler, set up in the VM indenpently from the OS.

I recommend you to leave the default on both hosts and guests.

This recommendation is not useful, as it's not based on facts, only a belief that "defaults must be the the best". Unfortunately, defaults are often not good, in this case they are really bad.

Hi

For cpanel kvm what do you guys use as scheduler. We have RAID 10 Hardware raid with BBU writeback and 6 x SSD disks

I see it sais proxmox uses deadline
https://pve.proxmox.com/wiki/IO_Scheduler

Does this mean we should leave scheduler to "none" on the KVM Guests? or should we set to deadline or noop

We have to talk about the host and the guest separately, but the general rule is you only want IO reordering (what the scheduler does) to take place at a single point in your system.

Host and hardware
Proxmox on the host uses the deadline scheduler by default, which tries to keep a low latency for each request, but it was created a long time ago, when SSDs weren't used in servers, only spinning HDDs, so it's not making optimal decisions. Another important fact is that your HW RAID card has it's own IO scheduler, which works using very different metrics as it knows much more about the disk layout. (At this time we are not talking about the fact that your SSDs have yet another scheduler inside them, hopefully you have enterprise SSDs that can sustain high IOPS.)

What all this means is that you are better off using the noop scheduler on the host, as it simply forwards your IO queue to the HW RAID driver, without introducing any unnecessary latency. We benchmarked noop extensively and compared it with deadline, and on HDD based HW RAID arrays they were similar, on SSDs noop was always better.

KVM guest
Depending on the distribution you would see CFQ or deadline set as default in your Linux guest, the latter is especially bad: CFQ introduces huge latency in many cases, so it's not advised to be used on servers at all, physical or virtual.

But what really matters is that a VM that knows absolutely nothing about the hosts's disk subsystem, and you already have an IO scheduler that works well (in your case in the HW RAID adapter), so you should always set it to noop. This is true for all kinds of disk subsystems BTW, your KVM guests should not do any IO reordering.

 

[gkovacs]

One more thing about the guest: in many distributions using kernels 3.16 and later, the OS will not use a traditional IO scheduler in VMs, rather using the multiqueue block layer for fast devices, so when you see:

# cat /sys/block/vda/queue/scheduler
none

then you have nothing else to do, your OS is already aware that it's running in a VM and disables the scheduler.

 

[Q-wulf]

This is quite interesting.

What all this means is that you are better off using the noop scheduler on the host, as it simply forwards your IO queue to the HW RAID driver, without introducing any unnecessary latency. We benchmarked noop extensively and compared it with deadline, and on HDD based HW RAID arrays they were similar, on SSDs noop was always better.

So if i read this right:
If you have SSD based system, you run noop scheduler ??


Another question:
Say you have a centos 7 VM and execute the command you describe.
Lets further say you got the output:

noop [deadline] cfg

what is it you are looking at ?

 

[gkovacs]

This is quite interesting.

So if i read this right:
If you have SSD based system, you run noop scheduler ??

Yes, because SSDs are usually much faster than what the schedulers were designed for.

Another question:
Say you have a centos 7 VM and execute the command you describe.
Lets further say you got the output:

noop [deadline] cfg

what is it you are looking at ?

The above means you have the deadline scheduler set for your disk. You will be better off with the noop scheduler when inside a VM.

 

[Q-wulf]

Yes, because SSDs are usually much faster than what the schedulers were designed for.
[...]

The above means you have the deadline scheduler set for your disk. You will be better off with the noop scheduler when inside a VM.

I just stumbled upon this little piece of info:
https://wiki.debian.org/SSDOptimization
They actually recommend deadline.

I shall test noop and report back

 

[gkovacs]

I just stumbled upon this little piece of info:
https://wiki.debian.org/SSDOptimization
They actually recommend deadline.

I shall test noop and report back

We already did test it on Proxmox, but I look forward to seeing your benchmarks.

They do recommend deadline, but not for VMs (Debian 8 disables the IO scheduler inside VMs and on NVMe SSDs). Also they are not talking about HW RAID cards, for which noop is the only sensible choice.

 

[Q-wulf]

We already did test it on Proxmox, but I look forward to seeing your benchmarks.

Sry, What i should have said is, i shall try it on my lab setup before using it in production ( I test every change before i f*** up a working a system, cant blame nobody but yourself that way)

Debian 8 disables the IO scheduler inside VMs and on NVMe SSDs

I think for centos 7 this is only the case of disks you have passed through via virtio or of virtio vDisk itself.
/dev/vda .. etc
When it is e.g. a sata or a scsi vDisk which reads as /dev/sda on the Vm, it will enable deadline

in any case, cheers for the tip.