Increased wait IO on Proxmox 4 with deadline scheduler

[Patrick Zippenfenig]

Hi,

I recently migrated from Proxmox 3 to 4. Workload: 1TB SSD with 12-hourly sequential write (direct IO) and random read (buffered IO).
Wait IO increased massively. It's not about ext4 journal/barriers ;-)


1. Mark: Proxmox 4 upgrade
2. Mark: Switch from deadline to noop

In a later stage I also disabled barriers and switched to data=writeback, which reduced latency spikes but wait IO still remains significantly higher. I do not care about data consistency. These drives are expected to fail.


On another server I use XFS instead of Ext4 (same workload):

(Server was offline for 2 days). On XFS increased IO wait is even worse. Switching from deadline to noop improved the situation (midday on Wed 23), but wait IO was almost twice compared to Ext4. nobarrier on XFS did not change much. Switching from XFS to Ext4 shows now the same IO wait as on other servers.

As it is quite a big jump in kernel version from Proxmox 3 to 4, it's difficult to pin-point certain kernel changes to these issues. Deadline scheduler received some updates in kernel 3.15. Ext4 barriers and data journal enabled per default.

Did anyone else experience increased IO wait on Proxmox 4? Any ideas to further improve the situation?


Thanks!
Patrick

 

[noname]

Patrick
i have the same problem a lot io delays huge difference between proxmox 3 vs proxmox 4
i wait also official answer from proxmox staff to give me some guide lines

 

[alex3137]

I have the same problem after upgrading to Proxmox 4. I disabled barriers, switched to data=writeback and use chroot disk for containers which helps to improve the situation but IOwait is still a problem. I opened a thread here.

Proxmox staff does not react to the problem despite many threads on this forum. I am considering downgrading to Proxmox 3.4 or switch to another virtualisation solution.

 

[noname]

The same here revert back to proxmox 3.4, proxmox it has a big problem with new machines like dell r730 or 430 especially with raid disk controller p730mini.

 

[Patrick Zippenfenig]

Did you try noop scheduler?

Despite increase in IO wait, application latency dropped. I'm fairly happy with LXC and ZFS on Proxmox 4

 

[noname]

Did you try noop scheduler?

Despite increase in IO wait, application latency dropped. I'm fairly happy with LXC and ZFS on Proxmox 4

Patrick thanks for the information
use deadline or elevator ;

GRUB_CMDLINE_LINUX_DEFAULT="... elevator=deadline"

or:

GRUB_CMDLINE_LINUX_DEFAULT="... elevator=cfq"

 

[Patrick Zippenfenig]

echo noop > /sys/block/sdX/queue/scheduler

and verify with

cat /sys/block/sdX/queue/scheduler
[noop] deadline cfq

I use udev rules to set different attributes for block devices

 

[noname]

echo noop > /sys/block/sdX/queue/scheduler

and verify with

cat /sys/block/sdX/queue/scheduler
[noop] deadline cfq

I use udev rules to set different attributes for block devices

running via crontab ; how often ?
or
put direct to grub ;
when make these changes do you have any benchmarks;
regards

 

[pizza]

Patrick
i have the same problem a lot io delays huge difference between proxmox 3 vs proxmox 4
i wait also official answer from proxmox staff to give me some guide lines

On the same hardware creating a database within my application, the performance decrease was a factor 2