IO Delay after upgrade to 5.0

[e2rd]

Hello,
Yestarday I've upgraded proxmox server from 4.4 to 5.0. Everything was ok (Nothing changed in VMs) but now I see that IO delay is 5x greater. See images attached (between 19:00 and 12:00 is not normal work - backups, etc). During normal usage with v4.4 io/delay was about 2%, with v.5.0 is about 10%.
Is it normal and what is the reason?
Phisical 2 disks in raid1 with ext4. VM disk - virtio and no cache set.

 

[aderumier]

Hi,

maybe can you try to install pve-kernel 4.4 from proxmox4, to see if it's a kernel problem ?

 

[mmenaz]

Don't know about upgrades, but fresh 5.0 install uses CFQ chedule, while 4.x abandoned it in favour of Deadline variant, that for me worked far better.
check what scheduler you have

# cat /sys/block/sda/queue/scheduler
noop anticipatory [deadline] cfq

(above deadline is the active one)
If you have cfq and want to try deadline you can do on the fly like with

echo deadline > /sys/block/sda/queue/scheduler

without rebooting(if sda is the disk, maybe do it for all your disks)
or if you want permanent you have to edit/etc/default/grub, change a line so you have

GRUB_CMDLINE_LINUX_DEFAULT="quiet elevator=deadline"

and then issue the command "update-grub" and reboot

 

[Symbol]

Don't know about upgrades, but fresh 5.0 install uses CFQ chedule, while 4.x abandoned it in favour of Deadline variant, that for me worked far better.

Here, I can see that our fresh installs use deadline as expected (I guess this is the default in the PVE kernel).

What must usually be done is to configure the guests with noop scheduler, however.

 

[micro]

I think you are in the same virtio IO wait/delay boat with us
https://forum.proxmox.com/threads/3000-msec-ping-and-packet-drops-with-virtio-under-load.36687/

Unfortunately there isn't any official Proxmox team response yet about this and the only temporary workaround found so far is to switch from disk virtio bus to ide for all VMs.

 

[aderumier]

fresh install of proxmox 5 have deadline scheduler for me too

 

[e2rd]

Hello
I thnik "problem" is related to kernel: I've checked in this night kernel 4.4.76 from proxmox 4.4 and i/o delay was low as before upgrade (~2%)
I've also checked scheludes and default is deadline (v5.0), but in 4.4 also is deadline (most of checked machines - one has [noop] ????). Changed to cfq as mmenas said give no reslut in io/delay.
Other interesting thing is that one VM is multicast source and before upgrade when we have "high" (~10%) i/o delay there were loss in multicast packets. High io was caused by other VM wich has postgres db - container, after "transform" to kvm i/o drop to 2%. Now in spite of 10% io delay we don't observe loss in multicasts.
io delay is casued by disk writes becouse when I shutdown machine whcih has most of disk operations (~500k/s) io going down to 2-3%.

 

[e2rd]

Micro: I've switched disk from virtio do ide but io delay is still the same :/

 

[aderumier]

maybe could it be related to new kernel 4.10 feature : writeback throttling ? (it's enable in proxmox 4.10 kernel)

CONFIG_BLK_WBT: Enable support for block device writeback throttling
General informations
The Linux kernel configuration item CONFIG_BLK_WBT:

• prompt: Enable support for block device writeback throttling
• type: bool
• depends on: (none)
• defined in block/Kconfig
• found in Linux kernels: 4.10–4.13, 4.14-rc+HEAD

https://lwn.net/Articles/682582/



if you don't use zfs, maybe can you try to install kernel 4.9 from debian stretch repo, to compare ?

 

[mac.linux.free]

maybe could it be related to new kernel 4.10 feature : writeback throttling ? (it's enable in proxmox 4.10 kernel)

CONFIG_BLK_WBT: Enable support for block device writeback throttling
General informations
The Linux kernel configuration item CONFIG_BLK_WBT:

• prompt: Enable support for block device writeback throttling
• type: bool
• depends on: (none)
• defined in block/Kconfig
• found in Linux kernels: 4.10–4.13, 4.14-rc+HEAD

https://lwn.net/Articles/682582/



if you don't use zfs, maybe can you try to install kernel 4.9 from debian stretch repo, to compare ?

that is true I verified

cat /boot/config-4.10.17-3-pve | grep WBT
CONFIG_BLK_WBT=y
# CONFIG_BLK_WBT_SQ is not set
CONFIG_BLK_WBT_MQ=y

could it be disabled via kernel option in grub? or udev rules?

how could I compile the pve-4.10.17-3-pve without wbt?

 

[aderumier]

that is true I verified

cat /boot/config-4.10.17-3-pve | grep WBT
CONFIG_BLK_WBT=y
# CONFIG_BLK_WBT_SQ is not set
CONFIG_BLK_WBT_MQ=y

could it be disabled via kernel option in grub? or udev rules?

how could I compile the pve-4.10.17-3-pve without wbt?

I'm not sure, buy maybe

"
wb_lat_usec (RW)
----------------
If the device is registered for writeback throttling, then this file shows
the target minimum read latency. If this latency is exceeded in a given
window of time (see wb_window_usec), then the writeback throttling will start
scaling back writes. Writing a value of '0' to this file disables the
feature. Writing a value of '-1' to this file resets the value to the
default setting.
"
echo 0 > /sys/block/sdX/queue/wbt_lat_usec

if it don't work, you can rebuild the kernel

git clone git://git.proxmox.com/git/pve-kernel.git
edit Makefile
PVE_CONFIG_OPTS= \
....
-d CONFIG_BLK_WBT
-d CONFIG_BLK_WBT_MQ

then #make

 

[mac.linux.free]

the sysfs key is not visible on my raid system but on a system with a single hhd/ssd

 

[aderumier]

(note that pve-kernel git is now at 4.13, so I think it'll be release in pvetest soon)

 

[e2rd]

setting echo 0 > /sys/block/sdX|mdX/queue/wbt_lat_usec gives no change in io/delay

 

[fabian]

(note that pve-kernel git is now at 4.13, so I think it'll be release in pvetest soon)

correct, see https://forum.proxmox.com/threads/p...3-latest-zfs-lxc-2-1.36943/page-2#post-182836