IO Delay after upgrade to 5.0

e2rd

New Member
Sep 19, 2017
4
0
1
45
Hello,
Yestarday I've upgraded proxmox server from 4.4 to 5.0. Everything was ok (Nothing changed in VMs) but now I see that IO delay is 5x greater. See images attached (between 19:00 and 12:00 is not normal work - backups, etc). During normal usage with v4.4 io/delay was about 2%, with v.5.0 is about 10%.
Is it normal and what is the reason?
Phisical 2 disks in raid1 with ext4. VM disk - virtio and no cache set.
1hour-avg.png 1day-avg.png
 
Hi,

maybe can you try to install pve-kernel 4.4 from proxmox4, to see if it's a kernel problem ?
 
Don't know about upgrades, but fresh 5.0 install uses CFQ chedule, while 4.x abandoned it in favour of Deadline variant, that for me worked far better.
check what scheduler you have
Code:
# cat /sys/block/sda/queue/scheduler
noop anticipatory [deadline] cfq
(above deadline is the active one)
If you have cfq and want to try deadline you can do on the fly like with
Code:
echo deadline > /sys/block/sda/queue/scheduler
without rebooting(if sda is the disk, maybe do it for all your disks)
or if you want permanent you have to edit/etc/default/grub, change a line so you have
Code:
GRUB_CMDLINE_LINUX_DEFAULT="quiet elevator=deadline"
and then issue the command "update-grub" and reboot
 
Don't know about upgrades, but fresh 5.0 install uses CFQ chedule, while 4.x abandoned it in favour of Deadline variant, that for me worked far better.
Here, I can see that our fresh installs use deadline as expected (I guess this is the default in the PVE kernel).

What must usually be done is to configure the guests with noop scheduler, however.
 
Hello
I thnik "problem" is related to kernel: I've checked in this night kernel 4.4.76 from proxmox 4.4 and i/o delay was low as before upgrade (~2%)
I've also checked scheludes and default is deadline (v5.0), but in 4.4 also is deadline (most of checked machines - one has [noop] ????). Changed to cfq as mmenas said give no reslut in io/delay.
Other interesting thing is that one VM is multicast source and before upgrade when we have "high" (~10%) i/o delay there were loss in multicast packets. High io was caused by other VM wich has postgres db - container, after "transform" to kvm i/o drop to 2%. Now in spite of 10% io delay we don't observe loss in multicasts.
io delay is casued by disk writes becouse when I shutdown machine whcih has most of disk operations (~500k/s) io going down to 2-3%.
 
Micro: I've switched disk from virtio do ide but io delay is still the same :/
 
maybe could it be related to new kernel 4.10 feature : writeback throttling ? (it's enable in proxmox 4.10 kernel)

CONFIG_BLK_WBT: Enable support for block device writeback throttling
General informations
The Linux kernel configuration item CONFIG_BLK_WBT:

  • prompt: Enable support for block device writeback throttling
  • type: bool
  • depends on: (none)
  • defined in block/Kconfig
  • found in Linux kernels: 4.10–4.13, 4.14-rc+HEAD

https://lwn.net/Articles/682582/



if you don't use zfs, maybe can you try to install kernel 4.9 from debian stretch repo, to compare ?
 
maybe could it be related to new kernel 4.10 feature : writeback throttling ? (it's enable in proxmox 4.10 kernel)

CONFIG_BLK_WBT: Enable support for block device writeback throttling
General informations
The Linux kernel configuration item CONFIG_BLK_WBT:

  • prompt: Enable support for block device writeback throttling
  • type: bool
  • depends on: (none)
  • defined in block/Kconfig
  • found in Linux kernels: 4.10–4.13, 4.14-rc+HEAD

https://lwn.net/Articles/682582/



if you don't use zfs, maybe can you try to install kernel 4.9 from debian stretch repo, to compare ?

that is true I verified

cat /boot/config-4.10.17-3-pve | grep WBT
CONFIG_BLK_WBT=y
# CONFIG_BLK_WBT_SQ is not set
CONFIG_BLK_WBT_MQ=y

could it be disabled via kernel option in grub? or udev rules?

how could I compile the pve-4.10.17-3-pve without wbt?
 
Last edited:
that is true I verified

cat /boot/config-4.10.17-3-pve | grep WBT
CONFIG_BLK_WBT=y
# CONFIG_BLK_WBT_SQ is not set
CONFIG_BLK_WBT_MQ=y

could it be disabled via kernel option in grub? or udev rules?

how could I compile the pve-4.10.17-3-pve without wbt?

I'm not sure, buy maybe

"
wb_lat_usec (RW)
----------------
If the device is registered for writeback throttling, then this file shows
the target minimum read latency. If this latency is exceeded in a given
window of time (see wb_window_usec), then the writeback throttling will start
scaling back writes. Writing a value of '0' to this file disables the
feature. Writing a value of '-1' to this file resets the value to the
default setting.
"
echo 0 > /sys/block/sdX/queue/wbt_lat_usec

if it don't work, you can rebuild the kernel

git clone git://git.proxmox.com/git/pve-kernel.git
edit Makefile
PVE_CONFIG_OPTS= \
....
-d CONFIG_BLK_WBT
-d CONFIG_BLK_WBT_MQ

then #make
 
  • Like
Reactions: sneaker15
(note that pve-kernel git is now at 4.13, so I think it'll be release in pvetest soon)
 
setting echo 0 > /sys/block/sdX|mdX/queue/wbt_lat_usec gives no change in io/delay
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!