Increased IO Delay radically, how to reduce?

Discussion in 'Proxmox VE: Installation and configuration' started by fintnovo, Apr 12, 2019.

  1. fintnovo

    fintnovo New Member

    Joined:
    Jan 4, 2014
    Messages:
    23
    Likes Received:
    0
    Good day.
    3 month ago average IO Delay increased dramatically, about 10 or 15 times. Nothing radically changed in VMs. Hypervisor hardware also the same. I have 17 VM running: (10 with Windows 7,10, Server 2012) and (7 with Linux Ubuntu 18.04).
    There are 3 disks also:
    upload_2019-4-12_17-51-51.png
    all VMs on nvme0n1 (m.2 ssd)
    sda - only backups (sata hdd)
    sdb - PVE + swap (sata ssd)
    Here is screenshot with typical server load (avarage IO Delay about 10%)
    Free memory about 40%. Average CPU usage about 20% ....
    upload_2019-4-12_17-50-42.png
    Any idea?
     
  2. PeterH896

    PeterH896 New Member

    Joined:
    Jan 14, 2018
    Messages:
    5
    Likes Received:
    0
    Hi,

    you could use "iotop" to see which process is causing this IO delay.
     
  3. Whatever

    Whatever Member

    Joined:
    Nov 19, 2012
    Messages:
    198
    Likes Received:
    5
  4. lhorace

    lhorace Member

    Joined:
    Oct 17, 2015
    Messages:
    83
    Likes Received:
    8
    You also want to look at the total IOPS when the storage is NOT busy
     
  5. fintnovo

    fintnovo New Member

    Joined:
    Jan 4, 2014
    Messages:
    23
    Likes Received:
    0
    Good day, Peter/
    Here is iotop output:
    upload_2019-4-15_8-49-48.png
     
  6. fintnovo

    fintnovo New Member

    Joined:
    Jan 4, 2014
    Messages:
    23
    Likes Received:
    0
    Thanks, I saw this post. I do some modifications in sysctl.conf and reboot, but nothing changed:
    upload_2019-4-15_8-57-18.png
     
  7. fintnovo

    fintnovo New Member

    Joined:
    Jan 4, 2014
    Messages:
    23
    Likes Received:
    0
    I shutdown all VMs, then run "iotops", here is output:
    upload_2019-4-15_9-17-32.png
    also here is average hourly output with disabled VMs (IO Delay reduced..):
    upload_2019-4-15_9-24-57.png
     
  8. Whatever

    Whatever Member

    Joined:
    Nov 19, 2012
    Messages:
    198
    Likes Received:
    5
    _ratio parameters depend on total RAM. Thus it's recommended to use _bytes instead
    a

    one more thing to be clarified. Could you print the output of "zpool status" command (there is l2arc_feed process on one of your screenshots)
     
  9. fintnovo

    fintnovo New Member

    Joined:
    Jan 4, 2014
    Messages:
    23
    Likes Received:
    0
  1. This site uses cookies to help personalise content, tailor your experience and to keep you logged in if you register.
    By continuing to use this site, you are consenting to our use of cookies.
    Dismiss Notice