Reading IO Wait times from the Proxmox Host instead of inside the VM

[eugenevdm]

I want to monitor IO Wait, also called wa in top. I have a script that outputs the values, which works perfectly inside the VM:

$ sh check.sh
2024-10-28 06:03:11 - IO Wait: 18%, Load averages: 1.13, 1.15, 1.26
2024-10-28 06:03:13 - IO Wait: 21%, Load averages: 1.20, 1.16, 1.27
2024-10-28 06:03:15 - IO Wait: 13%, Load averages: 1.20, 1.16, 1.27
2024-10-28 06:03:17 - IO Wait: 12%, Load averages: 1.20, 1.16, 1.27

This is a poorly performing server, as IO Wait is > 1.0 and jumping around a lot. I am aware that this server has a slow disk and this is precisely the type of thing I want to monitor.

My question is how can I monitor IO Wait times from the Hypervisor instread? I would like to do this across many VMs if possible.

So far I have found the Proxmox RRD API to be really useful for some statistics, e.g. I can retrieve all of this really quick:

"disk" => 0
"cpu" => 0.039755835139147
"diskwrite" => 145681.06666667
"diskread" => 0
"time" => 1730013480
"maxcpu" => 6
"mem" => 5157403648
"maxdisk" => 161061273600
"netin" => 3505.6083333333
"netout" => 980.375
"maxmem" => 8589934592

The problem is IO Wait and CPU isn't correlated. Nor is IO Wait and Load Average.

Any tips?

Here is the script inside the VM I use to monitor load average and IO Wait:

#!/bin/bash

while true; do
# Get the load averages
load=$(uptime | awk -F'load average: ' '{ print $2 }')

    # Get the IO wait time from vmstat (5th field in the last row)
    iowait=$(vmstat 1 2 | tail -1 | awk '{print $16}')

    # Print the values with a timestamp
    echo "$(date '+%Y-%m-%d %H:%M:%S') - Load averages: $load, IO Wait: $iowait%"

    # Wait for 1 second before next check
    sleep 1
done

 

[waltar]

Maybe that help you (apt install iotop and even use further awk formatting options):
iotop -o -b -k|egrep -v '{Total|Current}'|grep kvm|awk '{print $4" "$5" "$6" "$7" "$10" "$11" "$12" "$14}'
eg:
0.00 K/s 120.98 K/s 0.00 % kvm 138
0.00 K/s 67.53 K/s 0.00 % kvm 143
0.00 K/s 41.27 K/s 0.00 % kvm 146
0.00 K/s 59.72 K/s 0.00 % kvm 143