Finding guests which generate allot of disk IO

David Herselman · Aug 9, 2018

The following is most probably a little crude but essentially collects concurrent storage IO statistics from cluster nodes and replaces 'rbdX' references with Ceph RBD image names.

Useful in finding guests which generate allot of disk IO:

Code:

#!/bin/sh
hosts='kvm1a kvm1b kvm1c kvm1d kvm1e kvm1f';
time=30;
for host in $hosts; do
  if [ "$HOSTNAME" == "$host" ]; then
    iostat -xdy $time 1 | grep '^rbd' > /tmp/"$host"_iostat &
  else
    ( ssh -o StrictHostKeyChecking=no root@$host.company.com "iostat -xdy $time 1 | grep '^rbd'" > /tmp/"$host"_iostat ) &
  fi
done
sleep $[$time+5];

echo "Device:               rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util" > /tmp/io_stats;
startpath=`pwd`;
cd /tmp;
for file in *_iostat; do
  host=${file%_iostat};
  if [ "$HOSTNAME" == "$host" ]; then
    rbd showmapped | grep -P '^\d+\s+' | perl -pe 's/\d+\s+(\S+)\s+(\S+).*\/dev\/(\S+).*/\3 \1\\\\\/\2/g' | while read dev name; do
      perl -i -pe "s/^$dev\s+/$name\t/g" $file;
    done
  else
    ssh -o StrictHostKeyChecking=no root@$host.company.com "rbd showmapped" | grep -P '^\d+\s+' | perl -pe 's/\d+\s+(\S+)\s+(\S+).*\/dev\/(\S+).*/\3 \1\\\\\/\2/g' | while read dev name; do
      perl -i -pe "s/^$dev\s+/$name\t/g" $file;
    done
  fi
  cat $file >> /tmp/io_stats;
  rm -f $file;
done
cd $startpath;

cat /tmp/io_stats;

Sample output:

Code:

Device:               rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
rbd_hdd/vm-209-disk-1   0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
rbd_hdd/vm-107-disk-1   0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
rbd_nvme/vm-101-disk-1  0.00     9.93    0.90    5.37     6.00    61.20    21.45     0.03    4.98    2.52    5.39   2.36   1.48
rbd_hdd/vm-108-disk-1   0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
rbd_hdd/vm-104-disk-1   0.00     0.37    0.00    0.67     0.00     4.13    12.40     0.00    0.40    0.00    0.40   0.40   0.03
rbd_hdd/vm-109-disk-1   0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
rbd_hdd/vm-123-disk-1   0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
rbd_hdd/vm-138-disk-1   0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
rbd_hdd/vm-143-disk-1   0.00     0.03    0.00    0.03     0.00     0.27    16.00     0.00    0.00    0.00    0.00   0.00   0.00
rbd_hdd/vm-144-disk-1   0.00     0.03    0.00    0.07     0.00     0.40    12.00     0.00    0.00    0.00    0.00   0.00   0.00
rbd_hdd/vm-141-disk-1   0.00     0.17    0.00    0.13     0.00     1.20    18.00     0.00    1.00    0.00    1.00   1.00   0.01
rbd_hdd/vm-145-disk-1   0.00     0.13    0.07    0.10     0.27     0.93    14.40     0.00    0.00    0.00    0.00   0.00   0.00
rbd_hdd/vm-151-disk-1   0.00     0.50    0.00    1.23     0.00     6.93    11.24     0.00    0.86    0.00    0.86   0.65   0.08
rbd_hdd/vm-154-disk-1   0.00     0.10    0.00    0.20     0.00     1.20    12.00     0.00    1.33    0.00    1.33   1.33   0.03
rbd_hdd/vm-204-disk-1   0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
rbd_nvme/vm-100-disk-1  0.00    96.50    0.00   14.80     0.00   445.20    60.16     0.15   10.05    0.00   10.05   1.36   2.01
rbd_hdd/vm-285-disk-1   0.00     2.50    0.10    3.67     0.40    24.67    13.31     0.00    1.10    1.33    1.09   0.25   0.09
rbd_hdd/vm-291-disk-1   0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
rbd_nvme/vm-212-disk-1  0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
rbd_nvme/vm-211-disk-1  0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
rbd_nvme/vm-212-disk-2  0.00   190.57    0.00   43.93     0.00   938.00    42.70     0.92   21.04    0.00   21.04   1.49   6.56
rbd_nvme/vm-211-disk-2  0.00   240.83    0.00   68.00     0.00  1235.33    36.33     1.41   20.71    0.00   20.71   1.40   9.49
rbd_nvme/vm-212-disk-3  0.00    56.20    0.00    1.03     0.00   228.93   443.10     0.01    6.19    0.00    6.19   1.94   0.20
rbd_nvme/vm-211-disk-3  0.00    28.00    0.00    1.63     0.00   118.53   145.14     0.02   10.04    0.00   10.04   2.86   0.47
rbd_hdd/vm-170-disk-1   0.00     1.50    0.03    0.80     0.13     9.20    22.40     0.00    0.00    0.00    0.00   0.00   0.00
rbd_hdd/vm-300-disk-1   0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
rbd_hdd/vm-171-disk-1   0.00     5.00    0.00    4.30     0.00    37.20    17.30     0.00    0.87    0.00    0.87   0.22   0.09
rbd_hdd/vm-139-disk-1   0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
rbd_hdd/vm-155-disk-1   0.00     0.17    0.00    0.20     0.00     1.47    14.67     0.00    1.33    0.00    1.33   1.33   0.03
rbd_nvme/vm-295-disk-1  0.00   253.67    0.00   42.10     0.00  1420.40    67.48     0.23    5.55    0.00    5.55   1.15   4.84
rbd_nvme/vm-295-disk-2  0.00   424.97    0.00   62.87     0.00  1951.33    62.08     0.15    2.42    0.00    2.42   2.36  14.85
rbd_hdd/vm-282-disk-1   0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
rbd_nvme/vm-172-disk-1  0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
rbd_nvme/vm-172-disk-2  0.00    97.50    0.03   25.93     0.13   493.73    38.04     0.47   18.24    4.00   18.26   1.97   5.11
rbd_nvme/vm-172-disk-3  0.00   534.27    0.00   79.63     0.00  2455.60    61.67     2.05   25.81    0.00   25.81   1.28  10.23
rbd_hdd/vm-169-disk-1   0.00     0.00    0.00    0.17     0.00     0.67     8.00     0.00    0.00    0.00    0.00   0.00   0.00
rbd_nvme/vm-238-disk-1  0.00     0.67    0.00    1.13     0.00     7.20    12.71     0.01    6.00    0.00    6.00   3.65   0.41
rbd_nvme/vm-213-disk-1  0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
rbd_nvme/vm-213-disk-2  0.00    28.27    0.00    2.87     0.00   124.53    86.88     0.02    6.74    0.00    6.74   1.49   0.43
rbd_nvme/vm-213-disk-3  0.00   196.37    0.00   14.97     0.00   845.33   112.96     0.30   19.94    0.00   19.94   1.75   2.61
rbd_hdd/base-280-disk-1 0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
rbd_hdd/vm-301-disk-1   0.00    19.13    0.00   18.77     0.00   151.60    16.16     0.04    2.18    0.00    2.18   0.83   1.56
rbd_hdd/vm-106-disk-1   0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
rbd_hdd/vm-105-disk-1   0.00     0.20    0.07    0.13     0.27     1.33    16.00     0.00   11.33   34.00    0.00  11.33   0.23
rbd_hdd/vm-103-disk-1   0.00    22.33    0.00    6.43     0.00   164.93    51.27     0.01    1.14    0.00    1.14   0.46   0.29
rbd_hdd/vm-110-disk-1   0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
rbd_hdd/vm-114-disk-1   0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
rbd_hdd/vm-111-disk-1   0.00     0.17    0.00    0.13     0.00     1.20    18.00     0.00    0.00    0.00    0.00   0.00   0.00
rbd_hdd/vm-113-disk-1   0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
rbd_hdd/vm-124-disk-1   0.00     0.13    0.00    0.10     0.00     0.93    18.67     0.00    2.67    0.00    2.67   1.33   0.01
rbd_hdd/vm-115-disk-1   0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
rbd_hdd/vm-126-disk-1   0.00     0.17    0.00    0.13     0.00     1.20    18.00     0.00    1.00    0.00    1.00   1.00   0.01
rbd_hdd/vm-120-disk-1   0.00     0.03    0.00    0.03     0.00     0.27    16.00     0.00    0.00    0.00    0.00   0.00   0.00
rbd_hdd/vm-131-disk-1   0.00     1.97    0.00    1.43     0.00    20.93    29.21     0.01    8.09    0.00    8.09   2.70   0.39
rbd_hdd/vm-142-disk-1   0.00     4.53    0.03    2.17     0.40    37.60    34.55     0.00    0.73    0.00    0.74   0.24   0.05
rbd_hdd/vm-132-disk-1   0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
rbd_hdd/vm-140-disk-1   0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
rbd_hdd/vm-146-disk-1   0.00     1.70    0.03    3.67     0.13    21.47    11.68     0.00    1.23    0.00    1.24   0.65   0.24
rbd_hdd/vm-148-disk-1   0.00     3.67    0.07    2.20     0.27    23.47    20.94     0.00    0.76    0.00    0.79   0.24   0.05
rbd_hdd/vm-146-disk-2   0.00     0.00    0.00    0.07     0.00     0.27     8.00     0.00    0.00    0.00    0.00   0.00   0.00
rbd_hdd/vm-149-disk-1   0.00     9.50    0.07    9.33     0.27    75.33    16.09     0.01    1.09    0.00    1.10   0.55   0.52
rbd_hdd/vm-150-disk-1   0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
rbd_hdd/vm-152-disk-1   0.00     0.13    0.07    0.10     0.27     0.93    14.40     0.00    0.00    0.00    0.00   0.00   0.00
rbd_hdd/vm-161-disk-1   0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
rbd_hdd/vm-153-disk-1   0.00    10.00    0.00    7.60     0.00    70.93    18.67     0.02    2.37    0.00    2.37   0.32   0.24
rbd_hdd/vm-165-disk-1   0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
rbd_hdd/vm-166-disk-1   0.00     4.37    0.00    1.63     0.00    24.00    29.39     0.00    2.53    0.00    2.53   0.41   0.07
rbd_hdd/vm-168-disk-1   0.00     2.83    0.00    3.00     0.00    23.33    15.56     0.00    0.93    0.00    0.93   0.36   0.11
rbd_hdd/vm-173-disk-1   0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
rbd_hdd/vm-175-disk-1   0.00     2.87    0.00    1.27     0.00    16.67    26.32     0.00    1.58    0.00    1.58   1.05   0.13
rbd_hdd/vm-182-disk-1   0.00     2.97    0.00    2.60     0.00    22.40    17.23     0.00    0.41    0.00    0.41   0.21   0.05
rbd_hdd/vm-181-disk-1   0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
rbd_hdd/vm-200-disk-1   0.00     2.93    0.00    1.87     0.00    19.47    20.86     0.00    1.50    0.00    1.50   0.43   0.08
rbd_hdd/vm-205-disk-1   0.00     0.13    0.07    0.10     0.27     0.93    14.40     0.00    0.80    0.00    1.33   0.80   0.01
rbd_hdd/vm-202-disk-1   0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
rbd_hdd/vm-206-disk-1   0.00     2.77    0.03    2.50     0.13    21.07    16.74     0.02    8.42    0.00    8.53   7.58   1.92
rbd_hdd/vm-215-disk-1   0.00     1.47    0.00    0.83     0.00     9.20    22.08     0.00    1.28    0.00    1.28   0.64   0.05
rbd_hdd/vm-216-disk-1   0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
rbd_hdd/vm-220-disk-1   0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
rbd_hdd/vm-221-disk-1   0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
rbd_hdd/vm-222-disk-1   0.00     7.80    0.00    6.63     0.00    57.73    17.41     0.01    1.49    0.00    1.49   0.42   0.28
rbd_hdd/vm-223-disk-1   0.00     6.80    0.00    5.20     0.00    48.00    18.46     0.01    1.44    0.00    1.44   0.38   0.20
rbd_hdd/vm-225-disk-1   0.00     1.27    0.17    1.73     0.67    12.13    13.47     0.00    1.96    3.20    1.85   0.56   0.11
rbd_hdd/vm-226-disk-1   0.00     0.17    0.03    0.50     0.13     2.67    10.50     0.00    3.25    0.00    3.47   0.75   0.04
rbd_hdd/vm-259-disk-1   0.00     6.57    0.30    7.07     1.73    54.53    15.28     0.03    4.05   71.11    1.21   2.68   1.97
rbd_hdd/vm-259-disk-2   0.00     3.47    0.00    5.73     0.00    36.80    12.84     0.01    1.00    0.00    1.00   1.00   0.57
rbd_hdd/vm-261-disk-1   0.00     0.73    0.00    1.00     0.00     6.93    13.87     0.00    0.13    0.00    0.13   0.13   0.01
rbd_hdd/vm-260-disk-1   0.00     3.90    0.23    4.53     0.93    33.73    14.55     0.01    2.71   18.29    1.91   1.59   0.76
rbd_hdd/vm-265-disk-1   0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
rbd_hdd/vm-266-disk-1   0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
rbd_hdd/vm-267-disk-1   0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
rbd_hdd/vm-272-disk-1   0.00     0.07    0.03    0.30     0.13     1.47     9.60     0.00    0.40    0.00    0.44   0.40   0.01
rbd_hdd/vm-270-disk-1   0.00     5.93    0.00    2.87     0.00    42.27    29.49     0.01    4.33    0.00    4.33   3.12   0.89
rbd_hdd/vm-273-disk-1   0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
rbd_hdd/vm-274-disk-1   0.00    10.10    0.00    3.57     0.00    54.67    30.65     0.01    1.57    0.00    1.57   0.30   0.11
rbd_hdd/vm-277-disk-1   0.00     0.10    0.03    0.47     0.13     2.27     9.60     0.00    0.80    0.00    0.86   0.53   0.03
rbd_hdd/vm-276-disk-1   0.00     0.00    0.00    0.03     0.00     0.13     8.00     0.00    0.00    0.00    0.00   0.00   0.00
rbd_hdd/vm-279-disk-1   0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
rbd_hdd/vm-278-disk-1   0.00     4.53    0.03    2.67     0.13    28.80    21.43     0.01    2.02    0.00    2.05   0.59   0.16
rbd_hdd/vm-281-disk-1   0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
rbd_hdd/vm-287-disk-1   0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
rbd_hdd/vm-293-disk-1   0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
rbd_hdd/vm-112-disk-1   0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
rbd_hdd/vm-102-disk-1   0.00     0.17    0.07    0.13     0.27     1.20    14.67     0.00    0.00    0.00    0.00   0.00   0.00
rbd_hdd/vm-116-disk-1   0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
rbd_hdd/vm-119-disk-1   0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
rbd_hdd/vm-125-disk-1   0.00     4.47    0.07    3.00     0.93    30.00    20.17     0.01    1.78   26.00    1.24   0.96   0.29
rbd_hdd/vm-118-disk-1   0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
rbd_hdd/vm-129-disk-1   0.00    11.90    0.03    3.40     0.13    63.47    37.05     0.01    1.48    0.00    1.49   0.35   0.12
rbd_hdd/vm-128-disk-1   0.00     0.90    0.00    1.20     0.00     8.40    14.00     0.00    1.78    0.00    1.78   0.33   0.04
rbd_hdd/vm-127-disk-1   0.00     2.13    0.00    1.30     0.00    13.73    21.13     0.00    1.13    0.00    1.13   0.31   0.04
rbd_hdd/vm-130-disk-1   0.00     1.07    0.00    0.57     0.00     6.53    23.06     0.00    0.94    0.00    0.94   0.24   0.01
rbd_hdd/vm-147-disk-1   0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
rbd_hdd/vm-157-disk-1   0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
<snip>
rbd_hdd/vm-158-disk-1   0.00     0.63    0.00    0.80     0.00     5.73    14.33     0.00    1.00    0.00    1.00   0.50   0.04
rbd_hdd/vm-163-disk-1   0.00     2.30    0.07    4.53     0.27    27.33    12.00     0.01    1.16    0.00    1.18   0.55   0.25
rbd_hdd/vm-174-disk-1   0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
rbd_hdd/vm-162-disk-1   0.00     6.63    0.10    4.03     0.40    42.80    20.90     0.01    1.55   18.67    1.12   1.42   0.59
rbd_hdd/vm-177-disk-1   0.00     5.73    0.13    2.53     0.53    33.07    25.20     0.00    1.60   15.00    0.89   1.10   0.29
rbd_hdd/vm-178-disk-1   0.00     0.83    0.03    0.83     0.13     6.67    15.69     0.00    0.77    0.00    0.80   0.46   0.04

David Herselman · Mar 14, 2019

Herewith a refined version:

Code:

#!/bin/bash
time='60';
filter='rbd_hdd';

function getstats() {
  for host in `ls -1A /etc/pve/nodes`; do
    if [[ "$HOSTNAME" == "$host" ]]; then
      iostat -xkdy $time 1 | grep '^rbd' > /tmp/"$host"_iostat &
    else
      ( ssh -o StrictHostKeyChecking=no admin@$host.syrex.co.za "iostat -xkdy $time 1 | grep '^rbd'" > /tmp/"$host"_iostat ) &
    fi
  done
  sleep $[$time+5];
}

function getrbdnames() {
  rm -f /tmp/io_stats;
  for file in /tmp/*_iostat; do
    temp=${file#/tmp/};
    host=${temp%_iostat};
    if [[ "$HOSTNAME" == "$host" ]]; then
      rbd showmapped | grep -P '^\d+\s+' | perl -pe 's/\d+\s+(\S+)\s+(\S+).*\/dev\/(\S+).*/\3 \1\\\\\/\2/g' | while read dev name; do
        perl -i -pe "s/^$dev\s+/$name\t/g" $file;
      done
    else
      ssh -o StrictHostKeyChecking=no admin@$host.syrex.co.za "rbd showmapped" | grep -P '^\d+\s+' | perl -pe 's/\d+\s+(\S+)\s+(\S+).*\/dev\/(\S+).*/\3 \1\\\\\/\2/g' | while read dev name; do
        perl -i -pe "s/^$dev\s+/$name\t/g" $file;
      done
    fi
    grep $filter $file >> /tmp/io_stats;
    rm -f $file;
  done
}

function getvmnames() {
  cat /tmp/io_stats | while read block other; do
    temp=${block#*/*-};
    id=${temp%-disk*};
    temp=`grep -m1 name /etc/pve/nodes/*/qemu-server/$id.conf 2> /dev/null`;
    [ $? -ne 0 ] && temp='name: unknown';
    block=`echo $block | perl -nle 'print quotemeta'`;
    name=`printf "%-30s" ${temp#name: } | perl -nle 'print quotemeta'`;
    perl -i -pe "s/^($block)/$name\t\1/g" /tmp/io_stats;
  done
}

function showtop() {
  for fields in \
    '3 rrqm merged read requests per second' \
    '5 r/s reads per second' \
    '7 rkB/s KB read per second' \
    '4 wrqm merged write requests per second' \
    '6 w/s writes per second' \
    '8 wkB/s KB written per second'; do
    echo $fields | while read column name desc; do
      echo -e "Top $name ($desc):";
      echo "vm_name                         ceph_rbd_image        rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util";
      sort -g -k $column -r /tmp/io_stats | head;
      echo -e "\n";
    done
  done
}

case "$1" in
  g|-g|--get)
    getstats;
    getrbdnames; # Replace block device references and consolidate
    getvmnames;  # Prefix VM name
  ;;
  t|-t|--top)
    showtop;
  ;;
  *)
    echo "Argument does not define any operation."
    echo "";
    echo "g|-g|--getGet cluster IO statistics, for $time seconds";
    echo "t|-t|--topShow top VM disc statistics, for last 'get'";
    echo "";
    echo "Auxiliary commands:";
    echo "  for f in 180 235; do echo -n \"\$f: \"; grep name /etc/pve/nodes/*/qemu-server/\$f.conf; done";
  ;;
esac

We run Ceph storage with kernel RBD and primarily seek out guests using spinner pool storage...

Search

Search

Finding guests which generate allot of disk IO

David Herselman

Renowned Member

David Herselman

Renowned Member