I have a strange situation using LVM-Thin on my Proxmox install.
I have guest VM (Ubuntu 18.04), with vdb drive (Proxmox LVM-Thin volume, dm-42), storage is RAID 10 by 4 SATA drives.
And when I compared IO statistics provided host and guest about this device I show a difference in ops/s.
Metrics by guest VM:
I have suspicion (based by my application workload, which running in guest VM and high iowait) that there is trouble with I/O performance.
100-200ops/s is not very much for my raid and 5 megabytes/s is normal. There is other VMs on this host, but they have a very low disk usage. But 1200 ops/s is not very good and can cause me trouble
Can someone explain very big difference in measurements and how to fix it?
Host:
2x Intel Xeon e5-2650v4/ 128Gb / Adaptec 6405 raid 10 by 4xWD1002FAEX, storage LVM-Thin
Guest:
Ubuntu 18.04, 8CPUs /24 Gb / Virtio drive
# pveversion -v
proxmox-ve: 5.2-2 (running kernel: 4.15.18-1-pve)
pve-manager: 5.2-5 (running version: 5.2-5/eb24855a)
pve-kernel-4.15: 5.2-4
pve-kernel-4.13: 5.2-2
pve-kernel-4.15.18-1-pve: 4.15.18-16
pve-kernel-4.15.17-1-pve: 4.15.17-9
pve-kernel-4.13.16-4-pve: 4.13.16-51
pve-kernel-4.13.16-2-pve: 4.13.16-48
pve-kernel-4.13.16-1-pve: 4.13.16-46
pve-kernel-4.13.13-2-pve: 4.13.13-33
corosync: 2.4.2-pve5
criu: 2.11.1-1~bpo90
glusterfs-client: 3.8.8-1
ksm-control-daemon: 1.2-2
libjs-extjs: 6.0.1-2
libpve-access-control: 5.0-8
libpve-apiclient-perl: 2.0-5
libpve-common-perl: 5.0-35
libpve-guest-common-perl: 2.0-17
libpve-http-server-perl: 2.0-9
libpve-storage-perl: 5.0-24
libqb0: 1.0.1-1
lvm2: 2.02.168-pve6
lxc-pve: 3.0.0-3
lxcfs: 3.0.0-1
novnc-pve: 1.0.0-1
proxmox-widget-toolkit: 1.0-19
pve-cluster: 5.0-28
pve-container: 2.0-24
pve-docs: 5.2-4
pve-firewall: 3.0-13
pve-firmware: 2.0-5
pve-ha-manager: 2.0-5
pve-i18n: 1.0-6
pve-libspice-server1: 0.12.8-3
pve-qemu-kvm: 2.11.2-1
pve-xtermjs: 1.0-5
qemu-server: 5.0-29
smartmontools: 6.5+svn4324-1
spiceterm: 3.0-5
vncterm: 1.5-3
zfsutils-linux: 0.7.9-pve1~bpo9
I have guest VM (Ubuntu 18.04), with vdb drive (Proxmox LVM-Thin volume, dm-42), storage is RAID 10 by 4 SATA drives.
And when I compared IO statistics provided host and guest about this device I show a difference in ops/s.
Metrics by guest VM:
- provided by iostat: drive w/s average is about 140
5th column delta from /sys/class/block/vdb/stat shows about same values - load is about 5.5MB/s
- iowait is about 5%
- CPU usage about 20 - 25%
- provided by iostat: drive w/s average is about 1200
5th column delta from /sys/class/block/dm-42/stat shows about same values - load is about 5.5MB/s
- iowait is about 0.2%
I have suspicion (based by my application workload, which running in guest VM and high iowait) that there is trouble with I/O performance.
100-200ops/s is not very much for my raid and 5 megabytes/s is normal. There is other VMs on this host, but they have a very low disk usage. But 1200 ops/s is not very good and can cause me trouble
Can someone explain very big difference in measurements and how to fix it?
Host:
2x Intel Xeon e5-2650v4/ 128Gb / Adaptec 6405 raid 10 by 4xWD1002FAEX, storage LVM-Thin
Guest:
Ubuntu 18.04, 8CPUs /24 Gb / Virtio drive
# pveversion -v
proxmox-ve: 5.2-2 (running kernel: 4.15.18-1-pve)
pve-manager: 5.2-5 (running version: 5.2-5/eb24855a)
pve-kernel-4.15: 5.2-4
pve-kernel-4.13: 5.2-2
pve-kernel-4.15.18-1-pve: 4.15.18-16
pve-kernel-4.15.17-1-pve: 4.15.17-9
pve-kernel-4.13.16-4-pve: 4.13.16-51
pve-kernel-4.13.16-2-pve: 4.13.16-48
pve-kernel-4.13.16-1-pve: 4.13.16-46
pve-kernel-4.13.13-2-pve: 4.13.13-33
corosync: 2.4.2-pve5
criu: 2.11.1-1~bpo90
glusterfs-client: 3.8.8-1
ksm-control-daemon: 1.2-2
libjs-extjs: 6.0.1-2
libpve-access-control: 5.0-8
libpve-apiclient-perl: 2.0-5
libpve-common-perl: 5.0-35
libpve-guest-common-perl: 2.0-17
libpve-http-server-perl: 2.0-9
libpve-storage-perl: 5.0-24
libqb0: 1.0.1-1
lvm2: 2.02.168-pve6
lxc-pve: 3.0.0-3
lxcfs: 3.0.0-1
novnc-pve: 1.0.0-1
proxmox-widget-toolkit: 1.0-19
pve-cluster: 5.0-28
pve-container: 2.0-24
pve-docs: 5.2-4
pve-firewall: 3.0-13
pve-firmware: 2.0-5
pve-ha-manager: 2.0-5
pve-i18n: 1.0-6
pve-libspice-server1: 0.12.8-3
pve-qemu-kvm: 2.11.2-1
pve-xtermjs: 1.0-5
qemu-server: 5.0-29
smartmontools: 6.5+svn4324-1
spiceterm: 3.0-5
vncterm: 1.5-3
zfsutils-linux: 0.7.9-pve1~bpo9