VM IOPS Monitor / Measuring

[zhoid]

Hi,

I have 2 proxmox clusters connected to a Unity SAN via 10Gbps Fibre networking, using NFS Shares on the SAN

After weeks of troubleshooting random latency issues to a number of VM workloads which I thought stemmed from the SAN it appears that a specific filesystem is under stain.

I have localized the NFS file system but I am unable to identify which VM workloads could be potentially using allot of IOPS

I have already limited each VM disk bandwidth to 10MB/s and IOPS 100 io/s.

Checked each VM from the dashboard view to see which VM is using an unusual amount of disk IO Bandwidth

I am still unable to troubleshoot this successfully.

Any input or advise would be greatly appreciated.

 

[bbgeek17]

Have a look into "netdata" - a very nice product that provides a lot of intelligence in an easy to use UI.
Obviously you can collect any of that statistics directly as well.

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[zhoid]

Have a look into "netdata" - a very nice product that provides a lot of intelligence in an easy to use UI.
Obviously you can collect any of that statistics directly as well.

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

Thanks for your reply, no I have not looked at "netdata"

Does the FREE version of "netdata" provide sufficient information to troubleshoot these issues?

How is it integrated with Proxmox environments? API ? or agent that needs to run on the host?

Thanks

 

[bbgeek17]

Does the FREE version of "netdata" provide sufficient information to troubleshoot these issues?

It may. When the problem is relatively unknown/unclear - one will have to use the tool to find out whether its helpful. We used basic netdata install for Openstack troubleshooting - an arguably much more complex system than PVE (which is neither bad or good).

How is it integrated with Proxmox environments? API ? or agent that needs to run on the host?

https://forum.proxmox.com/threads/installing-netdata.52711/
https://forum.proxmox.com/threads/proxmox-and-netdata.111925/

yes, an agent is involved - Proxmox is a Debian/KVM for all netdata cares.

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[zhoid]

It may. When the problem is relatively unknown/unclear - one will have to use the tool to find out whether its helpful. We used basic netdata install for Openstack troubleshooting - an arguably much more complex system than PVE (which is neither bad or good).

https://forum.proxmox.com/threads/installing-netdata.52711/
https://forum.proxmox.com/threads/proxmox-and-netdata.111925/

yes, an agent is involved - Proxmox is a Debian/KVM for all netdata cares.

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

Thanks for this, installed on 2 hosts and it does seem to collect allot of Data but does not collect the disk IO data for VM's
Most of the VM's I don't have access to so was hoping to get the IOPS information for each VM from the host.

 

[bbgeek17]

Thats the pitfall of using shared file based storage - there is no clear IO isolation - Its just files.
You may get a better insight trying to isolate/monitor "qemu" processes. Just a theory.

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[LnxBil]

Does iotop not work on NFS (never tried)? You can also visualize all kinds of stuff by using a metric server with PVE (e.g. influxdb+grafana works great)