Hello everyone,
we have a cluster with 5 nodes and 140 LXC/VMs (around 90 running).
Recently we have migrated to a new NAS storage (TrueNAS with 3 x 3 vdevs, Raid Z1, 4TB disks).
Also, at the same time we have upgraded all Proxmox nodes to run on 8.0.3.
We are mounting the storage via ISCSI on a separate 100Gbit network.
There are 5 volumes each 2TB, shared over the iSCSI, on which is created an LVM.
The LXC/VMs are then created on the LVMs.
Most of the LXC containers are Debian. There are around 10-20 VMs that are Windows.
On the VMs, I notice are small spikes over the time as in the pictures below:
Sometimes the spikes are going up to 1GB read/write.
On the host itself, I cannot see this as a network traffic.
This is the host network traffic over the same period of time.
This means that the IO spikes are locally on the VMs.
What may cause this issue?
I don't know where to look next, I've tried tcpdump/Wireshark on the host (because the ISCSI is network-mounted),and I cannot corelate the spikes.
Any suggestions? I don't know if its because of the NAS or the Proxmox upgrade to v8.
On the NAS itself, I cannot notice any unusual spikes on the network or disks.
we have a cluster with 5 nodes and 140 LXC/VMs (around 90 running).
Recently we have migrated to a new NAS storage (TrueNAS with 3 x 3 vdevs, Raid Z1, 4TB disks).
Also, at the same time we have upgraded all Proxmox nodes to run on 8.0.3.
We are mounting the storage via ISCSI on a separate 100Gbit network.
There are 5 volumes each 2TB, shared over the iSCSI, on which is created an LVM.
The LXC/VMs are then created on the LVMs.
Most of the LXC containers are Debian. There are around 10-20 VMs that are Windows.
On the VMs, I notice are small spikes over the time as in the pictures below:
Sometimes the spikes are going up to 1GB read/write.
On the host itself, I cannot see this as a network traffic.
This is the host network traffic over the same period of time.
This means that the IO spikes are locally on the VMs.
What may cause this issue?
I don't know where to look next, I've tried tcpdump/Wireshark on the host (because the ISCSI is network-mounted),and I cannot corelate the spikes.
Any suggestions? I don't know if its because of the NAS or the Proxmox upgrade to v8.
On the NAS itself, I cannot notice any unusual spikes on the network or disks.
