After updating our Proxmox Cluster from 7.3 to 8.1 we are experiencing disk latency increase, here are a graph of disk latency for one of the VM's hosted int eh cluster
First thing we thougth it was a dirver issue but this is happening in all nodes, some are DELL one , other are HP's and even DELL nodes are not the same family.
We have two storage types, SSD (local) and NFS (Using Netapp), this latency increment is only happening in disks hosted in NFS sotorage.
We changed nfs mount options from
To:
And downgraded nfs-common package from 1:2.6.2-4 to 1:1.3.4-6.
After restarting all Nodes disk latency has decreased but not at the same level:
In our soiwtches monitoring we have also detected that unicast traffic has been icreased too:
As we did not need it, IPv6 has been disabled with same results.
This is our current pveversion output:
Does someone has a cloud about what can be happening?
Thank you
First thing we thougth it was a dirver issue but this is happening in all nodes, some are DELL one , other are HP's and even DELL nodes are not the same family.
We have two storage types, SSD (local) and NFS (Using Netapp), this latency increment is only happening in disks hosted in NFS sotorage.
We changed nfs mount options from
Code:
rw,relatime,vers=3,rsize=65536,wsize=65536,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountvers=3,mountport=635,mountproto=udp,local_lock=none
Code:
noatime,nodiratime,rw,relatime,vers=3,rsize=65536,wsize=65536,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountvers=3,mountport=635,mountproto=udp,local_lock=noneAnd downgraded nfs-common package from 1:2.6.2-4 to 1:1.3.4-6.
After restarting all Nodes disk latency has decreased but not at the same level:
In our soiwtches monitoring we have also detected that unicast traffic has been icreased too:
As we did not need it, IPv6 has been disabled with same results.
This is our current pveversion output:
Code:
proxmox-ve: 8.1.0 (running kernel: 6.5.13-1-pve)
pve-manager: 8.1.4 (running version: 8.1.4/ec5affc9e41f1d79)
proxmox-kernel-helper: 8.1.0
pve-kernel-5.15: 7.4-11
pve-kernel-5.13: 7.1-9
proxmox-kernel-6.5.13-1-pve-signed: 6.5.13-1
proxmox-kernel-6.5: 6.5.13-1
pve-kernel-5.4: 6.4-15
pve-kernel-5.15.143-1-pve: 5.15.143-1
pve-kernel-5.15.85-1-pve: 5.15.85-1
pve-kernel-5.15.39-4-pve: 5.15.39-4
pve-kernel-5.13.19-6-pve: 5.13.19-15
pve-kernel-5.4.174-2-pve: 5.4.174-2
pve-kernel-5.4.73-1-pve: 5.4.73-1
ceph-fuse: 16.2.11+ds-2
corosync: 3.1.7-pve3
criu: 3.17.1-2
glusterfs-client: 10.3-5
ifupdown: residual config
ifupdown2: 3.2.0-1+pmx8
ksm-control-daemon: 1.4-1
libjs-extjs: 7.0.0-4
libknet1: 1.28-pve1
libproxmox-acme-perl: 1.5.0
libproxmox-backup-qemu0: 1.4.1
libproxmox-rs-perl: 0.3.3
libpve-access-control: 8.1.1
libpve-apiclient-perl: 3.3.1
libpve-common-perl: 8.1.0
libpve-guest-common-perl: 5.0.6
libpve-http-server-perl: 5.0.5
libpve-network-perl: 0.9.5
libpve-rs-perl: 0.8.8
libpve-storage-perl: 8.0.5
libqb0: 1.0.5-1
libspice-server1: 0.15.1-1
lvm2: 2.03.16-2
lxc-pve: 5.0.2-4
lxcfs: 5.0.3-pve4
novnc-pve: 1.4.0-3
proxmox-backup-client: 3.1.4-1
proxmox-backup-file-restore: 3.1.4-1
proxmox-kernel-helper: 8.1.0
proxmox-mail-forward: 0.2.3
proxmox-mini-journalreader: 1.4.0
proxmox-offline-mirror-helper: 0.6.5
proxmox-widget-toolkit: 4.1.3
pve-cluster: 8.0.5
pve-container: 5.0.8
pve-docs: 8.1.3
pve-edk2-firmware: 4.2023.08-4
pve-firewall: 5.0.3
pve-firmware: 3.9-2
pve-ha-manager: 4.0.3
pve-i18n: 3.2.0
pve-qemu-kvm: 8.1.5-3
pve-xtermjs: 5.3.0-3
qemu-server: 8.0.10
smartmontools: 7.3-pve1
spiceterm: 3.3.0
swtpm: 0.8.0+pve1
vncterm: 1.8.0
zfsutils-linux: 2.2.2-pve2Does someone has a cloud about what can be happening?
Thank you
Last edited: