Proxmox cluster load high

P

pvpaulo

Member
Jun 15, 2022
56
1
13
Hello, good morning everyone.

Could you help me?

I'm facing a problem where my Proxmox is experiencing a very high load of 28.

However, there were several VMs in this Proxmox.

I thought it was the VMs consuming too much of the virtualizer's processor.

However, since this environment is in a cluster,

I migrated all the VMs to a second Proxmox.

Now the environment has no VMs.

However, even without VMs, the load is high.

I don't know what to do.

The environment is configured with 3 iSCSI LUNs using multipath.

The Proxmox version is 8.2.

My server has plenty of resources, and I also have another Proxmox in a cluster with the same hardware configuration.

However, the other virtualizer is running normally, with all VMs and a normal load.

What could be the problem? The versions of the two Proxmox processors are the same, and the physical hardware resources are also the same.
What could be the problem, and how can I check?





root@ROMA02:~# pveversion -v
proxmox-ve: 8.2.0 (running kernel: 6.8.4-2-pve)
pve-manager: 8.2.2 (running version: 8.2.2/9355359cd7afbae4)
proxmox-kernel-helper: 8.1.0
proxmox-kernel-6.8: 6.8.4-2
proxmox-kernel-6.8.4-2-pve-signed: 6.8.4-2
ceph-fuse: 17.2.7-pve3
corosync: 3.1.7-pve3
criu: 3.17.1-2
glusterfs-client: 10.3-5
ifupdown2: 3.2.0-1+pmx8
ksm-control-daemon: 1.5-1
libjs-extjs: 7.0.0-4
libknet1: 1.28-pve1
libproxmox-acme-perl: 1.5.0
libproxmox-backup-qemu0: 1.4.1
libproxmox-rs-perl: 0.3.3
libpve-access-control: 8.1.4
libpve-apiclient-perl: 3.3.2
libpve-cluster-api-perl: 8.0.6
libpve-cluster-perl: 8.0.6
libpve-common-perl: 8.2.1
libpve-guest-common-perl: 5.1.1
libpve-http-server-perl: 5.1.0
libpve-network-perl: 0.9.8
libpve-rs-perl: 0.8.8
libpve-storage-perl: 8.2.1
libspice-server1: 0.15.1-1
lvm2: 2.03.16-2
lxc-pve: 6.0.0-1
lxcfs: 6.0.0-pve2
novnc-pve: 1.4.0-3
proxmox-backup-client: 3.2.0-1
proxmox-backup-file-restore: 3.2.0-1
proxmox-kernel-helper: 8.1.0
proxmox-mail-forward: 0.2.3
proxmox-mini-journalreader: 1.4.0
proxmox-offline-mirror-helper: 0.6.6
proxmox-widget-toolkit: 4.2.1
pve-cluster: 8.0.6
pve-container: 5.0.10
pve-docs: 8.2.1
pve-edk2-firmware: 4.2023.08-4
pve-esxi-import-tools: 0.7.0
pve-firewall: 5.0.5
pve-firmware: 3.11-1
pve-ha-manager: 4.0.4
pve-i18n: 3.2.2
pve-qemu-kvm: 8.1.5-5
pve-xtermjs: 5.3.0-3
qemu-server: 8.2.1
smartmontools: 7.3-pve1
spiceterm: 3.3.0
swtpm: 0.8.0+pve1
vncterm: 1.8.0
zfsutils-linux: 2.2.3-pve2





pvecm status
Cluster information
-------------------
Name: roma
Config Version: 2
Transport: knet
Secure auth: on

Quorum information
------------------
Date: Fri Nov 28 08:20:01 2025
Quorum provider: corosync_votequorum
Nodes: 2
Node ID: 0x00000001
Ring ID: 1.188d9e
Quorate: Yes

Votequorum information
----------------------
Expected votes: 4
Highest expected: 4
Total votes: 4
Quorum: 1
Flags: 2Node Quorate WaitForAll

Membership information
----------------------
Nodeid Votes Name
0x00000001 2 200.201.235.106 (local)
0x00000002 2 200.201.235.107
 
First, upgrade. I've had high load on empty / low use PVEs before on HPE servers, until I upgraded to 9.0. We're now at 9.1, upgrade. You version is more than 1.5y old, a LOT has been fixed and improved since.

If this is a production system of if it's important in any capacity, get a support subscription.

Fabián Rodríguez | Le Goût du Libre Inc. | Montreal, Canada | Mastodon
Proxmox Silver Partner, server and desktop enterprise support in French, English and Spanish
 
  • Like
Reactions: UdoB
This is a productive client environment.
They don't want to provide support.
I have a cluster with 2 physical Proxmox processors.
This is my processor:
CPU(s): 56
Intel(R) Xeon(R) CPU E5-2680 v4 @ 2.40GHz

Is there any solution?
Can I update to version 9?
Is there a risk of losing the VMs?
What would be the ideal procedure to correct this?

Or is there a way to fix it and keep it in this version?
 
pvpaulo said:
This is a productive client environment.
They don't want to provide support.
I have a cluster with 2 physical Proxmox processors.
This is my processor:
CPU(s): 56
Intel(R) Xeon(R) CPU E5-2680 v4 @ 2.40GHz

Is there any solution?
Can I update to version 9?
Is there a risk of losing the VMs?
What would be the ideal procedure to correct this?

Or is there a way to fix it and keep it in this version?
Click to expand...

A few ideas before attempting an upgrade :

- Make sure all three systems are running the latest in Proxmox 8.x series, and booting into the same kernel version:

Code: 
apt-get update && apt-get dist-upgrade -y

This will NOT upgrade to 9.1, for now in my opinion it's best to try and find the problem before upgrading.

A single Proxmox host can have a bad iSCSI path, a flaky switch port, a kernel regression, or blocked I/O tasks while the others stay healthy. High load with no VMs almost always comes from I/O wait on that specific node. Search for “Proxmox high load I/O wait” or “multipath troubleshooting” and follow the usual diagnostics.

BTW, running a 2‑node Proxmox cluster is generally not good practice unless you add a QDevice. It's not the cause of high load, though.
 
Usually very high load shows problems with storage, eg disks cannot write fast enough. First look at disks.
 
This problem occurred after I added a Proxmox Backup Server VM.
After it performed the backups, I had this problem.
I don't know if it was a coincidence or if that caused the situation.

I will look for a way to troubleshoot it.
 
iotop-c -cP
root@ROMA02:~# top -H -b -n1 | head -n 50
top - 09:05:26 up 4 days, 8:02, 2 users, load average: 28.22, 28.14, 28.10
Threads: 756 total, 1 running, 755 sleeping, 0 stopped, 0 zombie
%Cpu(s): 50.5 us, 0.9 sy, 0.0 ni, 48.6 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
MiB Mem : 257698.5 total, 245555.7 free, 13024.0 used, 562.3 buff/cache
MiB Swap: 8192.0 total, 8192.0 free, 0.0 used. 244674.4 avail Mem

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
1603022 root 20 0 11620 4928 3136 R 21.1 0.0 0:00.07 top
1288603 root 20 0 4813260 3.1g 11648 S 5.3 1.2 94:28.27 kvm
1288755 root 20 0 4813260 3.1g 11648 S 5.3 1.2 32:20.12 CPU 0/K+
1288756 root 20 0 4813260 3.1g 11648 S 5.3 1.2 28:51.68 CPU 1/K+
1 root 20 0 168716 11648 8960 S 0.0 0.0 0:14.37 systemd
2 root 20 0 0 0 0 S 0.0 0.0 0:00.21 kthreadd
3 root 20 0 0 0 0 S 0.0 0.0 0:00.00 pool_wo+
4 root 0 -20 0 0 0 I 0.0 0.0 0:00.00 kworker+
5 root 0 -20 0 0 0 I 0.0 0.0 0:00.00 kworker+
6 root 0 -20 0 0 0 I 0.0 0.0 0:00.00 kworker+
7 root 0 -20 0 0 0 I 0.0 0.0 0:00.00 kworker+
9 root 0 -20 0 0 0 I 0.0 0.0 0:00.00 kworker+
12 root 0 -20 0 0 0 I 0.0 0.0 0:00.00 kworker+
13 root 20 0 0 0 0 I 0.0 0.0 0:00.00 rcu_tas+
14 root 20 0 0 0 0 I 0.0 0.0 0:00.00 rcu_tas+
15 root 20 0 0 0 0 I 0.0 0.0 0:00.00 rcu_tas+
16 root 20 0 0 0 0 S 0.0 0.0 0:00.01 ksoftir+
17 root 20 0 0 0 0 I 0.0 0.0 2:10.39 rcu_pre+
18 root rt 0 0 0 0 S 0.0 0.0 0:00.74 migrati+
19 root -51 0 0 0 0 S 0.0 0.0 0:00.00 idle_in+
20 root 20 0 0 0 0 S 0.0 0.0 0:00.00 cpuhp/0
21 root 20 0 0 0 0 S 0.0 0.0 0:00.00 cpuhp/1
22 root -51 0 0 0 0 S 0.0 0.0 0:00.00 idle_in+
23 root rt 0 0 0 0 S 0.0 0.0 0:02.34 migrati+
24 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ksoftir+
25 root 20 0 0 0 0 I 0.0 0.0 0:00.00 kworker+
26 root 0 -20 0 0 0 I 0.0 0.0 0:00.00 kworker+
27 root 20 0 0 0 0 S 0.0 0.0 0:00.00 cpuhp/2
28 root -51 0 0 0 0 S 0.0 0.0 0:00.00 idle_in+
29 root rt 0 0 0 0 S 0.0 0.0 0:02.29 migrati+
30 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ksoftir+
32 root 0 -20 0 0 0 I 0.0 0.0 0:00.00 kworker+
33 root 20 0 0 0 0 S 0.0 0.0 0:00.00 cpuhp/3
34 root -51 0 0 0 0 S 0.0 0.0 0:00.00 idle_in+
35 root rt 0 0 0 0 S 0.0 0.0 0:02.34 migrati+
36 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ksoftir+
38 root 0 -20 0 0 0 I 0.0 0.0 0:00.00 kworker+
39 root 20 0 0 0 0 S 0.0 0.0 0:00.00 cpuhp/4
40 root -51 0 0 0 0 S 0.0 0.0 0:00.00 idle_in+
41 root rt 0 0 0 0 S 0.0 0.0 0:02.30 migrati+
42 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ksoftir+
43 root 20 0 0 0 0 I 0.0 0.0 0:00.00 kworker+
44 root 0 -20 0 0 0 I 0.0 0.0 0:00.00 kworker+
root@ROMA02:~# ps -eo pid,stat,comm | grep " D"
root@ROMA02:~# ps aux | awk '$8=="D"'
root@ROMA02:~# dmesg | grep -i -E "blk|scsi|error|fail|multipath|timed out|reset"
[ 1.865307] ACPI: _OSC evaluation for CPUs failed, trying _PDC
[ 1.949534] SCSI subsystem initialized
[ 1.988326] pci 0000:03:00.0: ROM [mem size 0x00100000 pref]: failed to assign
[ 1.988374] pci 0000:07:00.1: ROM [mem size 0x00080000 pref]: failed to assign
[ 2.084579] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 243)
[ 2.098883] ERST: Error Record Serialization Table (ERST) support is initialized.
[ 2.492031] RAS: Correctable Errors collector initialized.
[ 2.861138] scsi host1: ahci
[ 2.861342] scsi host2: ahci
[ 2.861534] scsi host3: ahci
[ 2.861680] scsi host4: ahci
[ 2.889890] scsi host5: ahci
[ 2.890032] scsi host6: ahci
[ 2.890162] scsi host7: ahci
[ 2.890370] scsi host8: ahci
[ 2.890493] scsi host9: ahci
[ 2.890621] scsi host10: ahci
[ 3.191262] megaraid_sas 0000:03:00.0: Online Controller Reset(OCR) : Enabled
[ 3.191271] megaraid_sas 0000:03:00.0: FW provided TM TaskAbort/Reset timeout : 0 secs/0 secs
[ 3.241795] scsi host0: Avago SAS based MegaRAID driver
[ 3.270604] scsi 0:2:0:0: Direct-Access DELL PERC H330 Mini 4.29 PQ: 0 ANSI: 5
[ 3.305222] sd 0:2:0:0: Attached scsi generic sg0 type 0
[ 3.309671] sd 0:2:0:0: [sda] Attached SCSI disk
[ 6.585876] systemd[1]: systemd 252.30-1~deb12u2 running in system mode (+PAM +AUDIT +SELINUX +APPARMOR +IMA +SMACK +SECCOMP +GCRYPT -GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 -IDN +IPTC +KMOD +LIBCRYPTSETUP +LIBFDISK +PCRE2 -PWQUALITY +P11KIT +QRENCODE +TPM2 +BZIP2 +LZ4 +XZ +ZLIB +ZSTD -BPF_FRAMEWORK -XKBCOMMON +UTMP +SYSVINIT default-hierarchy=unified)
[ 6.776754] systemd[1]: Listening on multipathd.socket - multipathd control socket.
[ 6.844878] systemd[1]: Starting multipathd.service - Device-Mapper Multipath Device Controller...
[ 6.897852] systemd[1]: Started multipathd.service - Device-Mapper Multipath Device Controller.
[ 9.868627] ACPI Error: AE_NOT_EXIST, Returned by Handler for [IPMI] (20230628/evregion-300)
[ 9.868633] ACPI Error: Region IPMI (ID=7) has no handler (20230628/exfldio-261)
[ 9.868646] ACPI Error: Aborting method \_SB.PMI0._GHL due to previous error (AE_NOT_EXIST) (20230628/psparse-529)
[ 9.868655] ACPI Error: Aborting method \_SB.PMI0._PMC due to previous error (AE_NOT_EXIST) (20230628/psparse-529)
[ 9.868663] ACPI: \_SB_.PMI0: _PMC evaluation failed: AE_NOT_EXIST
[ 12.883398] Loading iSCSI transport class v2.0-870.
[ 12.908004] iscsi: registered transport (tcp)
[ 45.997093] scsi host11: iSCSI Initiator over TCP/IP
[ 46.007770] scsi host12: iSCSI Initiator over TCP/IP
[ 46.018339] scsi host13: iSCSI Initiator over TCP/IP
[ 46.029282] scsi host14: iSCSI Initiator over TCP/IP
[ 46.040109] scsi host15: iSCSI Initiator over TCP/IP
[ 46.051125] scsi host16: iSCSI Initiator over TCP/IP
[ 46.062079] scsi host17: iSCSI Initiator over TCP/IP
[ 46.073126] scsi host18: iSCSI Initiator over TCP/IP
[ 46.085984] scsi 11:0:0:253: Direct-Access PURE FlashArray 8888 PQ: 0 ANSI: 6
[ 46.086486] scsi 12:0:0:253: Direct-Access PURE FlashArray 8888 PQ: 0 ANSI: 6
[ 46.086692] scsi 13:0:0:253: Direct-Access PURE FlashArray 8888 PQ: 0 ANSI: 6
[ 46.087310] scsi 14:0:0:253: Direct-Access PURE FlashArray 8888 PQ: 0 ANSI: 6
[ 46.087980] scsi 15:0:0:254: Direct-Access PURE FlashArray 8888 PQ: 0 ANSI: 6
[ 46.088367] scsi 16:0:0:254: Direct-Access PURE FlashArray 8888 PQ: 0 ANSI: 6
[ 46.088942] scsi 17:0:0:254: Direct-Access PURE FlashArray 8888 PQ: 0 ANSI: 6
[ 46.089006] scsi 18:0:0:254: Direct-Access PURE FlashArray 8888 PQ: 0 ANSI: 6
[ 46.092299] sd 12:0:0:253: Attached scsi generic sg1 type 0
[ 46.092470] sd 14:0:0:253: Attached scsi generic sg2 type 0
[ 46.092736] sd 13:0:0:253: Attached scsi generic sg3 type 0
[ 46.092939] sd 15:0:0:254: Attached scsi generic sg4 type 0
[ 46.093118] scsi 14:0:0:254: Direct-Access PURE FlashArray 8888 PQ: 0 ANSI: 6
[ 46.093199] scsi 12:0:0:254: Direct-Access PURE FlashArray 8888 PQ: 0 ANSI: 6
[ 46.093450] scsi 13:0:0:254: Direct-Access PURE FlashArray 8888 PQ: 0 ANSI: 6
[ 46.094658] sd 11:0:0:253: Attached scsi generic sg5 type 0
[ 46.095787] sd 17:0:0:254: Attached scsi generic sg6 type 0
[ 46.096430] scsi 11:0:0:254: Direct-Access PURE FlashArray 8888 PQ: 0 ANSI: 6
[ 46.098233] sd 16:0:0:254: Attached scsi generic sg7 type 0
[ 46.102145] sd 13:0:0:254: Attached scsi generic sg8 type 0
[ 46.102260] sd 12:0:0:254: Attached scsi generic sg9 type 0
[ 46.102674] sd 18:0:0:254: Attached scsi generic sg10 type 0
[ 46.102887] sd 14:0:0:254: Attached scsi generic sg11 type 0
[ 46.102917] sd 17:0:0:254: [sdf] Attached SCSI disk
[ 46.105233] sd 11:0:0:254: Attached scsi generic sg12 type 0
[ 46.106415] sd 15:0:0:254: [sde] Attached SCSI disk
[ 46.106877] sd 16:0:0:254: [sdh] Attached SCSI disk
[ 46.111292] sd 13:0:0:253: [sdb] Attached SCSI disk
[ 46.111575] sd 12:0:0:253: [sdc] Attached SCSI disk
[ 46.113352] sd 14:0:0:253: [sdd] Attached SCSI disk
[ 46.117924] sd 18:0:0:254: [sdj] Attached SCSI disk
[ 46.119044] sd 11:0:0:253: [sdg] Attached SCSI disk
[ 46.126582] sd 13:0:0:254: [sdi] Attached SCSI disk
[ 46.130548] sd 11:0:0:254: [sdm] Attached SCSI disk
[ 46.134076] sd 12:0:0:254: [sdl] Attached SCSI disk
[ 46.148793] sd 14:0:0:254: [sdk] Attached SCSI disk
[ 46.192599] device-mapper: multipath round-robin: version 1.2.0 loaded
root@ROMA02:~# multipath -ll
MATRIX-SAN4T (3624a93707e6ebe82bf5047b9000325a2) dm-7 PURE,FlashArray
size=4.0T features='1 queue_if_no_path' hwhandler='1 alua' wp=rw
`-+- policy='round-robin 0' prio=50 status=active
|- 13:0:0:253 sdb 8:16 active ready running
|- 14:0:0:253 sdd 8:48 active ready running
|- 12:0:0:253 sdc 8:32 active ready running
`- 11:0:0:253 sdg 8:96 active ready running
MATRIX-SAN5T (3624a93709b2e3b0a952b4d380001f6c3) dm-6 PURE,FlashArray
size=5.0T features='1 queue_if_no_path' hwhandler='1 alua' wp=rw
`-+- policy='round-robin 0' prio=50 status=active
|- 16:0:0:254 sdh 8:112 active ready running
|- 15:0:0:254 sde 8:64 active ready running
|- 18:0:0:254 sdj 8:144 active ready running
`- 17:0:0:254 sdf 8:80 active ready running
MATRIX-SAN5T_2 (3624a93707e6ebe82bf5047b90001eb5a) dm-8 PURE,FlashArray
size=5.0T features='1 queue_if_no_path' hwhandler='1 alua' wp=rw
`-+- policy='round-robin 0' prio=50 status=active
|- 13:0:0:254 sdi 8:128 active ready running
|- 11:0:0:254 sdm 8:192 active ready running
|- 12:0:0:254 sdl 8:176 active ready running
`- 14:0:0:254 sdk 8:160 active ready running
root@ROMA02:~# pidstat -u 1 10
Linux 6.8.4-2-pve (ROMA02) 12/02/2025 _x86_64_ (56 CPU)

09:07:50 AM UID PID %usr %system %guest %wait %CPU CPU Command
09:07:51 AM 0 2323 0.00 0.94 0.00 0.00 0.94 46 pmxcfs
09:07:51 AM 0 2670 0.00 0.94 0.00 0.00 0.94 55 corosync
09:07:51 AM 0 1288603 0.00 1.89 4.72 0.00 5.66 34 kvm
09:07:51 AM 0 1599907 0.00 0.94 0.00 0.00 0.94 41 kworker/u112:4-iscsi_q_18
09:07:51 AM 0 1603651 1.89 2.83 0.00 0.00 4.72 35 pidstat

root@ROMA02:~# ps -eo pid,tid,comm,pcpu | egrep "kworker|ksoftirqd|rcu" | sort -k4 -nr | head -n 40
9 9 kworker/0:0H-ev 0.0
98 98 kworker/13:0H-e 0.0
97 97 kworker/13:0-mm 0.0
976 976 kworker/49:1H-k 0.0
972 972 kworker/47:1H-k 0.0
96 96 ksoftirqd/13 0.0
963 963 kworker/27:1H-k 0.0
952 952 kworker/19:1H-k 0.0
92 92 kworker/12:0H-e 0.0
91 91 kworker/12:0-ev 0.0
916 916 kworker/26:1H-k 0.0
915 915 kworker/6:2-eve 0.0
90 90 ksoftirqd/12 0.0
888 888 kworker/R-kmpat 0.0
887 887 kworker/R-kmpat 0.0
86 86 kworker/11:0H-e 0.0
85 85 kworker/11:0-mm 0.0
856 856 kworker/24:1H-k 0.0
84 84 ksoftirqd/11 0.0
832 832 kworker/45:1H-k 0.0
823 823 kworker/R-ext4- 0.0
819 819 kworker/33:1H-k 0.0
80 80 kworker/10:0H-e 0.0
79 79 kworker/10:0-mm 0.0
78 78 ksoftirqd/10 0.0
780 780 kworker/R-kdmfl 0.0
7 7 kworker/R-netns 0.0
779 779 kworker/R-dm-th 0.0
778 778 kworker/R-kcopy 0.0
777 777 kworker/R-kdmfl 0.0
751 751 kworker/R-kdmfl 0.0
750 750 kworker/R-kdmfl 0.0
749 749 kworker/3:2-eve 0.0
748 748 kworker/R-dm_bu 0.0
74 74 kworker/9:0H-ev 0.0
745354 745354 kworker/52:0-cg 0.0
742 742 kworker/25:1H 0.0
739 739 kworker/R-kdmfl 0.0
738 738 kworker/R-kdmfl 0.0
73 73 kworker/9:0-eve 0.0

multipath -ll
MATRIX-SAN4T (3624a93707e6ebe82bf5047b9000325a2) dm-7 PURE,FlashArray
size=4.0T features='1 queue_if_no_path' hwhandler='1 alua' wp=rw
`-+- policy='round-robin 0' prio=50 status=active
|- 13:0:0:253 sdb 8:16 active ready running
|- 14:0:0:253 sdd 8:48 active ready running
|- 12:0:0:253 sdc 8:32 active ready running
`- 11:0:0:253 sdg 8:96 active ready running
MATRIX-SAN5T (3624a93709b2e3b0a952b4d380001f6c3) dm-6 PURE,FlashArray
size=5.0T features='1 queue_if_no_path' hwhandler='1 alua' wp=rw
`-+- policy='round-robin 0' prio=50 status=active
|- 16:0:0:254 sdh 8:112 active ready running
|- 15:0:0:254 sde 8:64 active ready running
|- 18:0:0:254 sdj 8:144 active ready running
`- 17:0:0:254 sdf 8:80 active ready running
MATRIX-SAN5T_2 (3624a93707e6ebe82bf5047b90001eb5a) dm-8 PURE,FlashArray
size=5.0T features='1 queue_if_no_path' hwhandler='1 alua' wp=rw
`-+- policy='round-robin 0' prio=50 status=active
|- 13:0:0:254 sdi 8:128 active ready running
|- 11:0:0:254 sdm 8:192 active ready running
|- 12:0:0:254 sdl 8:176 active ready running
`- 14:0:0:254 sdk 8:160 active ready running




I ran some tests, but I didn't find any trace of a multipath problem.
I don't know what else to do.

I've already restarted the server.

However, the loading speed still increases considerably.
 
I consider this unreadable. Please use code blocks so the formatting is preserved.

1764678441409.png
 
You must log in or register to reply here.
Top Bottom