proxmox 7.2-11 problem with 'pvesh' command speed response.

[Szymons]

Hello,

I have problem with 'pvesh' command speed response.
My cluster with 12x nodes with same kernel/proxmox version :

CPU(s) 64 x AMD EPYC 7513 32-Core Processor (2 Sockets)
Kernel Version Linux 5.19.7-2-pve #1 SMP PREEMPT_DYNAMIC PVE 5.19.7-2 (Tue, 04 Oct 2022 17:18:40 +
PVE Manager Version pve-manager/7.2-11/b76d3178

I prepared little test script in bash :

root@HV-BASE-C-01:~# cat TIME.sh
#!/bin/bash
while [ true ]
do
DATE1=`date +%s`
pvesh get /nodes/HV-BASE-C-01/qemu --full --output-format=json > /dev/null
DATE2=`date +%s`
DATE3=`expr $DATE2 - $DATE1`
echo $DATE3
done

and results are not very satisficating about 5 seconds

root@HV-BASE-C-01:~# ./TIME.sh
5
6
6
4
5
4
6
^C

How Can I debug what is wrong ? or maybe is there any option that slows down respose ?

 

[fiona]

Hi,
pvesh is written in Perl and not really designed for efficiency. But the bigger issue is that with --full, each running VM needs to be queried individually for the full status. How many running VMs do you have approximately? You could take a look at using an API client instead of pvesh, but again, --full is not cheap

 

[Szymons]

I had few VM runing but on 2 of them I had IOPS test turn on with fio program. When I stop them response came back to normal state.

 

[fiona]

With --full QMP commands are used to query the QEMU process. And if the QEMU main loop is busy doing IO, it can take a bit to get the QMP response. You could try enabling the iothread setting for the disks (also requires setting the SCSI controller to VirtIO SCSI single for SCSI disks), then IO is done in it's own thread.