PVE API timeout GET /api2/json/nodes/{node}/storage/{storage}/content

guilranc

Member
Mar 13, 2023
5
0
6
Hello !
I have troubles with API2 of PVE when i try to GET content of some storages which supporting more approximatively 180/200 images.
The requests "GET /api2/json/nodes/{node}/storage/{storage}/content" made from API requests or pvesh return an error 500 or 596 with a timeout of 30seconds.
But by GUI on the same node and same store the display is ok after 15seconds.
I check logs in /var/log/pveproxy/access.log and see that GUI try a GET with [...]/content?content=images, if i add this eol in my previously requests i constate the return 200 in 15sec also.
I constate this trouble on differents nodes with differents storage (RBD or NFS for example) but alsowith approximatively this limit.

Anyway...

My question is : Is it a known bug or a limitation ? Is there the possibility to increase timeout ? Why this GET is so long ?

I can't modify the requests for example when this GET is called by Terraform or other process... :/


Thanks you for reading and help !

My pveversion :

Code:
proxmox-ve: 8.0.2 (running kernel: 6.2.16-12-pve)
pve-manager: 8.0.4 (running version: 8.0.4/d258a813cfa6b390)
proxmox-kernel-helper: 8.0.3
pve-kernel-5.15: 7.4-5
proxmox-kernel-6.2.16-12-pve: 6.2.16-12
proxmox-kernel-6.2: 6.2.16-12
proxmox-kernel-6.2.16-10-pve: 6.2.16-10
pve-kernel-5.15.111-1-pve: 5.15.111-1
pve-kernel-5.15.30-2-pve: 5.15.30-3
ceph-fuse: 16.2.11+ds-2
corosync: 3.1.7-pve3
criu: 3.17.1-2
glusterfs-client: 10.3-5
ifupdown2: 3.2.0-1+pmx4
ksm-control-daemon: 1.4-1
libjs-extjs: 7.0.0-4
libknet1: 1.25-pve1
libproxmox-acme-perl: 1.4.6
libproxmox-backup-qemu0: 1.4.0
libproxmox-rs-perl: 0.3.1
libpve-access-control: 8.0.5
libpve-apiclient-perl: 3.3.1
libpve-common-perl: 8.0.8
libpve-guest-common-perl: 5.0.4
libpve-http-server-perl: 5.0.4
libpve-network-perl: 0.8.1
libpve-rs-perl: 0.8.5
libpve-storage-perl: 8.0.2
libspice-server1: 0.15.1-1
lvm2: 2.03.16-2
lxc-pve: 5.0.2-4
lxcfs: 5.0.3-pve3
novnc-pve: 1.4.0-2
proxmox-backup-client: 3.0.2-1
proxmox-backup-file-restore: 3.0.2-1
proxmox-kernel-helper: 8.0.3
proxmox-mail-forward: 0.2.0
proxmox-mini-journalreader: 1.4.0
proxmox-offline-mirror-helper: 0.6.2
proxmox-widget-toolkit: 4.0.6
pve-cluster: 8.0.3
pve-container: 5.0.4
pve-docs: 8.0.4
pve-edk2-firmware: 3.20230228-4
pve-firewall: 5.0.3
pve-firmware: 3.8-2
pve-ha-manager: 4.0.2
pve-i18n: 3.0.5
pve-qemu-kvm: 8.0.2-5
pve-xtermjs: 4.16.0-3
qemu-server: 8.0.7
smartmontools: 7.3-pve1
spiceterm: 3.3.0
swtpm: 0.8.0+pve1
vncterm: 1.8.0
zfsutils-linux: 2.1.12-pve1
 
Last edited:
Hello !
I have troubles with API2 of PVE when i try to GET content of some storages which supporting more approximatively 180/200 images.
The requests "GET /api2/json/nodes/{node}/storage/{storage}/content" made from API requests or pvesh return an error 500 or 596 with a timeout of 30seconds.
But by GUI on the same node and same store the display is ok after 15seconds.
I check logs in /var/log/pveproxy/access.log and see that GUI try a GET with [...]/content?content=images, if i add this eol in my previously requests i constate the return 200 in 15sec also.
I constate this trouble on differents nodes with differents storage (RBD or NFS for example) but alsowith approximatively this limit.

Anyway...

My question is : Is it a known bug or a limitation ? Is there the possibility to increase timeout ? Why this GET is so long ?

I can't modify the requests for example when this GET is called by Terraform or other process... :/


Thanks you for reading and help !

My pveversion :

Code:
proxmox-ve: 8.0.2 (running kernel: 6.2.16-12-pve)
pve-manager: 8.0.4 (running version: 8.0.4/d258a813cfa6b390)
proxmox-kernel-helper: 8.0.3
pve-kernel-5.15: 7.4-5
proxmox-kernel-6.2.16-12-pve: 6.2.16-12
proxmox-kernel-6.2: 6.2.16-12
proxmox-kernel-6.2.16-10-pve: 6.2.16-10
pve-kernel-5.15.111-1-pve: 5.15.111-1
pve-kernel-5.15.30-2-pve: 5.15.30-3
ceph-fuse: 16.2.11+ds-2
corosync: 3.1.7-pve3
criu: 3.17.1-2
glusterfs-client: 10.3-5
ifupdown2: 3.2.0-1+pmx4
ksm-control-daemon: 1.4-1
libjs-extjs: 7.0.0-4
libknet1: 1.25-pve1
libproxmox-acme-perl: 1.4.6
libproxmox-backup-qemu0: 1.4.0
libproxmox-rs-perl: 0.3.1
libpve-access-control: 8.0.5
libpve-apiclient-perl: 3.3.1
libpve-common-perl: 8.0.8
libpve-guest-common-perl: 5.0.4
libpve-http-server-perl: 5.0.4
libpve-network-perl: 0.8.1
libpve-rs-perl: 0.8.5
libpve-storage-perl: 8.0.2
libspice-server1: 0.15.1-1
lvm2: 2.03.16-2
lxc-pve: 5.0.2-4
lxcfs: 5.0.3-pve3
novnc-pve: 1.4.0-2
proxmox-backup-client: 3.0.2-1
proxmox-backup-file-restore: 3.0.2-1
proxmox-kernel-helper: 8.0.3
proxmox-mail-forward: 0.2.0
proxmox-mini-journalreader: 1.4.0
proxmox-offline-mirror-helper: 0.6.2
proxmox-widget-toolkit: 4.0.6
pve-cluster: 8.0.3
pve-container: 5.0.4
pve-docs: 8.0.4
pve-edk2-firmware: 3.20230228-4
pve-firewall: 5.0.3
pve-firmware: 3.8-2
pve-ha-manager: 4.0.2
pve-i18n: 3.0.5
pve-qemu-kvm: 8.0.2-5
pve-xtermjs: 4.16.0-3
qemu-server: 8.0.7
smartmontools: 7.3-pve1
spiceterm: 3.3.0
swtpm: 0.8.0+pve1
vncterm: 1.8.0
zfsutils-linux: 2.1.12-pve1
Hi,
the timeout for this calls is not configurable afaik. However, listing the storage contents should not be such an excessive task. By passing the content=images optional parameter, you limit the listing to images only, which leads me to suspect that there might be a huge number of backups or the like on the same storage. Is there something unusual about this particular storage, e.g. symlinks or mounts on subfolders ecc.?

Can you please share the storage config cat /etc/pve/storage.cfg and specify which storage the images reside on? Also try to output only the specific content types and see which is the largest by running
Bash:
pvesh get /nodes/{node}/storage/{storage}/content --content <content-type>
# <content-type> being one of rootdir, images, vztmpl, iso, backup, snippets