Problem when i proceed a clone of a template (got timeout)

[UmanGarbag]

Hi People !

I recently upgrade my proxmox host to 7.4-15 version, after that i trying to provisionning some vm with terraform, but i remark that is take a lot of time than usual (3~4 minutes instead of 20 seconds).

proxmox-ve: 7.4-1 (running kernel: 5.15.108-1-pve)
pve-manager: 7.4-15 (running version: 7.4-15/a5d2a31e)
pve-kernel-5.15: 7.4-4
pve-kernel-5.15.108-1-pve: 5.15.108-1
pve-kernel-5.15.30-2-pve: 5.15.30-3
ceph-fuse: 15.2.16-pve1
corosync: 3.1.7-pve1
criu: 3.15-1+pve-1
glusterfs-client: 9.2-1
ifupdown2: 3.1.0-1+pmx4
ksm-control-daemon: 1.4-1
libjs-extjs: 7.0.0-1
libknet1: 1.24-pve2
libproxmox-acme-perl: 1.4.4
libproxmox-backup-qemu0: 1.3.1-1
libproxmox-rs-perl: 0.2.1
libpve-access-control: 7.4.1
libpve-apiclient-perl: 3.2-1
libpve-common-perl: 7.4-2
libpve-guest-common-perl: 4.2-4
libpve-http-server-perl: 4.2-3
libpve-rs-perl: 0.7.7
libpve-storage-perl: 7.4-3
libspice-server1: 0.14.3-2.1
lvm2: 2.03.11-2.1
lxc-pve: 5.0.2-2
lxcfs: 5.0.3-pve1
novnc-pve: 1.4.0-1
openvswitch-switch: 2.15.0+ds1-2+deb11u4
proxmox-backup-client: 2.4.2-1
proxmox-backup-file-restore: 2.4.2-1
proxmox-kernel-helper: 7.4-1
proxmox-mail-forward: 0.1.1-1
proxmox-mini-journalreader: 1.3-1
proxmox-offline-mirror-helper: 0.5.2
proxmox-widget-toolkit: 3.7.3
pve-cluster: 7.3-3
pve-container: 4.4-6
pve-docs: 7.4-2
pve-edk2-firmware: 3.20230228-4~bpo11+1
pve-firewall: 4.3-4
pve-firmware: 3.6-5
pve-ha-manager: 3.6.1
pve-i18n: 2.12-1
pve-qemu-kvm: 7.2.0-8
pve-xtermjs: 4.16.0-2
qemu-server: 7.4-4
smartmontools: 7.2-pve3
spiceterm: 3.2-2
swtpm: 0.8.0~bpo11+3
vncterm: 1.7-1
zfsutils-linux: 2.1.11-pve1

I trying to troubleshoot and understand why, so i look to the syslog file and see that the pvestad service telling me this :



I using a share nfs to host my VM and my backup, The server hosting it is a Truenas server, i check the log of the nfs server, i don't see any problem on the server side.

I see some user in the forum that have the same problem, but there are no real solution for this.

Could someone help me ?

Thanks

 

[supermicro_server]

what appen if you try to delete and recreate the nfs storage? Do you have some errors?

 

[fiona]

Hi,
how does the CPU/IO/network load on the server and NFS look like during the operation? If its too high, you can try setting a bandwidth limit for the clone operation for that storage:

       pvesm set <storage> [OPTIONS]

       Update storage configuration.

       <storage>: <string>
           The storage identifier.

       --bwlimit [clone=<LIMIT>] [,default=<LIMIT>] [,migration=<LIMIT>] [,move=<LIMIT>] [,restore=<LIMIT>]
           Set I/O bandwidth limit for various operations (in KiB/s).

 

[UmanGarbag]

what appen if you try to delete and recreate the nfs storage? Do you have some errors?

Hello ! I try to disconnect/reconnect the nfs share, and the clone was much quickly than before, but get stuck at 100% (take 2m19s) to make.

I didn't see any error on pvestatd, good news. I'm gonna try with terraform.

I let you know

 

[UmanGarbag]

Hi,
how does the CPU/IO/network load on the server and NFS look like during the operation? If its too high, you can try setting a bandwidth limit for the clone operation for that storage:

       pvesm set <storage> [OPTIONS]

       Update storage configuration.

       <storage>: <string>
           The storage identifier.

       --bwlimit [clone=<LIMIT>] [,default=<LIMIT>] [,migration=<LIMIT>] [,move=<LIMIT>] [,restore=<LIMIT>]
           Set I/O bandwidth limit for various operations (in KiB/s).

Hello,

The load is OK, i have 10Gb interface connect directly to my nas, the io and cpu is ok, so there are no problem on load. I'm gonna check the bwlimit that are actually set.

Thanks you for answering

 

[UmanGarbag]

Hello ! I try to disconnect/reconnect the nfs share, and the clone was much quickly than before, but get stuck at 100% (take 2m19s) to make.

I didn't see any error on pvestatd, good news. I'm gonna try with terraform.

I let you know

See to be random, i reboot my nas and my proxmox server. But the issue still present

 

[supermicro_server]

You can see the content of the storage nfs?

 

[UmanGarbag]

When i tried to access as the same timeas the error message, i didn't get any error in the webui but the loading is very slow and i access to the data

 

[supermicro_server]

May be there is an incompability from your proxmox to your nfs server, Try to update nfs server and retry.