Failed backup - unable to activate storage

[tommisan]

Hi everyone,
at present we have a production environment with 3 nodes Proxmox VE cluster (9.1.6) with a Lenovo DM5000H as NFS (4.2) vm/lxc storage and a dedicated physical Proxmox PBS (4.1.x).
PVE nodes e PBS enterprise repos.
All the appliance are connected with redundant 10Gbit port switches (Cisco-Juniper).
In the last months we have experienced some backup jobs failed for one/two vms or lxc containers (i.e ERROR: Backup of VM 135 failed - unable to activate storage 'dm5000h' - directory '/mnt/pve/dm5000h' does not exist or is unreachable) over almost 30.
We thought at first it was a network issue but we couldn't point out what's going on wrong cause we have also other appliances in the same datacenter that are ok.
We also thought it was a node issue but the backjob fails on different PVE nodes (last 2 nodes of three, 3 vms).
On Proxmox PVE node we got something like pvestatd[2720]: unable to activate storage 'dm5000h' - directory '/mnt/pve/dm5000h' does not exist or is unreachable
Do you have any ideas?
Thank you in advance

 

[fiona]

Hi,
are you using the same network for PBS and NFS traffic? How does the load look like when the issue occurs? It might transient issues where the check whether the NFS mount point is reachable times out.

 

[tommisan]

Hi Fiona,
thank you for your reply. We use different and dedicated networks (VLAN) for PBS and NFS traffic. PBS, NFS server and PVE nodes loads and network throughputs and latency are ok and with the same values as always. The backup job doesn't fail on the same vm/lxc, and it fails not regarding to the vm size, threfore we tought at first it was a network issue, but metrics are really good. From the source code it seems to me that proxmox waits for 2 seconds (that for our network it would be ok) before declaring the storage unreachable, am I right?