Failed backup - unable to activate storage

T

tommisan

Renowned Member
Proxmox Subscriber
Dec 9, 2014
40
0
71
Hi everyone,
at present we have a production environment with 3 nodes Proxmox VE cluster (9.1.6) with a Lenovo DM5000H as NFS (4.2) vm/lxc storage and a dedicated physical Proxmox PBS (4.1.x).
PVE nodes e PBS enterprise repos.
All the appliance are connected with redundant 10Gbit port switches (Cisco-Juniper).
In the last months we have experienced some backup jobs failed for one/two vms or lxc containers (i.e ERROR: Backup of VM 135 failed - unable to activate storage 'dm5000h' - directory '/mnt/pve/dm5000h' does not exist or is unreachable) over almost 30.
We thought at first it was a network issue but we couldn't point out what's going on wrong cause we have also other appliances in the same datacenter that are ok.
We also thought it was a node issue but the backjob fails on different PVE nodes (last 2 nodes of three, 3 vms).
On Proxmox PVE node we got something like pvestatd[2720]: unable to activate storage 'dm5000h' - directory '/mnt/pve/dm5000h' does not exist or is unreachable
Do you have any ideas?
Thank you in advance
 
Last edited:
Hi Fiona,
thank you for your reply. We use different and dedicated networks (VLAN) for PBS and NFS traffic. PBS, NFS server and PVE nodes loads and network throughputs and latency are ok and with the same values as always. The backup job doesn't fail on the same vm/lxc, and it fails not regarding to the vm size, threfore we tought at first it was a network issue, but metrics are really good. From the source code it seems to me that proxmox waits for 2 seconds (that for our network it would be ok) before declaring the storage unreachable, am I right?
 
Last edited:
tommisan said:
Hi Fiona,
thank you for your reply. We use different and dedicated networks (VLAN) for PBS and NFS traffic. PBS, NFS server and PVE nodes loads and network throughputs and latency are ok and with the same values as always. The backup job doesn't fail on the same vm/lxc, and it fails not regarding to the vm size, threfore we tought at first it was a network issue, but metrics are really good. From the source code it seems to me that proxmox waits for 2 seconds (that for our network it would be ok) before declaring the storage unreachable, am I right?
Click to expand...
So still the same physical network? Yes, the timeout for checking the mount directory is 2 seconds. Telling from the error message, this is the command that fails for you and since you see the issue sporadically/randomly it might be related to the network after all.
 
Hi,


This looks like the NFS storage is temporarily disconnecting during backup.


Since it happens on different nodes and VMs, it’s likely a network or NFS mount issue, not the VM itself.


Please check NFS connectivity, server logs, and mount stability on /mnt/pve/dm5000h.
 
Hi,
It's one physical network 10Gbit separated into vlans.

Based on the logs and performance data collected during the latest backup cycle, here is a summary of the findings.
Latency of nfs storage: baseline latency is 0.2ms, under load it is 10ms and it can spike up to 28ms during backups, but I think it's still very good.
Throughput Saturation: Aggregated throughput reached 800 MB/s. Simultaneous peaks from PVE nodes (up to 800 MiB/s) could cause maybe contention.
Connectivity: nfsstat shows zero retransmissions.

We are implementing a temporary limit of 400 MiB/s bandwidth limit per node on backup job to preserve headroom for system control traffic.
 
Hi,
we lower to 200 MiB/s bandwidth limit per node on backup job, but the problem sometimes still happens and one or 2 vm/contaniner are skipped.
Load, bandwith and latency are not a issue. We can't figure out which is the problem.
Proxmox VE cluster upgraded to the latest 9.2.2.

here is an extract (backup job skipped on vm and then keep going tille the end on job)
[...]
116: 2026-05-30 23:37:32 INFO: backup was done incrementally, reused 208.00 GiB (100%)
116: 2026-05-30 23:37:32 INFO: transferred 208.00 GiB in 1073 seconds (198.5 MiB/s)
116: 2026-05-30 23:37:32 INFO: stopping kvm after backup task
116: 2026-05-30 23:37:33 INFO: adding notes to backup
116: 2026-05-30 23:37:33 INFO: Finished Backup of VM 116 (00:17:57)

119: 2026-05-30 23:37:33 INFO: Starting Backup of VM 119 (qemu)
119: 2026-05-30 23:37:33 INFO: status = stopped
119: 2026-05-30 23:37:33 INFO: backup mode: stop
119: 2026-05-30 23:37:33 INFO: bandwidth limit: 204800 KiB/s
119: 2026-05-30 23:37:33 INFO: ionice priority: 7
119: 2026-05-30 23:37:33 INFO: VM Name: rslurm-wn01
119: 2026-05-30 23:37:33 INFO: include disk 'scsi0' 'dm5000h:119/vm-119-disk-0.qcow2' 80G
119: 2026-05-30 23:37:42 ERROR: Backup of VM 119 failed - unable to activate storage 'dm5000h' - directory '/mnt/pve/dm5000h' does not exist or is unreachable

120: 2026-05-30 23:37:42 INFO: Starting Backup of VM 120 (qemu)
120: 2026-05-30 23:37:42 INFO: status = stopped
120: 2026-05-30 23:37:42 INFO: backup mode: stop
120: 2026-05-30 23:37:42 INFO: bandwidth limit: 204800 KiB/s
120: 2026-05-30 23:37:42 INFO: ionice priority: 7
120: 2026-05-30 23:37:42 INFO: VM Name: rslurm-wn02
120: 2026-05-30 23:37:42 INFO: include disk 'scsi0' 'dm5000h:120/vm-120-disk-0.qcow2' 80G
120: 2026-05-30 23:37:42 INFO: creating Proxmox Backup Server archive 'vm/120/2026-05-30T21:37:42Z'
120: 2026-05-30 23:37:42 INFO: starting kvm to execute backup task
120: 2026-05-30 23:37:43 INFO: starting backup via QMP command
120: 2026-05-30 23:37:43 INFO: started backup task 'add57cf5-672b-47fa-aeab-338e2a93eb19'
120: 2026-05-30 23:37:43 INFO: scsi0: dirty-bitmap status: created new
[...]
 
You must log in or register to reply here.
Top Bottom
Back