ERROR: Backup of VMID failed - job failed with err -116 - Stale file handle

[Gilberto Ferreira]

Hi there.
I am trying to get a backup to this particular VM and I am getting the error: ERROR: Backup of VM 117 failed - job failed with err -116 - Stale file handle
I have another 30 vms or so, and all 29 got backup nicely.
But this one it's being a problem.

The PBS has disk directly attach in the server. It's a standlone server. No NFS, no CIFS/SMB.

vzdump 117 --mode snapshot --notification-mode auto --storage bkp-storage1106 --node proxmox01 --remove 0 --notes-template '{{guestname}}'

117: 2025-08-02 20:04:32 INFO: Starting Backup of VM 117 (qemu)
117: 2025-08-02 20:04:32 INFO: status = running
117: 2025-08-02 20:04:32 INFO: VM Name: SDBSQLQA
117: 2025-08-02 20:04:32 INFO: include disk 'scsi0' 'STG-VMS:117/vm-117-disk-1.qcow2' 100G
117: 2025-08-02 20:04:32 INFO: include disk 'scsi1' 'STG-VMS:117/vm-117-disk-2.raw' 230G
117: 2025-08-02 20:04:32 INFO: include disk 'scsi2' 'STG-VMS:117/vm-117-disk-3.raw' 100G
117: 2025-08-02 20:04:32 INFO: include disk 'scsi3' 'STG-VMS:117/vm-117-disk-4.raw' 20G
117: 2025-08-02 20:04:32 INFO: include disk 'scsi4' 'STG-VMS:117/vm-117-disk-5.qcow2' 250G
117: 2025-08-02 20:04:32 INFO: include disk 'efidisk0' 'STG-VMS:117/vm-117-disk-0.qcow2' 528K
117: 2025-08-02 20:04:32 INFO: backup mode: snapshot
117: 2025-08-02 20:04:32 INFO: ionice priority: 7
117: 2025-08-02 20:04:32 INFO: creating Proxmox Backup Server archive 'vm/117/2025-08-02T23:04:32Z'
117: 2025-08-02 20:04:32 INFO: issuing guest-agent 'fs-freeze' command
117: 2025-08-02 20:04:39 INFO: issuing guest-agent 'fs-thaw' command
117: 2025-08-02 20:04:40 INFO: started backup task 'd8c7e45c-4ddf-487e-9848-cf982da6fede'
117: 2025-08-02 20:04:40 INFO: resuming VM again
117: 2025-08-02 20:04:40 INFO: efidisk0: dirty-bitmap status: created new
117: 2025-08-02 20:04:40 INFO: scsi0: dirty-bitmap status: created new
117: 2025-08-02 20:04:40 INFO: scsi1: dirty-bitmap status: created new
117: 2025-08-02 20:04:40 INFO: scsi2: dirty-bitmap status: created new
117: 2025-08-02 20:04:40 INFO: scsi3: dirty-bitmap status: created new
117: 2025-08-02 20:04:40 INFO: scsi4: dirty-bitmap status: created new
117: 2025-08-02 20:04:45 INFO:   0% (3.5 GiB of 700.0 GiB) in 5s, read: 712.8 MiB/s, write: 103.2 MiB/s
.....
117: 2025-08-02 20:33:07 INFO:  44% (314.6 GiB of 700.0 GiB) in 28m 27s, read: 159.1 MiB/s, write: 89.8 MiB/s
117: 2025-08-02 20:33:07 ERROR: job failed with err -116 - Stale file handle
117: 2025-08-02 20:33:07 INFO: aborting backup job
117: 2025-08-02 20:33:07 INFO: resuming VM again
117: 2025-08-02 20:33:07 ERROR: Backup of VM 117 failed - job failed with err -116 - Stale file handle

Any help will be appreciated.

Thanks.

 

[Chris]

Hi,
what about the source? Is the scsi4 drive located on an shared storage?

 

[Gilberto Ferreira]

Hi,
what about the source? Is the scsi4 drive located on an shared storage?

Hi there... Thanks for reply...

The underlying storage for all VMS and disk is based on GlusterFS 11, but is presented to Proxmox as a shared disk storage.
All other VMs works fine, either VMs itself or backup task.

Is not clear to me why on this VM has this problem.
I alread update all nodes and restart it.
It's now runnning Proxmox VE 8.46.

Best regards

 

[Chris]

Stale file handle

Looks like something is invalidating the file descriptor while performing the backup. Is there any other, concurrent operation accessing the VM disk? Does it always fail on the disk attached at iscsi4?

 

[Gilberto Ferreira]

Looks like something is invalidating the file descriptor while performing the backup. Is there any other, concurrent operation accessing the VM disk? Does it always fail on the disk attached at iscsi4?

In this VM there is at least for my knowledge, a MS SQL server.
But I cannot access it directly, because is customer discretion.
Now, I had have perform a backup manualy, bypassing the job task scheduled to run by night, but get same error.
I will try to do a manualy backup again, but I will mark the scsi4 to do not been backup.

I will post the results later.

Thanks again.