PBS 3.4.8 datastore corruption causing backup failures with "Structure needs cleaning (os error 117)" and restore CRC checksum errors

[Pavitra]

Hello everyone,

I am facing an issue with my Proxmox Backup Server and would appreciate any guidance.

Environment​

• Proxmox Backup Server Version: 3.4.8
• Installation Type: Dedicated PBS Server
• Filesystem: XFS
• Datastore: Demo Datastore
• Storage Capacity: ~100 TB
• Number of connected Proxmox VE nodes: Multiple

Problem​

Backups from multiple Proxmox VE nodes were working normally, but now almost every backup fails while uploading data to the PBS datastore.

The backup starts successfully:

• VM snapshot is created successfully.
• QEMU Guest Agent freeze/thaw works correctly.
• Incremental (dirty bitmap) backup starts.
• Backup progresses normally (sometimes 40%, 70%, 85%, or even 96%).

However, it eventually fails with the following error:

backup write data failed: command error: write_data upload error:
pipelined request failed:
inserting chunk on store 'Enjay' failed

Atomic rename failed for file:
/mnt/datastore/Enjay/.chunks/...

Structure needs cleaning (os error 117)

In addition to the backup failures, restoring an older backup also fails with:

Failed to parse chunk ...
Data blob has wrong CRC checksum


Because of this:

• Multiple VM backups fail.
• Some restore operations also fail due to corrupted chunks.

Expected Behavior​

Backups should complete successfully, store all chunks correctly on the PBS datastore, and restore without checksum errors.

What I Have Tried​

• Verified that the issue occurs with multiple different VMs.
• Confirmed the problem is not specific to a single Proxmox VE node.
• Confirmed snapshots and QEMU Guest Agent are working correctly.
• Observed that failures always occur while PBS is writing chunks to the datastore.
• Restore testing also reports CRC checksum errors on stored chunks.

Error Messages​

Backup:

Structure needs cleaning (os error 117)
Atomic rename failed
write_data upload error
Restore:

Failed to parse chunk
Data blob has wrong CRC checksum

Questions​

1. Does "Structure needs cleaning (os error 117)" indicate filesystem corruption only, or could it also be caused by failing storage hardware?
2. Is it safe to run proxmox-backup-manager verify before repairing the filesystem?
3. Should I repair the filesystem first (fsck/xfs_repair depending on the filesystem) before attempting any further backups?
4. Is there any supported method to recover or salvage corrupted chunks from the datastore?
5. Could this issue be related to running Garbage Collection multiple times on the datastore? We have executed Garbage Collection several times in the past. Could repeated GC operations contribute to datastore corruption, or is this more likely to be caused by an underlying filesystem or storage hardware issue?

Additional Information​

• The issue affects multiple VMs across multiple Proxmox VE nodes.
• The failures are always on the same PBS datastore (Enjay).
• Previous backups were working correctly before this issue started.
• This is the second time I have encountered this problem. The first time, it occurred on a different Proxmox Backup Server. At that time, I was unable to recover the datastore and had to perform a complete reinstallation and reconfiguration of the PBS server.
• I would like to understand the root cause of this issue and whether there is a recommended way to repair or recover the datastore without having to rebuild the entire PBS server if it happens again.

 

[Chris]

Does "Structure needs cleaning (os error 117)" indicate filesystem corruption only, or could it also be caused by failing storage hardware?

The error points towards a filesystem corruption, the causes thereof might be manifold.

Is it safe to run proxmox-backup-manager verify before repairing the filesystem?

I would rather recommend to put your PBS datastore into maintenance mode offline and create a full disk clone to another storage medium via some dedicated tooling to have a backup before doing anything else.

Should I repair the filesystem first (fsck/xfs_repair depending on the filesystem) before attempting any further backups?

Yes, but first it is strongly suggested to create the full disk clone as suggested above.

Is there any supported method to recover or salvage corrupted chunks from the datastore?

You might be able to recover the chunks if VMs/CTs still contain the data of the chunks and you perform a non-incremental backup. Other than that it depends on the corruption.

Could this issue be related to running Garbage Collection multiple times on the datastore?

No, this is a filesystem level corruption, garbage collection of PBS has nothing to do with it.

 

[waltar]

I would strongly take a look onto your disks and raid system under the xfs as cannot write or read chunks are followon errors to xfs and even then to pbs.

 

[J-Rod]

Definitely provide the underlying hardware information/configuration.

For PBS, I would suggest using ZFS with at least one disk's worth of redundancy so it can correct errors.

If hardware RAID, set up a continuous background scrub (reduced rate) so it can scan for and repair errors.