pipelined request failed: inserting chunk on datastore

[hspindel]

I have successfully configured PBS with a local disk for backups, and all of my PVE VMs get backed up just fine.

Now I'm trying to add a second datastore to PBS that is a CIFS share on a Synology DS1522 instead of a disk local to PBS. Whenever I try to run a backup from PVE with that datastore as a target the backup job aborts with backup write data failed: command error: write_data_upload error: pipelined request failed: inserting chunk on <datastore> failed.

This happens from multiple PVE sources.

On PBS, the CIFS-shared datastore is listed under Datastore, the summary data shows an appropriate amount of storage used/available, and the Content pane shows alll my VMs. So I believe the datastore is connected correctly. If I go to the Synology and view the contents of the VM directory, I see the lots of empty backup files, so PBS isn't sending the backups (as expected due to the above error).

pveversion of the PVE machines is pve-manager/9.0.11/ (running kernel 6.14.11-4-pve).
pveversion command is not available on PBS, but the PBS machine is running kernel 6.14.11-4.

I tried to research this, and mostly what I found is that this may be a permissions problem, but I can't find it. On PBS, the CIFS mount point is owner/group "backup" and the permissions are drwxr-xr-x. On the Synology, the CIFS share is allowed for the credentials of the directory which is mounted on PBS.

If instead of trying to backup from PVE through PBS to the Synology, I create a datastore under PVE that points directly to the Synology. When I do this, backups work fine. But I lose the functionality that PBS provides doing it this way.

In case it matters, the Synology uses BTRFS.

I do not know what other information to provide to get help, but I'm happy to provide anything anyone asks for.

Thank you very much.

 

[Sviat]

Hi ,

There at least few possible issues can be with the Synology share:
1. "verify checksum integrity" check if it's enabled (better to do not use at all due to performance issues caused)
2. Check that there no snapshots enabled on storage and share level
3. Check that recycle bin disabled
4. Check that the share mounted with the correct SMB version
5. Check that there no BTRFS scrubbing running in time when you have a backups
6. If you have thin provisioned volume it also can be an issue with performance, better use thick
7. Check that share mounted with 'relatime' option

And finally try to use NFS instead of CIFS (usually it much faster and reliable)

 

[hspindel]

Thank you very much, but I may be confused. Bit of a newbie to pve.

1. In PBS, I turned off the option to Verify New Snapshots. It made no difference. Is that the setting to which you are referring? Is there something I should change in the tuning options? (chunk iteration order, data sync level)
2. There are snapshots at both pve and the Syno, but neither is running when I try the backup.
3. Recycle bin is disabled on the Syno.
4. Not sure how to check this. The share is mounted in pve fstab, and the type is just CIFS. On the Syno, minimum level is SMB2, max is SMB3.
5. No scrubbing active on the Syno. pve is not using BTRFS.
6. PBS Storage was created as Datastore Type: Local, Backing Path: Absolute path pointing to the mount point where the Syno share is mounted. Or are you asking about the volume type on PVE that is getting backed up? On one of the PVE machines, it is thin, on the other two it is ZFS.
7. From what I can find, if the mount in fstab does NOT specify "norelatime" then the default is relatime. My actual fstab entry is:
//shareIP/shareName /mnt/localMountPoint cifs rw,auto,uid=34,noforceuid,gid=34,noforcegid,nofail,x-systemd.devicetimeout=10s,credentials=<credentialFile>

I'm not using NFS anywhere, but willing to try it if that's my best option for the next step.

 

[Sviat]

Hi,

1. Not exactly but disabled verification can help you to reduce I/O load.
I mean share settings, see screenshot attached.
2. If you have snapshots enabled on Synology it can significantly reduce performance.
3. SSH on PVE and check by the command: mount
5. No scrubbing is good. PVE is using Syno storage on BTRFS so it mean that it use it. Main problem that BTRFS not always stable as you can expect.
6. It was about Synology volume (it can be Thick-provisioned or Thin-provisioned)
7. Try with norelatime mount option, it disables the updating of file access times unless explicitly requested and shiould improve disk performance by reducing write operations.
And finally CIFS/SMB is inherently slower than NFS, especially for small file operations or high-latency networks.

 

[Onslow]

Try with norelatime mount option

Doesn't it nullify PBS' method for garbage collection?