Local Datastore sync slow down after few minutes

[ccorsani]

I have a PBS with local Datastore (ext4 on 3 disk raid) and I configure some external disk as removable on the 4th hotplug sata slot. Every day I change a disk and launch a sync between local Datastore and removable disk to have a daily offline backup. I just backup the last 3 backups of vms. When the sync starts it works fine with an average of 70MiB/s write speed (some top speed over 200MiB/s), after some times it decrease down to 1MiB/s. Can't find why

 

[Chris]

What is your proxmox-backup-manager version --verbose? Also, please share the sync task log, thanks!

 

[ccorsani]

root@psb3:~# proxmox-backup-manager version --verbose
proxmox-backup 4.0.0 running kernel: 6.14.8-2-pve
proxmox-backup-server 4.1.0-1 running version: 4.0.14
proxmox-kernel-helper 9.0.4
proxmox-kernel-6.14.8-2-pve-signed 6.14.8-2
proxmox-kernel-6.14 6.14.11-4
proxmox-kernel-6.8.12-13-pve-signed 6.8.12-13
proxmox-kernel-6.8 6.8.12-13
proxmox-kernel-6.8.12-8-pve-signed 6.8.12-8
ifupdown2 3.3.0-1+pmx11
libjs-extjs 7.0.0-5
proxmox-backup-docs 4.1.0-1
proxmox-backup-client 4.1.0-1
proxmox-mail-forward 1.0.2
proxmox-mini-journalreader 1.6
proxmox-offline-mirror-helper 0.7.3
proxmox-widget-toolkit 5.1.2
pve-xtermjs 5.5.0-3
smartmontools 7.4-pve1
zfsutils-linux 2.3.4-pve1

 

[Chris]

What sustained speeds do you expect from the disk? You might want to benchmark it using e.g. fio using random sync write patterns with 4M data payloads. Also, do you see an increased IOdelay, what is the output of cat /proc/pressure/io

Edit: see also https://forum.proxmox.com/threads/pbs-pull-sync-job-slows-down-about-x20-after-a-while.177873/