Backup to NFS has long delay after INFO: transferred

[RokModTwo]

Have been doing backups to a NFS share on a TrueNAS successfully from a PVE 3-node cluster

Now trying to do backups to NFS share on a Synology NAS from the cluster.

Finding that the backups eventually finish... problem is a major delay from when the task viewer messages show INFO: transferred XX.YY GB in ZZZ seconds... there is a huge delay minutes...before the TASK OK appears.

Example. a VM that backs up in 2 minutes to the TrueNAS finished in 9 minutes to the Synology. I watch the Status tab on the backup task and see it gets to roughly the same 2 minute mark and "INFO: transferred..." message but then the delay....then it finishes TASK OK.

I haven't been able to determine root cause for the additional delay.

I did the same backup to the Synology via SMB/CIFS share... same behavior (gets to the INFO: transferred... message, but major delay until I see a TASK OK message)
Tested backups of VMs from different nodes in the same cluster - same behaviour.

However, tried the same from a 2nd PVE single node to the same Synology NAS via NFS.... and it works normal, i.e. only a second or so delay between INFO: transferred and TASK OK message.

Cluster is 7.1-8 as is the single PVE server.

Ideas as to what might be causing the delay from the cluster?

 

[LnxBil]

Ideas as to what might be causing the delay from the cluster?

I assume it's caching on the PVE side. I also see the delays and it will increase with the free memory on the host.

In the beginning of the backup process it states that it has written more bytes than the physical speed can handle and after a while "it catches up", so it will wait until the NFS share acknowledges the write and then the throughput will be going down. If the backup process is finished, it will only really finish if the data is written on the other side and until that final acknowledge it will wait.

 

[RokModTwo]

Thanks for those points - that sounds plausible i.e. I am seeing a mis-match between messages in PVE task window vs actual writing of data.

However my point about the 2nd PVE... I did the same VM on both systems, offline/not running, and did the backup to Synology via NFS... I watch the timer on the PVE task and both PVE's hit roughly the same time elapse to get to the "INFO: transferred.." message BUT the single PVE finishes seconds later whereas the PVE node in the cluster takes the additional 7 minutes to get to the TASK OK/finished state. Since the single PVE node finishes ... as you mention it must receive the final acknowledge from the NFS/Synology. So why is the cluster / Synology not behaving the same... more things to investigate.