vzdump backups to CIFS -mounted storage sometimes fail for large backups

[Tapio Lehtonen]

# pveversion
pve-manager/6.4-4/337d6701 (running kernel: 5.4.106-1-pve)


I keep getting errors when backing up to CIFS mounted storage.
This on dedicated host rented from Hetzner.

# cat /etc/pve/storage.cfg

dir: local
    path /var/lib/vz
    content vztmpl,iso
    shared 0

zfspool: local-zfs
    pool rpool/data
    content images,rootdir
    sparse 1

cifs: box
    path /mnt/pve/storagebox
    server c123456.your-storagebox.de
    share backup
    content backup,iso
    prune-backups keep-daily=7,keep-monthly=1,keep-weekly=3
    username c123456

I used to get errors almost every time for big dumps until I added disk for swap.

root@hetrauta:/etc# free -h
              total        used        free      shared  buff/cache   available
Mem:          125Gi        99Gi        25Gi       258Mi       635Mi        25Gi
Swap:         223Gi       3.7Gi       219Gi

Now it mostly works. The hosts with small disk always work, it is the hosts
with about 100 GB disk that sometimes fail.

May 10 01:38:01 hetrauta systemd[1]: pvesr.service: Succeeded.
May 10 01:38:01 hetrauta systemd[1]: Started Proxmox VE replication runner.
May 10 01:38:45 hetrauta pvestatd[4651]: got timeout
May 10 01:38:45 hetrauta kernel: [687491.041047] CIFS VFS: \\u123456.your-storagebox.de Cancelling wait for mid 27567257 cmd: 
5
May 10 01:38:45 hetrauta kernel: [687491.042170] CIFS VFS: \\u123456.your-storagebox.de Cancelling wait for mid 27567258 cmd: 
16
May 10 01:38:45 hetrauta pvestatd[4651]: unable to activate storage 'box' - directory '/mnt/pve/box' does not exist or is unre
achable
May 10 01:38:51 hetrauta kernel: [687496.709765] CIFS VFS: Close unmatched open
May 10 01:38:51 hetrauta vzdump[34263]: ERROR: Backup of VM 5111 failed - zstd --rsyncable --threads=1 failed - wrong exit status 1
May 10 01:38:51 hetrauta vzdump[34263]: INFO: Starting Backup of VM 5112 (qemu)

This is from the e-mail report vzdump sent:

5111: 2021-05-10 01:38:37 INFO:  99% (238.7 GiB of 240.0 GiB) in 15m 49s, read: 891.3 MiB/s, write: 1.3 KiB/s
  5111: 2021-05-10 01:38:39 INFO: 100% (240.0 GiB of 240.0 GiB) in 15m 51s, read: 644.6 MiB/s, write: 4.0 KiB/s
  5111: 2021-05-10 01:38:39 INFO: backup is sparse: 97.33 GiB (40%) total zero data
  5111: 2021-05-10 01:38:39 INFO: transferred 240.00 GiB in 951 seconds (258.4 MiB/s)
  5111: 2021-05-10 01:38:51 ERROR: Backup of VM 5111 failed - zstd --rsyncable --threads=1 failed - wrong exit status 1

There were 16 backup dumps, the above was in the middle. The ones before and after went OK.

How can I get more info on what is wrong to maybe fix this?