LXC backup fails with rsync exit code 11

[ateshabaev]

I have LXC container with 120GB disk space, 35GB out of 120GB is used, which is roughly 35% taken.

I have connected external storage box from Hetzner for backup purposes.

When I manually click "Backup now" (or scheduled one) the job fails with "rsync exit code 11 ", see spoiler below for job details.

I Googled rsync exit code 11, it's related to "not enough storage space", but it seems like a have more than enough to store temporary backup file before it's moved to external storage.

Spoiler: Backup job log

INFO: starting new backup job: vzdump 101 --compress zstd --node mynode --mode suspend --storage hetzner-storage-box --notes-template '{{guestname}}' --remove 0
INFO: filesystem type on dumpdir is 'cifs' -using /var/tmp/vzdumptmp3456915_101 for temporary files
INFO: Starting Backup of VM 101 (lxc)
INFO: Backup started at 2023-01-07 13:44:35
INFO: status = running
INFO: backup mode: suspend
INFO: ionice priority: 7
INFO: CT Name: dev.domain.io
INFO: including mount point rootfs ('/') in backup
INFO: starting first sync /proc/3445057/root/ to /var/tmp/vzdumptmp3456915_101
ERROR: Backup of VM 101 failed - command 'rsync --stats -h -X -A --numeric-ids -aH --delete --no-whole-file --sparse --one-file-system --relative '--exclude=/tmp/?*' '--exclude=/var/tmp/?*' '--exclude=/var/run/?*.pid' /proc/3445057/root//./ /var/tmp/vzdumptmp3456915_101' failed: exit code 11
INFO: Failed at 2023-01-07 13:45:15
INFO: Backup job finished with errors
TASK ERROR: job errors

By the way, I tried different backup modes: suspend and snapshot. All of these fails with same error code.

Did anyone encountered similar problem before? And if so, how to overcome that?

 

[ateshabaev]

Any clues guys?

 

[Tharos47]

Hello, I'm commenting here since this thread show up when googling this problem.
It's fairly simple. As far as I understand vzdump create a local temporary file before sending it to the destination. It's by default created in /var/tmp as your log show. Your /var/tmp is probably on a smaller disk and doesn't have enough free space.

Here is how to check :
df -h /var/tmp
If there is not enough space for your temp files you should edit /etc/vzdump.conf
the setting to change is this :

tmpdir: /var/lib/vz/tmp_backup
#you can use any folder as long as there is enough space on the disk where this folder is located

 

[rmwerder]

tmpdir: /var/lib/vz/tmp_backup
#you can use any folder as long as there is enough space on the disk where this folder is located

I've made an account just to thank you for this post. Obviously, it solved this problem

 

[kutyurin]

tmpdir: /var/lib/vz/tmp_backup
#you can use any folder as long as there is enough space on the disk where this folder is located

Thanks! Your message helped a lot!