Backup Job Error - No space left on device

[HomerJ.S.]

Hi Community,

I recently set up a PBS, my PVE runs since a few months and now it´s time for reliable backups. So I brought my old ubuntu server to a new life, here the specs:

Intel(R) Celeron(R) CPU G3900
4 GB RAM
System SSD 120 GB
2 8 TB HDD
(additional 3 TB installed after errors came up)

The whole system is old, but no errors till now. The SMART values of the hard drives pass.

Here a list of my VMs and container:
VM - OpenMediaVault - 4 TB
CT - Openhab - 60 GB
CT - Debian with mosqitto - 8 GB
CT - Debian with paperless - 13 GB
CT - Debian with NGX - 6 GB
CT - Debian with Nextcloud - 1 TB
CT - Debian with PhotoPrism - 1 TB
CT - Debian with Immich - 100 GB
VM - Debian with TVHeadend - 1 TB

I could backup all of this VMs and CTs on a HDD on the PVE server.

I installed the PBS, connected the 8 TB drives to my PVE and started the backup jobs. Most of the backups run without errors, but 2 (nextcloud and photoprism) bring only an error.

Here the log of one of the error backup jobs:

INFO: starting new backup job: vzdump 106 --notes-template '{{guestname}}' --fleecing 0 --node proxmox --mode snapshot --storage PBS-Backup-HD3-3TB --all 0
INFO: Starting Backup of VM 106 (lxc)
INFO: Backup started at 2025-01-05 12:06:22
INFO: status = running
INFO: CT Name: DebianPhotoPrism
INFO: including mount point rootfs ('/') in backup
INFO: including mount point mp0 ('/mnt/daten') in backup
INFO: mode failure - some volumes do not support snapshots
INFO: trying 'suspend' mode instead
INFO: backup mode: suspend
INFO: ionice priority: 7
INFO: CT Name: DebianPhotoPrism
INFO: including mount point rootfs ('/') in backup
INFO: including mount point mp0 ('/mnt/daten') in backup
INFO: starting first sync /proc/4639/root/ to /var/lib/vz/tmp_backup/vzdumptmp140451_106/
ERROR: rsync: [receiver] write failed on "/var/lib/vz/tmp_backup/vzdumptmp140451_106/mnt/daten/photoprism/originals/Bilder/Gallerie/Urlaube/2024 \#303\#204gypten/DSC00851.jpg": No space left on device (28)
ERROR: rsync error: error in file IO (code 11) at receiver.c(380) [receiver=3.2.7]
ERROR: rsync: [sender] write error: Broken pipe (32)
ERROR: Backup of VM 106 failed - command 'rsync --stats -h -X -A --numeric-ids -aH --delete --no-whole-file --sparse --one-file-system --relative '--exclude=/tmp/?*' '--exclude=/var/tmp/?*' '--exclude=/var/run/?*.pid' /proc/4639/root//./ /proc/4639/root//./mnt/daten /var/lib/vz/tmp_backup/vzdumptmp140451_106/' failed: exit code 11
INFO: Failed at 2025-01-05 12:13:39
INFO: Backup job finished with errors
TASK ERROR: job errors

Error code 28 - no space left on device: The target HDD on the PBS has a size of 8 TB and is empty.
Error code 11- My local disk on PVE has a free space of 69 GB, the PBS local disk 80 GB. I followed this thread, without success:
https://forum.proxmox.com/threads/lxc-backup-fails-with-rsync-exit-code-11.120667/

So I have enough space on the disks, the local backup on PVE work without a problem, the big VM backup of the OMV (4TB) worked!!! Is there a difference between a local PVE backup and an PBS backup, or is there a difference between a PBS VM backup and a PBS CT backup!

Is the local SSD of my PVE to small? It has only 100 GB, with 69 free, but I do not remember to set this size. Should I try to resize this space?


I´m grateful for every tips!!!

Thanks Alex

 

[fireboyff]

Hello

I installed a new PBS 4 days ago. I have the same problem, I was able to backup my LXC103 once via PBS, on the same day at night I ran into the same error. The size of the system disk only changed by a few MB between the 1st and 2nd backup. I also have several TB free on my PBS, but the LXC system disk is only 400GB in size, so it should be anything but a space problem.

the other VMs and LXC are currently not affected.

ERROR: rsync: [receiver] write failed on "/var/tmp/vzdumptmp1333869_103/data/compose/1/storage/cache/thumbnails/1/3/f/13da2967c8d9efb6a97_1280x1024_fit.jpg": No space left on device (28)
ERROR: rsync error: error in file IO (code 11) at receiver.c(380) [receiver=3.2.7]
ERROR: rsync: [sender] write error: Broken pipe (32)
ERROR: rsync error: error in file IO (code 11) at io.c(848) [sender=3.2.7]
ERROR: Backup of VM 103 failed - command 'rsync --stats -h -X -A --numeric-ids -aH --delete --no-whole-file --sparse --one-file-system --relative '--exclude=/tmp/?*' '--exclude=/var/tmp/?*' '--exclude=/var/run/?*.pid' /proc/1080/root//./ /var/tmp/vzdumptmp1333869_103' failed: exit code 11
INFO: Failed at 2025-01-14 17:50:09
INFO: Backup job finished with errors
TASK ERROR: job errors

Node Vers: Linux **-2 6.8.12-5-pve #1 SMP PREEMPT_DYNAMIC PMX 6.8.12-5 (2024-12-03T10:26Z) x86_64 (was installed this weekend an is uptodate)
PBS: 3.3.0 also installed this weekend and is uptodate

Best regards Fire

 

[fiona]

Hi,
from the log it seems like you are using suspend mode backup which uses a temporary directory (tmpdir in the vzdump settings) and that is /var/tmp/ by default.