Backup failing ENOSPC when there is plenty of free space

[guff666]

My backup jobs have just started failing. The following extract from the backup job's log shows that it failed with an ENOSPC error. However, there is plenty of space on all disks - including the backup server.

INFO: starting new backup job: vzdump --notes-template '{{guestname}}' --all 1 --storage pbs --mailnotification failure --mode snapshot --node pve1 --mailto <redacted>
INFO: Starting Backup of VM 100 (lxc)
INFO: Backup started at 2023-07-06 17:41:55
INFO: status = running
INFO: CT Name: emoncms
INFO: including mount point rootfs ('/') in backup
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: create storage snapshot 'vzdump'
  Logical volume "snap_vm-100-disk-0_vzdump" created.
INFO: creating Proxmox Backup Server archive 'ct/100/2023-07-06T16:41:55Z'
INFO: run: /usr/bin/proxmox-backup-client backup --crypt-mode=none pct.conf:/var/tmp/vzdumptmp4120784_100/etc/vzdump/pct.conf root.pxar:/mnt/vzsnap0 --include-dev /mnt/vzsnap0/./ --skip-lost-and-found --exclude=/tmp/?* --exclude=/var/tmp/?* --exclude=/var/run/?*.pid --backup-type ct --backup-id 100 --backup-time 1688661715 --repository root@pam@pbs.agdon.net:pve-backups
INFO: Starting backup: ct/100/2023-07-06T16:41:55Z
INFO: Client name: pve1
INFO: Starting backup protocol: Thu Jul  6 17:41:56 2023
INFO: Error: failed to move file at "/var/log/proxmox-backup/tasks/07/UPID:pbs:00000278:00000907:00001F36:64A6EED5:backup:pve\\x2dbackups\\x3act-100:root@pam:.tmp_Q1KHn0" into place at "/var/log/proxmox-backup/tasks/07/UPID:pbs:00000278:00000907:00001F36:64A6EED5:backup:pve\\x2dbackups\\x3act-100:root@pam:" - ENOSPC: No space left on device
INFO: cleanup temporary 'vzdump' snapshot
  Logical volume "snap_vm-100-disk-0_vzdump" successfully removed
ERROR: Backup of VM 100 failed - command '/usr/bin/proxmox-backup-client backup '--crypt-mode=none' pct.conf:/var/tmp/vzdumptmp4120784_100/etc/vzdump/pct.conf root.pxar:/mnt/vzsnap0 --include-dev /mnt/vzsnap0/./ --skip-lost-and-found '--exclude=/tmp/?*' '--exclude=/var/tmp/?*' '--exclude=/var/run/?*.pid' --backup-type ct --backup-id 100 --backup-time 1688661715 --repository root@pam@pbs.agdon.net:pve-backups' failed: exit code 255
INFO: Failed at 2023-07-06 17:41:57

root@pve1:~# df -h
Filesystem                    Size  Used Avail Use% Mounted on
udev                          7.8G     0  7.8G   0% /dev
tmpfs                         1.6G  1.4M  1.6G   1% /run
/dev/mapper/pve-root           94G  3.6G   86G   4% /
tmpfs                         7.8G   46M  7.8G   1% /dev/shm
tmpfs                         5.0M     0  5.0M   0% /run/lock
/dev/sda2                    1022M  344K 1022M   1% /boot/efi
/dev/mapper/vol_grp1-frigate  493G  3.7G  464G   1% /import/frigate
/dev/mapper/vol_grp1-docker5  251G  154G   86G  65% /import/docker5
/dev/mapper/vol_grp1-media    3.4T  2.3T  963G  71% /import/media
/dev/fuse                     128M   40K  128M   1% /etc/pve
tmpfs                         1.6G     0  1.6G   0% /run/user/0

 

[guff666]

Nomatter. I now see that the PBS is full of log files! They obviously aren't being purged.