backup job error no space left on device but there is space.

C

cTurtle98

Member
Jul 3, 2022
4
0
6
I'm Getting an error in backup jobs

Code: 
INFO: starting new backup job: vzdump --fleecing 0 --storage pbs0 --all 1 --quiet 1 --notes-template '{{node}} - {{vmid}} - {{guestname}}' --mode snapshot
INFO: Starting Backup of VM 100 (lxc)
INFO: Backup started at 2025-01-17 00:00:05
INFO: status = running
INFO: CT Name: PlexMediaServer
INFO: including mount point rootfs ('/') in backup
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: create storage snapshot 'vzdump'
INFO: creating Proxmox Backup Server archive 'ct/100/2025-01-17T08:00:05Z'
INFO: set max number of entries in memory for file-based backups to 1048576
INFO: run: lxc-usernsexec -m u:0:100000:65536 -m g:0:100000:65536 -- /usr/bin/proxmox-backup-client backup --crypt-mode=none pct.conf:/var/tmp/vzdumptmp3836161_100/etc/vzdump/pct.conf root.pxar:/mnt/vzsnap0 --include-dev /mnt/vzsnap0/./ --skip-lost-and-found --exclude=/tmp/?* --exclude=/var/tmp/?* --exclude=/var/run/?*.pid --backup-type ct --backup-id 100 --backup-time 1737100805 --entries-max 1048576 --repository pve1@pbs@pbs0.servers.cturtle98.net:pbs0-16tb-hdd-mirror --ns cTurtle98
INFO: Starting backup: [cTurtle98]:ct/100/2025-01-17T08:00:05Z   
INFO: Client name: pve1   
INFO: Starting backup protocol: Fri Jan 17 00:00:05 2025   
INFO: Error: failed to move file at "/var/log/proxmox-backup/tasks/5A/UPID:pbs0:00000421:0000075A:000004A9:678A0E06:backup:pbs0\\x2d16tb\\x2dhdd\\x2dmirror\\x3act-100:pve1@pbs:.tmp_joj84f" into place at "/var/log/proxmox-backup/tasks/5A/UPID:pbs0:00000421:0000075A:000004A9:678A0E06:backup:pbs0\\x2d16tb\\x2dhdd\\x2dmirror\\x3act-100:pve1@pbs:" - ENOSPC: No space left on device
INFO: cleanup temporary 'vzdump' snapshot
ERROR: Backup of VM 100 failed - command 'lxc-usernsexec -m u:0:100000:65536 -m g:0:100000:65536 -- /usr/bin/proxmox-backup-client backup '--crypt-mode=none' pct.conf:/var/tmp/vzdumptmp3836161_100/etc/vzdump/pct.conf root.pxar:/mnt/vzsnap0 --include-dev /mnt/vzsnap0/./ --skip-lost-and-found '--exclude=/tmp/?*' '--exclude=/var/tmp/?*' '--exclude=/var/run/?*.pid' --backup-type ct --backup-id 100 --backup-time 1737100805 --entries-max 1048576 --repository pve1@pbs@pbs0.servers.cturtle98.net:pbs0-16tb-hdd-mirror --ns cTurtle98' failed: exit code 255
INFO: Failed at 2025-01-17 00:00:07
INFO: Starting Backup of VM 101 (lxc)
INFO: Backup started at 2025-01-17 00:00:07
INFO: status = running
INFO: CT Name: assets.k0cdf.us
INFO: including mount point rootfs ('/') in backup
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: create storage snapshot 'vzdump'
INFO: creating Proxmox Backup Server archive 'ct/101/2025-01-17T08:00:07Z'
INFO: set max number of entries in memory for file-based backups to 1048576
INFO: run: lxc-usernsexec -m u:0:100000:65536 -m g:0:100000:65536 -- /usr/bin/proxmox-backup-client backup --crypt-mode=none pct.conf:/var/tmp/vzdumptmp3836161_101/etc/vzdump/pct.conf root.pxar:/mnt/vzsnap0 --include-dev /mnt/vzsnap0/./ --skip-lost-and-found --exclude=/tmp/?* --exclude=/var/tmp/?* --exclude=/var/run/?*.pid --backup-type ct --backup-id 101 --backup-time 1737100807 --entries-max 1048576 --repository pve1@pbs@pbs0.servers.cturtle98.net:pbs0-16tb-hdd-mirror --ns cTurtle98
INFO: Starting backup: [cTurtle98]:ct/101/2025-01-17T08:00:07Z   
INFO: Client name: pve1   
INFO: Starting backup protocol: Fri Jan 17 00:00:07 2025   
INFO: Error: mkstemp "/var/log/proxmox-backup/tasks/5A/UPID:pbs0:00000421:0000075A:000004AA:678A0E07:backup:pbs0\\x2d16tb\\x2dhdd\\x2dmirror\\x3act-101:pve1@pbs:.tmp_XXXXXX" failed: ENOSPC: No space left on device
INFO: cleanup temporary 'vzdump' snapshot
ERROR: Backup of VM 101 failed - command 'lxc-usernsexec -m u:0:100000:65536 -m g:0:100000:65536 -- /usr/bin/proxmox-backup-client backup '--crypt-mode=none' pct.conf:/var/tmp/vzdumptmp3836161_101/etc/vzdump/pct.conf root.pxar:/mnt/vzsnap0 --include-dev /mnt/vzsnap0/./ --skip-lost-and-found '--exclude=/tmp/?*' '--exclude=/var/tmp/?*' '--exclude=/var/run/?*.pid' --backup-type ct --backup-id 101 --backup-time 1737100807 --entries-max 1048576 --repository pve1@pbs@pbs0.servers.cturtle98.net:pbs0-16tb-hdd-mirror --ns cTurtle98' failed: exit code 255
INFO: Failed at 2025-01-17 00:00:07
INFO: Starting Backup of VM 102 (lxc)
INFO: Backup started at 2025-01-17 00:00:07
INFO: status = running
INFO: CT Name: UniFi-Monitor
INFO: including mount point rootfs ('/') in backup
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: create storage snapshot 'vzdump'
INFO: creating Proxmox Backup Server archive 'ct/102/2025-01-17T08:00:07Z'
INFO: set max number of entries in memory for file-based backups to 1048576
INFO: run: lxc-usernsexec -m u:0:100000:65536 -m g:0:100000:65536 -- /usr/bin/proxmox-backup-client backup --crypt-mode=none pct.conf:/var/tmp/vzdumptmp3836161_102/etc/vzdump/pct.conf root.pxar:/mnt/vzsnap0 --include-dev /mnt/vzsnap0/./ --skip-lost-and-found --exclude=/tmp/?* --exclude=/var/tmp/?* --exclude=/var/run/?*.pid --backup-type ct --backup-id 102 --backup-time 1737100807 --entries-max 1048576 --repository pve1@pbs@pbs0.servers.cturtle98.net:pbs0-16tb-hdd-mirror --ns cTurtle98
INFO: Starting backup: [cTurtle98]:ct/102/2025-01-17T08:00:07Z   
INFO: Client name: pve1   
INFO: Starting backup protocol: Fri Jan 17 00:00:07 2025   
INFO: Error: mkstemp "/var/log/proxmox-backup/tasks/5A/UPID:pbs0:00000421:0000075A:000004AB:678A0E07:backup:pbs0\\x2d16tb\\x2dhdd\\x2dmirror\\x3act-102:pve1@pbs:.tmp_XXXXXX" failed: ENOSPC: No space left on device
INFO: cleanup temporary 'vzdump' snapshot
ERROR: Backup of VM 102 failed - command 'lxc-usernsexec -m u:0:100000:65536 -m g:0:100000:65536 -- /usr/bin/proxmox-backup-client backup '--crypt-mode=none' pct.conf:/var/tmp/vzdumptmp3836161_102/etc/vzdump/pct.conf root.pxar:/mnt/vzsnap0 --include-dev /mnt/vzsnap0/./ --skip-lost-and-found '--exclude=/tmp/?*' '--exclude=/var/tmp/?*' '--exclude=/var/run/?*.pid' --backup-type ct --backup-id 102 --backup-time 1737100807 --entries-max 1048576 --repository pve1@pbs@pbs0.servers.cturtle98.net:pbs0-16tb-hdd-mirror --ns cTurtle98' failed: exit code 255
INFO: Failed at 2025-01-17 00:00:07
INFO: Starting Backup of VM 103 (lxc)
INFO: Backup started at 2025-01-17 00:00:07
INFO: status = running
INFO: CT Name: paperless-ngx
INFO: including mount point rootfs ('/') in backup
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: create storage snapshot 'vzdump'
INFO: creating Proxmox Backup Server archive 'ct/103/2025-01-17T08:00:07Z'
INFO: set max number of entries in memory for file-based backups to 1048576
INFO: run: lxc-usernsexec -m u:0:100000:65536 -m g:0:100000:65536 -- /usr/bin/proxmox-backup-client backup --crypt-mode=none pct.conf:/var/tmp/vzdumptmp3836161_103/etc/vzdump/pct.conf root.pxar:/mnt/vzsnap0 --include-dev /mnt/vzsnap0/./ --skip-lost-and-found --exclude=/tmp/?* --exclude=/var/tmp/?* --exclude=/var/run/?*.pid --backup-type ct --backup-id 103 --backup-time 1737100807 --entries-max 1048576 --repository pve1@pbs@pbs0.servers.cturtle98.net:pbs0-16tb-hdd-mirror --ns cTurtle98
INFO: Starting backup: [cTurtle98]:ct/103/2025-01-17T08:00:07Z   
INFO: Client name: pve1   
INFO: Starting backup protocol: Fri Jan 17 00:00:08 2025   
INFO: Error: failed to move file at "/var/log/proxmox-backup/tasks/5A/UPID:pbs0:00000421:0000075A:000004AC:678A0E08:backup:pbs0\\x2d16tb\\x2dhdd\\x2dmirror\\x3act-103:pve1@pbs:.tmp_mVHROH" into place at "/var/log/proxmox-backup/tasks/5A/UPID:pbs0:00000421:0000075A:000004AC:678A0E08:backup:pbs0\\x2d16tb\\x2dhdd\\x2dmirror\\x3act-103:pve1@pbs:" - ENOSPC: No space left on device
INFO: cleanup temporary 'vzdump' snapshot
ERROR: Backup of VM 103 failed - command 'lxc-usernsexec -m u:0:100000:65536 -m g:0:100000:65536 -- /usr/bin/proxmox-backup-client backup '--crypt-mode=none' pct.conf:/var/tmp/vzdumptmp3836161_103/etc/vzdump/pct.conf root.pxar:/mnt/vzsnap0 --include-dev /mnt/vzsnap0/./ --skip-lost-and-found '--exclude=/tmp/?*' '--exclude=/var/tmp/?*' '--exclude=/var/run/?*.pid' --backup-type ct --backup-id 103 --backup-time 1737100807 --entries-max 1048576 --repository pve1@pbs@pbs0.servers.cturtle98.net:pbs0-16tb-hdd-mirror --ns cTurtle98' failed: exit code 255
INFO: Failed at 2025-01-17 00:00:08
INFO: Starting Backup of VM 212 (qemu)
INFO: Backup started at 2025-01-17 00:00:08
INFO: status = running
INFO: VM Name: cloud.cturtle98.com
INFO: include disk 'virtio0' 'DATA:vm-212-disk-5' 8T
INFO: include disk 'efidisk0' 'DATA:vm-212-disk-3' 1M
INFO: include disk 'tpmstate0' 'DATA:vm-212-disk-4' 4M
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: skip unused drive 'DATA:vm-212-disk-0' (not included into backup)
INFO: skip unused drive 'DATA:vm-212-disk-1' (not included into backup)
INFO: skip unused drive 'DATA:vm-212-disk-2' (not included into backup)
INFO: creating Proxmox Backup Server archive 'vm/212/2025-01-17T08:00:08Z'
INFO: attaching TPM drive to QEMU for backup
INFO: issuing guest-agent 'fs-freeze' command
INFO: issuing guest-agent 'fs-thaw' command
ERROR: VM 212 qmp command 'backup' failed - backup connect failed: command error: mkstemp "/var/log/proxmox-backup/tasks/5A/UPID:pbs0:00000421:0000075A:000004AD:678A0E09:backup:pbs0\\x2d16tb\\x2dhdd\\x2dmirror\\x3avm-212:pve1@pbs:.tmp_XXXXXX" failed: ENOSPC: No space left on device
INFO: aborting backup job
INFO: resuming VM again
ERROR: Backup of VM 212 failed - VM 212 qmp command 'backup' failed - backup connect failed: command error: mkstemp "/var/log/proxmox-backup/tasks/5A/UPID:pbs0:00000421:0000075A:000004AD:678A0E09:backup:pbs0\\x2d16tb\\x2dhdd\\x2dmirror\\x3avm-212:pve1@pbs:.tmp_XXXXXX" failed: ENOSPC: No space left on device
INFO: Failed at 2025-01-17 00:00:09
INFO: Starting Backup of VM 228 (qemu)
INFO: Backup started at 2025-01-17 00:00:09
INFO: status = stopped
INFO: backup mode: stop
INFO: ionice priority: 7
INFO: VM Name: AMP
INFO: include disk 'scsi0' 'DATA:vm-228-disk-1' 2T
INFO: include disk 'efidisk0' 'DATA:vm-228-disk-0' 1M
INFO: include disk 'tpmstate0' 'DATA:vm-228-disk-2' 4M
INFO: creating Proxmox Backup Server archive 'vm/228/2025-01-17T08:00:09Z'
INFO: starting kvm to execute backup task
swtpm_setup: Not overwriting existing state file.
INFO: attaching TPM drive to QEMU for backup
ERROR: VM 228 qmp command 'backup' failed - backup connect failed: command error: failed to move file at "/var/log/proxmox-backup/tasks/5A/UPID:pbs0:00000421:0000075A:000004AE:678A0E0A:backup:pbs0\\x2d16tb\\x2dhdd\\x2dmirror\\x3avm-228:pve1@pbs:.tmp_JLqeer" into place at "/var/log/proxmox-backup/tasks/5A/UPID:pbs0:00000421:0000075A:000004AE:678A0E0A:backup:pbs0\\x2d16tb\\x2dhdd\\x2dmirror\\x3avm-228:pve1@pbs:" - ENOSPC: No space left on device
INFO: aborting backup job
INFO: stopping kvm after backup task
ERROR: Backup of VM 228 failed - VM 228 qmp command 'backup' failed - backup connect failed: command error: failed to move file at "/var/log/proxmox-backup/tasks/5A/UPID:pbs0:00000421:0000075A:000004AE:678A0E0A:backup:pbs0\\x2d16tb\\x2dhdd\\x2dmirror\\x3avm-228:pve1@pbs:.tmp_JLqeer" into place at "/var/log/proxmox-backup/tasks/5A/UPID:pbs0:00000421:0000075A:000004AE:678A0E0A:backup:pbs0\\x2d16tb\\x2dhdd\\x2dmirror\\x3avm-228:pve1@pbs:" - ENOSPC: No space left on device
INFO: Failed at 2025-01-17 00:00:10
INFO: Starting Backup of VM 233 (qemu)
INFO: Backup started at 2025-01-17 00:00:10
INFO: status = running
INFO: VM Name: HomeAssistant
INFO: include disk 'virtio0' 'DATA:vm-233-disk-2' 32G
INFO: include disk 'efidisk0' 'DATA:vm-233-disk-0' 1M
INFO: include disk 'tpmstate0' 'DATA:vm-233-disk-1' 4M
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: creating Proxmox Backup Server archive 'vm/233/2025-01-17T08:00:10Z'
INFO: attaching TPM drive to QEMU for backup
INFO: issuing guest-agent 'fs-freeze' command
INFO: issuing guest-agent 'fs-thaw' command
ERROR: VM 233 qmp command 'backup' failed - backup connect failed: command error: mkstemp "/var/log/proxmox-backup/tasks/5A/UPID:pbs0:00000421:0000075A:000004AF:678A0E0B:backup:pbs0\\x2d16tb\\x2dhdd\\x2dmirror\\x3avm-233:pve1@pbs:.tmp_XXXXXX" failed: ENOSPC: No space left on device
INFO: aborting backup job
INFO: resuming VM again
ERROR: Backup of VM 233 failed - VM 233 qmp command 'backup' failed - backup connect failed: command error: mkstemp "/var/log/proxmox-backup/tasks/5A/UPID:pbs0:00000421:0000075A:000004AF:678A0E0B:backup:pbs0\\x2d16tb\\x2dhdd\\x2dmirror\\x3avm-233:pve1@pbs:.tmp_XXXXXX" failed: ENOSPC: No space left on device
INFO: Failed at 2025-01-17 00:00:11
INFO: Backup job finished with errors
TASK ERROR: job errors

I'm confused why it says no space left on device because I have space available on all the devices.

My PBS root drive a a 32gb intel Optane m.2 module. it has space free
1737332685270.png

my pbs data store is 55% full
1737332748646.png

My boot drive on my pve is a raid1 of 500gb m.2 ssds
1737332811180.png
its basically empty.

my guest data storage on my pve is a raidz2 of 8x 8tb hdds
1737332868620.png
its also basically empty.

so where is this no space left on device error coming from?

I don't get it
 
the only think I can think it could be is if the pbs backup index files are stored on he pbs root drive maybe there isn't enough free space for new pbs backups?
 
please post a "df -h" from the PBS system..
 
Code: 
root@pbs0:~# df -h
Filesystem            Size  Used Avail Use% Mounted on
udev                   32G     0   32G   0% /dev
tmpfs                 6.3G  1.6M  6.3G   1% /run
/dev/mapper/pbs-root   27G   26G     0 100% /
tmpfs                  32G     0   32G   0% /dev/shm
tmpfs                 5.0M     0  5.0M   0% /run/lock
efivarfs              512K  121K  387K  24% /sys/firmware/efi/efivars
/dev/nvme0n1p2        511M  344K  511M   1% /boot/efi
pbs0-16tb-hdd-mirror   15T  7.9T  6.6T  55% /mnt/datastore/pbs0-16tb-hdd-mirror
tmpfs                 6.3G     0  6.3G   0% /run/user/0
root@pbs0:~#
 
disk usage says the proxmox backup task logs are taking up all my space.

Code: 
root@pbs0:/# du -h / 2>/dev/null | grep '[0-9\.]\+G'
1.8G    /var/log/journal/39fe4e9b343b44bf92c28b028d990e43
1.8G    /var/log/journal
1.4G    /var/log/proxmox-backup/tasks/F8
1.8G    /var/log/proxmox-backup/tasks/5A
6.3G    /var/log/proxmox-backup/tasks/41
12G     /var/log/proxmox-backup/tasks
12G     /var/log/proxmox-backup
21G     /var/log
21G     /var
^C
root@pbs0:/#
 
I would consider giving PBS more disk space .. to get out of the problem without that, you can delete older logs (maybe sort by size and delete the biggest offenders?)
 
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom