[SOLVED] Backups not working on hosts no longer part of cluster

[Curt Hall]

Not sure if the cluster really has anything to do with it but it was the most recent change that seemed significant. I always thought backups were not cluster aware. Each host just needed access to the backup location.

This is the error, which looks like a drive out space issue, but the backup datastore has 5TB free, and I can't even back up 1 80GB VM?

INFO: starting new backup job: vzdump 100 --all 0 --node proxmox6 --quiet 1 --storage PBS --mode snapshot --mailnotification always
INFO: Starting Backup of VM 100 (qemu)
INFO: Backup started at 2022-04-02 15:37:10
INFO: status = running
INFO: VM Name: NorthRMZ-1.57
INFO: include disk 'scsi0' 'RAIDZ:vm-100-disk-0' 80G
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: creating Proxmox Backup Server archive 'vm/100/2022-04-02T20:37:10Z'
ERROR: VM 100 qmp command 'backup' failed - backup connect failed: command error: write failed: No space left on device (os error 28)
INFO: aborting backup job
INFO: resuming VM again
ERROR: Backup of VM 100 failed - VM 100 qmp command 'backup' failed - backup connect failed: command error: write failed: No space left on device (os error 28)
INFO: Failed at 2022-04-02 15:37:10
INFO: Backup job finished with errors
TASK ERROR: job errors

All hosts: PVE version 6.4-14
PBS version 2.1-5

 

[Curt Hall]

Note size of each VM in PBS datastore just has spinning circle.

 

[dcsapak]

can you post the output of

cat /etc/proxmox-backup/datastore.cfg
df -h
df -i

on the pbs ?

 

[Curt Hall]

 

[dcsapak]

your root filesystem '/' is full, so i guess some lock or tmpfile cannot be created..

 

[Curt Hall]

yeah one test backup tried to go to root drive instead of the actual backup drive and filled it up, I'm freeing up space today.

 

[Curt Hall]

Everything working now. I guess the root partition needed some free space after all.