[SOLVED] Backup fails do to out of space but there is free space

[adamb]

Running into a issue were backups are failing and reporting out of space even though the PBS mount still has over 8TB free.

ERROR: Backup of VM 102 failed - VM 102 qmp command 'backup' failed - backup connect failed: command error: mkstemp "/var/log/proxmox-backup/tasks/6C/UPID:bunkproxbackup1:00000331:0000036C:0001423D:62973D34:backup:Migratebackup\\x3avm-102:root@pam:.tmp_XXXXXX" failed: ENOSPC: No space left on device

File system looks aok.

root@migrateprox:~# df -h
Filesystem Size Used Avail Use% Mounted on
udev 189G 0 189G 0% /dev
tmpfs 38G 2.6M 38G 1% /run
/dev/mapper/pve-root 94G 15G 75G 17% /
tmpfs 189G 46M 189G 1% /dev/shm
tmpfs 5.0M 0 5.0M 0% /run/lock
/dev/sda2 253M 288K 252M 1% /boot/efi
Migrate 33T 256K 33T 1% /Migrate
/dev/fuse 128M 20K 128M 1% /etc/pve
Storage 22T 128K 22T 1% /Storage
tmpfs 38G 0 38G 0% /run/user/0

The PBS Backup storage shows at 80.09% (34.60TB of 43.21TB)

 

[dcsapak]

what do the inodes say?

df -i /var/log/proxmox-backup

 

[adamb]

what do the inodes say?

df -i /var/log/proxmox-backup

root@migrateprox:/var/log# df -i /var/log/proxmox-backup
df: /var/log/proxmox-backup: No such file or directory

root@migrateprox:/var/log# df -i
Filesystem Inodes IUsed IFree IUse% Mounted on
udev 49505972 1381 49504591 1% /dev
tmpfs 49514237 2438 49511799 1% /run
/dev/mapper/pve-root 6291456 137529 6153927 3% /
tmpfs 49514237 99 49514138 1% /dev/shm
tmpfs 49514237 23 49514214 1% /run/lock
/dev/sda2 0 0 0 - /boot/efi
Migrate 69711747731 6 69711747725 1% /Migrate
/dev/fuse 262144 37 262107 1% /etc/pve
Storage 46169914603 6 46169914597 1% /Storage
tmpfs 9902847 18 9902829 1% /run/user/0

 

[dcsapak]

root@migrateprox:/var/log# df -i /var/log/proxmox-backup
df: /var/log/proxmox-backup: No such file or directory

you must run that on the pbs

 

[adamb]

you must run that on the pbs

Here it is.

root@bunkproxbackup1:~# df -i /var/log/proxmox-backup
Filesystem Inodes IUsed IFree IUse% Mounted on
/dev/mapper/pbs-root 31162368 1771864 29390504 6% /

This looks to be my issue, never thought about the root partition being filled on the PBS server.

I had a mount point on the server for a normal rsync backup which I removed, but forgot to remove the crontab's on the source server and it filled up / on the pbs server. Whooops

root@bunkproxbackup1:~# df -h /var/log/proxmox-backup
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/pbs-root 467G 465G 0 100% /

Thanks, for all the input!