Disk Full during Backup, no longer responsive

[scrub_nub888]

Hi All,

I had an automated backup setup. This backup was sent to a separate PBS node with plenty of storage. For some reason the node storage filled up to 100% and became unresponsive. I'm thinking the backup job filled up the local storage first? I'm not sure how to go about this. I can only access CLI physically on the node.

Any ideas?

 

[sb-jw]

First of all, check which folder or partition has actually filled up. You can do this with "df" for mountpoints and with "du" for folders. Only then will it be possible to see what has been filled up here.

 

[scrub_nub888]

Here's what i'm seeing. I see pve-1 is full but when I go in there there isn't much?

Not sure what I should/shouldn't delete

 

[leesteken]

The command from this thread might help find the space usage, and check /var/log.

 

[scrub_nub888]

The command from this thread might help find the space usage, and check /var/log

Awesome, that was helpful, I was able to find a file/directory called /var/tmp/vzdumptmp84023_106 that is 1.2T! How can I delete this? I tried rm -rf but that didn't seem to work

 

[leesteken]

Awesome, that was helpful, I was able to find a file/directory called /var/tmp/vzdumptmp84023_106 that is 1.2T! How can I delete this? I tried rm -rf but that didn't seem to work

Please be careful with rm -rf. It's a temporary file created during a backup, so that makes sense. rm /var/tmp/vzdumptmp84023_106 should work. What error are you getting?

 

[scrub_nub888]

Please be careful with rm -rf. It's a temporary file created during a backup, so that makes sense. rm /var/tmp/vzdumptmp84023_106 should work. What error are you getting?

gotcha, i was able to get it deleted - thank you!

How can I prevent backups from filling up this local storage?

 

[sb-jw]

How can I prevent backups from filling up this local storage?

As the saying goes, a lot helps a lot - at least that's the first option.
Are you actually using that much storage space or is there probably still a lot of free space in the VM? Do you use the discard flag on the VMs?

 

[scrub_nub888]

As the saying goes, a lot helps a lot - at least that's the first option.
Are you actually using that much storage space or is there probably still a lot of free space in the VM? Do you use the discard flag on the VMs?

so I have 2 pools an ssd pool and hdd pool.

ssd pool (proxmox is also installed on here) is 1.91TB with 1.32TB Free
hdd pool is 31.99TB with 22.36TB Free

This VM I was backing up was 2.4TB and is on the HDD pool, I guess it used the local drive (on the ssd pool)?

 

[leesteken]

gotcha, i was able to get it deleted - thank you!

How can I prevent backups from filling up this local storage?

You can set the tmpdir in /etc/vzdump.conf. Maybe using a directory on the HDD is better than the default /var/tmp on the SSD?

 

[scrub_nub888]

You can set the tmpdir in /etc/vzdump.conf. Maybe using a directory on the HDD is better than the default /var/tmp on the SSD?

awesome, got it setup. thank you!!!