Disk Full - can not delete old backups.

[mskenderian]

My disk is full and i cant prune the old backups.

I tried clearing out old log files in /var/log/
but it was not much since this system is only a few months old.

What else can I do? what can be deleted?

I wish there was a setting where I can set on the datastore where it will reserve x MB for thing like GC and prune.

 

[_gabriel]

you can move TEMPORARY some folders from your ".chunks" to another disk then do a Garbage Collection, then move again folder to ".chunks"

 

[mskenderian]

Is there anyway? the machine is remote and i dont have another disk?

I really wish pbs had a setting to not let it get full, to always leave enough space to be able to prune.

 

[sb-jw]

I really wish pbs had a setting to not let it get full, to always leave enough space to be able to prune.

Proxmox is not responsible for your monitoring, you are responsible for your systems in every respect.

Is there anyway?

You have to delete data for GC to work, no matter what. Maybe an apt clean or apt autoremove will help you, you can also delete old logs. But no one can help you in the dark, then you have to say something about the setup and your partition.

 

[Dunuin]

I really wish pbs had a setting to not let it get full, to always leave enough space to be able to prune.

Use a filesystem like ZFS that allows you to set quotas. Then you can't fill it up to a point where it is bricked. I always set up a quota of 90%, so when I completely fill it up by acciedent, I could always increase the quota from 90 to 95% to get lots of free space to run the GC.

And no, there isn`t much you can do except for deleting some non-datastore data or adding more storage.

 

[mskenderian]

ahhh, I didnt know ZFS has quota limit, ok thats perfect.
So i dont mind deleting data. I deleted some backup, but the space doesnt free up? last time i did this, and i also waiting for over 1 day and it the same results.

How can I free the data? I thought I would need to run GC. but i cant.

 

[ubu]

Please show output of df -h and zfs list

 

[mskenderian]

df -h



zfs list

 

[Dunuin]

ahhh, I didnt know ZFS has quota limit, ok thats perfect.
So i dont mind deleting data. I deleted some backup, but the space doesnt free up? last time i did this, and i also waiting for over 1 day and it the same results.

How can I free the data? I thought I would need to run GC. but i cant.

Deleting backups won't help as 99,99% of the backups are chunks that can only be deleted by running the GC which you can't without some free space...

You could delete the whole datastore folder but then of cause all backups are gone.

 

[mskenderian]

Is there another option?

Previously I had one dataset, with 1 namespace per PVE. My deduplication ratio was super high. For the sake of visual representation on the gui, I make a dataset per PVE. Ate up my storage quickly, since I have learned dedups don’t happen across datasets.

If that’s my only choice, I can delete one of the datasets.

Some PVE boxes are local, I can backup quickly.

 

[sb-jw]

If that’s my only choice, I can delete one of the datasets.

It's better to delete something in a controlled manner than to delete fragments from a large number of systems whose integrity is no longer given afterwards and you may not even notice it.

 

[mskenderian]

i didnt a datastore that i dont need. i tried to remove it from the gui, but again failed since there is no space.
i deleted the whole folder from terminal. but the space is free.

I am not sure what else to do.

 

[sb-jw]

Then you should now trigger the GC again.

 

[mskenderian]

@sb-jw stil I still cant, it says no free space.
after i deleted the files, running df -h or zfs list, has the same output, still hasnt free the space.

 

[ubu]

Check if you have a snapshot on your zfs, zfs list -t snapshots, and delete them to free the space

 

[mskenderian]

No I don’t have snapshots.

 

[Dunuin]

fstrim -a and zpool trim rpool?

 

[mskenderian]

both output nothing.

 

[ubu]

Ok, download some of your backup data from the .chunks directory over scp and delete it on remote machine, then do the backup prune, afterwards copy it back and prune again

 

[ubu]

Afterwards set quota so thi wont happen again:
zfs set refquota=850G rpool/encrypted_data, if you run out of space the you can TEMPORARILY set the quota higher, you should not fill your Filesystems over 80-90% anyway