Storage full - Delete vm without effect

[Schello805]

Dear All,
Since weeks, I try to find a solution for the problem, that the Storage of one of my Disk at the pbs, is full with 0Bytes free space. If I start the Garbage collection, I receive an error with:

Garbage collection failed: update atime failed for chunk/file "/mnt/datastore/Disk1/.chunks/bb34/bb34829d6e1764c92e532c54f77d2c5409aed168620c5fa9b100d7158ef6b7bb" - ENOSPC: No space left on device

 

[Dunuin]

Only option I see without losing data is to buy a bigger disk and copy all files over.

If you don't care about the backups you could try to delete some of the index files to free up a few KBs/MBs of data so the GC might work.
But then those backup snapshots, where you deleted the index files, will be lost.

Maybe you could also check if there is a snapshot you could delete or manually do a fstrim to free up some space.

 

[Schello805]

Only option I see without losing data is to buy a bigger disk and copy all files over.

If you don't care about the backups you could try to delete some of the index files to free up a few KBs/MBs of data so the GC might work.
But then those backup snapshots, where you deleted the index files, will be lost.

Maybe you could also check if there is a snapshot you could delete or manually do a fstrim to free up some space.

Thanks for your answer. It's a local server at my home. I already deleted around 500GB of backups without any effect :-( You recommend to do a fstrim -a? I´ll try it tomorrow. Thank you.

 

[Dunuin]

You recommend to do a fstrim -a?

Depends on your storage and how you set up your PBS. Might also be a "zpool trim" or "pct fstrim".

I already deleted around 500GB of backups without any effect

How?

 

[itNGO]

Isnt't it, that you need to run a Garbage Collection to realy free up the space?

 

[Schello805]

Depends on your storage and how you set up your PBS. Might also be a "zpool trim" or "pct fstrim".

How?

Strange... I cli I respond 50GB of free space.

But the Frontend say --> No Space. When I run the Garbage collection, I receive the Error... with the Email

Datastore: Disk1

Garbage collection failed: update atime failed for chunk/file "/mnt/datastore/Disk1/.chunks/ae43/ae43335b984a6c2a1bde30904928dca3b0e53885579539def13282d6ca63696f" - ENOSPC: No space left on device


Please visit the web interface for further details:

<https://pbs.schello:8007/#pbsServerAdministration:tasks>

Anyone another Idea :-(

 

[Dunuin]

But the Frontend say --> No Space. When I run the Garbage collection, I receive the Error... with the Email

"zpool" command shows free/used raw capacity. For actual space use the "zfs" command. zpool list -v && zfs list -o space should give you a better overview.

 

[Schello805]

"zpool" command shows free/used raw capacity. For actual space use the "zfs" command. zpool list -v && zfs list -o space should give you a better overview.

Yes, that shows, what the problem since beginning is. I have no space and don't know, how I delete files to get some free space :-(

 

[Dunuin]

Don't you got an empty bigger disk or a NAS that can fit like 2.5TB of data? Then I would move the datastore there, run the GC, and move the datastore back later. A datastore is just a normal folder with millions of files and subfolders. So you could simple move it with the tool of your choice, like rsync.
Also keep in mind that a ZFS pool will become slow when it gets over 90% filled. For best performance you might not want to fill it more than 80%. I personally always set a quota of 90% so you can't completely fill that pool by accident. And then I set up monitoring, so I get notified to add more disks or delete some stuff once the disk is more than 80% filled.

 

[Schello805]

Don't you got an empty bigger disk or a NAS that can fit like 2.5TB of data? Then I would move the datastore there, run the GC, and move the datastore back later. A datastore is just a normal folder with millions of files and subfolders. So you could simple move it with the tool of your choice, like rsync.
Also keep in mind that a ZFS pool will become slow when it gets over 90% filled. For best performance you might not want to fill it more than 80%. I personally always set a quota of 90% so you can't completely fill that pool by accident. And then I set up monitoring, so I get notified to add more disks or delete some stuff once the disk is more than 80% filled.

The server just accept max. 2 TB (it's an older one) - Thanks for your idea, but that seems to be a huge work. Isn't there any other solution, to get free space?

 

[Dunuin]

Isn't there any other solution, to get free space?

Not without sacrificing some backups. The key is to prevent completely filling up your pool in the first place.
Once you do there is no easy way to free up space.

 

[Schello805]

Thanks to all, but nothing helps. So I deleted the whole disk and initilise it again. So the disk is empty now and can be using again.

 

[Dunuin]

Then don't forget to set quotas this time so you won't end up in the same situation again.

 

[Schello805]

Then don't forget to set quotas this time so you won't end up in the same situation again.

Thanks for the reminder, but... how? I cannot find a view to set the quota.

 

[Dunuin]

You need to do that in CLI. Lets say you got a dataset "mypool/mydataset" and you want to limit it to 1.6TB because you got a 2TB disk: zfs set quota=1.6T mypool/mydataset

 

[Schello805]

You need to do that in CLI. Lets say you got a dataset "mypool/mydataset" and you want to limit it to 1.6TB because you got a 2TB disk: zfs set quota=1.6T mypool/mydataset

Do you know, why this not works?

 

[Dunuin]

You need to point it to the pool or dataset, not to its mountpoint. So in your case I would use: zfs set quota=1.59T Disk2 and zfs set quota=1.59T Disk1
That way always 10% of the pools should be kept free (which you usually want anyway because the pool will become slow when filling it more than 90%). And if you ever hit that quota you can temporarily disable your backup tasks, increase the quota again to have access to the full space, delete backups, wait for 24 hours, run the GC and then lower the quota again to 1.59T.

 

[Schello805]

You need to point it to the pool or dataset, not to its mountpoint. So in your case I would use: zfs set quota=1.59T Disk2 and zfs set quota=1.59T Disk1

Ah... Ok, thanks - no it works