PBS on ZFS doesn't release space

[fitful]

Hello,
Does anyone know how to solve the problem of not freeing up space in ZFS?

I have two physical machines:

pve1 and pve2
On pve1, as you can see, there is a PROD-PBS machine that has a disk resource allocated from pve1 (practically all of it for PBS)

The "backup" resource is linked for the copy resource from PBS.

As you can see in the screenshot (1) on the "backup" resource the space occupancy oscillates at one level all the time.


On the other hand, in the screenshot (2) you can see that the space is being eaten up all the time - the question is why this space is not being released?



(3) PROD-PBS settings


I will be grateful for any help.

 

[Dunuin]

What does zfs list -o space return?
Maybe any snapshots present, because this would prevent ZFS from freeing up space.

 

[fitful]

Unfortunately that's not it, that's what I checked in the first place.
I'll just add that PVE1 as well as PBS use ZFS as the file system.

As a test I added another disk to PBS, but this time I made an ext4 file system on it, I'll check in a moment if it frees up space.

 

[fitful]

Unfortunately, changing the file system for the datastore did not help. You may not see it clearly, but there is a trend that "local-zfs" on the pve1 host will grow relative to "backup"

Now there is such a configuration:



PROD-PBS


Probably when I return in a while local-zfs will grow until the space is exhausted.

This is not a specific configuration, everyone is probably able to reproduce it at their place.
I would like to know the dependency and how to solve it.

I'm almost convinced it's due to the fact that pve1 exposes the resource via ZFS, because if pve1 had a filesystem like ext4 then the problem wouldn't exist -
I mention this because I had this problem before, but I didn't go into details just did ext4 on a test basis and haven't had a space problem since.

 

[websmith]

It might be as simple as you not having enabled trim on the backup server.

i.e.
systemctl enable --now fstrim.timer

If discard is not enabled, and the system does not trim from time to time - there are no way space can be relaimed.

You can try to run a simple trim directly, i.e.

fstrim /backup

If that releases the space, its the missing periodic trim that is the culprit

 

[fitful]

It might be as simple as you not having enabled trim on the backup server.

i.e.
systemctl enable --now fstrim.timer

If discard is not enabled, and the system does not trim from time to time - there are no way space can be relaimed.

You can try to run a simple trim directly, i.e.

fstrim /backup

If that releases the space, its the missing periodic trim that is the culprit

yes, what you say would be correct for e.g. ext4, but at the time of not freeing up space the file system on PBS was ZFS

Ok I'm back, just with an update
After changing the file system to ext4 for PBS, fortunately the space freed up.
It's a little strange that no one observes such a case.

 

[websmith]

yes, what you say would be correct for e.g. ext4, but at the time of not freeing up space the file system on PBS was ZFS

Ok I'm back, just with an update
After changing the file system to ext4 for PBS, fortunately the space freed up.
It's a little strange that no one observes such a case.

ZFS only trims if its enabled.

And I am pretty sure consensus is to both enable autotrim zpool set autotrim=on on the pool and also some cron job that manually calls zpool trim

But still its all moot - if discard is not enabled on the VM disks, but you already have that - so my guess is that autotrim was not enabled - and you did not have a cron job that called zpool trim.

 

[Dunuin]

And I am pretty sure consensus is to both enable autotrim zpool set autotrim=on on the pool and also some cron job that manually calls zpool trim

Default settings is autrotrim=off and a systemd timer that runs zpool trim once per month.

 

[websmith]

Default settings is autrotrim=off and a systemd timer that runs zpool trim once per month.

And given the screenshots of space usage above are less than two weeks - its likely its the trim that simply has not run