Deleting Files Doesn't Free Space

[xaragon]

Hello!

I have an issue I can't seem to find a fix for, I have read trough loads of threads and can't seem to find a solution.

I have tested mounting and unmounting the pool, even exporting and importing but when deleting files it still uses the same amount of space. I have changed xattr to sa and it have helped me to free space on newly deleted files.

There are no snapshots present so that can't be the problem.

Has anyone seen this behavior before and is there a fix? Have struggled with this for about a week now and my hair is starting the turn grey.

Have also included a couple of codeblocks that might shed some light on the issue.

This is on zfs-initramfs 0.6.5-pve6~jessie and Proxmox v4.2-1

root@prox:~# df -h
Filesystem            Size  Used Avail Use% Mounted on
udev                   10M     0   10M   0% /dev
tmpfs                 6.3G   25M  6.3G   1% /run
/dev/dm-0              30G  1.5G   27G   6% /
tmpfs                  16G   43M   16G   1% /dev/shm
tmpfs                 5.0M     0  5.0M   0% /run/lock
tmpfs                  16G     0   16G   0% /sys/fs/cgroup
/dev/sdc1             917G   64G  807G   8% /mnt/evo1tb
/dev/sda1             2.7T  1.2T  1.4T  46% /mnt/red3tb
/dev/mapper/pve-data   59G   52M   59G   1% /var/lib/vz
priv_vol1             3.6T  3.2T  399G  89% /mnt/priv_vol1
tmpfs                 100K     0  100K   0% /run/lxcfs/controllers
cgmfs                 100K     0  100K   0% /run/cgmanager/fs
/dev/fuse              30M   16K   30M   1% /etc/pve
vol1                  1.9T  916G  965G  49% /mnt/vol1
vol1/share1           9.0T  8.1T  965G  90% /mnt/vol1/share1
vol1/shared           1.3T  339G  965G  26% /mnt/vol1/shared
root@prox:~#

root@prox:~# zpool list -o name,size,allocated,free,freeing
NAME        SIZE  ALLOC   FREE  FREEING
priv_vol1  3.62T  3.12T   515G        0
vol1       14.5T  12.8T  1.75T        0
root@prox:~#

root@prox:~# zfs list -t snapshot
no datasets available
root@prox:~#

root@prox:~# uname -a
Linux prox 4.2.6-1-pve #1 SMP Wed Dec 9 10:49:55 CET 2015 x86_64 GNU/Linux
root@prox:~#

root@prox:~# cat /etc/debian_version
8.4
root@prox:~#

Best regards Alex

 

[LnxBil]

Hi Alex,

How did you free the files? What files and where? Which pool has the problem?

Please post with CODE-Tag for preformatted text and display the filesystems in question

Best,
LnxBil

 

[xaragon]

Thanks for your time LnxBil!

I have now changed the original post to include CODE-tags. Should be a bit easier to read now.

The problem is on both pools. as you can see if you compare "df -h" output and "zpool list" output. On pool vol1 I have destroyed a couple of datasets that had the files on them and to see if this also occurred on priv_vol1 I used the rm-command. Same thing happened there as well.

I could see this problem occurring if I had any snapshots as it would hold the files. But as I don't have any snapshots, I should add that there were snapshots before that I deleted before I knew there was something strange going on.

Also tested to create a new snapshot and remove it but that had no effect. I am pretty stunned by the problem hoping there is some way to fix the issue.

 

[LnxBil]

Normally your zpool shows different free and used stuff due to the internal metadata. You have to compare zfs list vs. df. Could you please add the output of:

zfs list -o name,avail,used,refer,lused,lrefer,mountpoint,compress,compressratio

 

[xaragon]

root@prox:~# zfs list -o name,avail,used,refer,lused,lrefer,mountpoint,compress,compressratio
NAME         AVAIL   USED  REFER  LUSED  LREFER  MOUNTPOINT        COMPRESS  RATIO
priv_vol1     399G  3.12T  3.12T  3.15T   3.15T  /mnt/priv_vol1         lz4  1.00x
vol1          964G  9.26T   916G  9.46T   1.06T  /mnt/vol1              lz4  1.02x
vol1/share1   964G  8.04T  8.04T  8.06T   8.06T  /mnt/vol1/share1       lz4  1.00x
vol1/shared   964G   339G   339G   346G    346G  /mnt/vol1/shared       lz4  1.02x

root@prox:~# du -sh /mnt/vol1
9.3T    /mnt/vol1
root@prox:~#

Sure enough it looks like "df" and that output corresponds with each other. Does this look right to you. I could have sworn that I lost a lot of space. But what does the free and alloc column mean since they don't seem to match?

I also did checked the total space with du-command and sure enough it seems to be spot on.

 

[vkhera]

For reference, the reason for setting xattr=sa is here: https://github.com/zfsonlinux/zfs/issues/1548

And as you discovered, it only applies to newly created files. I'm not sure why you were observing the space not being reclaimed on export/import of the pool.