ZFS wrong used space

[pauet]

I have a raid1 pool for storage VM Disks, but it show a 99,99% used space when I should have some free space.

data/enc 1.76T 262M 192K /data/enc
data/enc/vm-104-disk-0 625G 558G 66.8G -
data/enc/vm-105-disk-0 1.15T 722G 450G -


The VM-104-disk-0 is set to 600GB and VM-105-disk-0 is set to 700GB, this would be 1300GB. There aren't any snapshot.

Is there any problem running a pool with a 99,99% used space? Why there are no free space? And why if I've set a 700GB Raw disk image the Zfs says it's 1.15TB?

Thanks

 

[pauet]

After a few days running everything seems working, but I wonder why is this happening and why I'm unable to create another virtual disk at this storage and make snapshots (I didn't try to create a snapshot).

 

[fabian]

you need to show more information to determine the cause of your space usage. e.g.,

zpool status -v
zpool list -v
zpool get ashift
zfs list -space,compressratio,recordsize,volblocksize -r -t all

in general, you should ensure that your zpools don't get full or even near-full. it's a copy on write filesystem, and if there is no space to quickly allocate for copies, performance will suffer massively.

 

[pauet]

root@kenia:~# zpool status -v
  pool: data
state: ONLINE
  scan: scrub repaired 0B in 0 days 00:00:01 with 0 errors on Sun Nov 10 00:24:02 2019
config:

    NAME                                 STATE     READ WRITE CKSUM
    data                                 ONLINE       0     0     0
      mirror-0                           ONLINE       0     0     0
        ata-ST2000DM006-2DM164_Z4Z8VJAZ  ONLINE       0     0     0
        ata-ST2000DM006-2DM164_Z8E0Q8N8  ONLINE       0     0     0

errors: No known data errors

root@kenia:~# zpool list -v
NAME                                                    SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP    HEALTH  ALTROOT
data                                                   1.81T   971G   885G        -         -     1%    52%  1.00x    ONLINE  -
  mirror                                               1.81T   971G   885G        -         -     1%  52.3%      -  ONLINE 
    ata-ST2000DM006-2DM164_Z4Z8VJAZ                        -      -      -        -         -      -      -      -  ONLINE 
    ata-ST2000DM006-2DM164_Z8E0Q8N8                        -      -      -        -         -      -      -      -  ONLINE

root@kenia:~# zpool get ashift
NAME   PROPERTY  VALUE   SOURCE
data   ashift    0       default

 

[pauet]

root@kenia:~# zfs list -o space,compressratio,recordsize,volblocksize -r -t all
NAME                                                             AVAIL   USED  USEDSNAP  USEDDS  USEDREFRESERV  USEDCHILD  RATIO  RECSIZE  VOLBLOCK
data                                                              241M  1.76T        0B     96K             0B      1.76T  1.00x     128K         -
data@zfs-auto-snap_hourly-2019-11-10-1217                            -     0B         -       -              -          -  1.00x        -         -
data@zfs-auto-snap_hourly-2019-11-10-1317                            -     0B         -       -              -          -  1.00x        -         -
data@zfs-auto-snap_hourly-2019-11-10-1417                            -     0B         -       -              -          -  1.00x        -         -
data@zfs-auto-snap_hourly-2019-11-10-1517                            -     0B         -       -              -          -  1.00x        -         -
data@zfs-auto-snap_hourly-2019-11-10-1617                            -     0B         -       -              -          -  1.00x        -         -
data@zfs-auto-snap_hourly-2019-11-10-1717                            -     0B         -       -              -          -  1.00x        -         -
data@zfs-auto-snap_hourly-2019-11-10-1817                            -     0B         -       -              -          -  1.00x        -         -
data@zfs-auto-snap_hourly-2019-11-10-1917                            -     0B         -       -              -          -  1.00x        -         -
data@zfs-auto-snap_hourly-2019-11-10-2017                            -     0B         -       -              -          -  1.00x        -         -
data@zfs-auto-snap_hourly-2019-11-10-2117                            -     0B         -       -              -          -  1.00x        -         -
data@zfs-auto-snap_hourly-2019-11-10-2217                            -     0B         -       -              -          -  1.00x        -         -
data@zfs-auto-snap_hourly-2019-11-10-2317                            -     0B         -       -              -          -  1.00x        -         -
data@zfs-auto-snap_hourly-2019-11-11-0017                            -     0B         -       -              -          -  1.00x        -         -
data@zfs-auto-snap_hourly-2019-11-11-0117                            -     0B         -       -              -          -  1.00x        -         -
data@zfs-auto-snap_hourly-2019-11-11-0217                            -     0B         -       -              -          -  1.00x        -         -
data@zfs-auto-snap_hourly-2019-11-11-0317                            -     0B         -       -              -          -  1.00x        -         -
data@zfs-auto-snap_hourly-2019-11-11-0417                            -     0B         -       -              -          -  1.00x        -         -
data@zfs-auto-snap_hourly-2019-11-11-0517                            -     0B         -       -              -          -  1.00x        -         -
data@zfs-auto-snap_daily-2019-11-11-0525                             -     0B         -       -              -          -  1.00x        -         -
data@zfs-auto-snap_hourly-2019-11-11-0617                            -     0B         -       -              -          -  1.00x        -         -
data@zfs-auto-snap_hourly-2019-11-11-0717                            -     0B         -       -              -          -  1.00x        -         -
data@zfs-auto-snap_frequent-2019-11-11-0745                          -     0B         -       -              -          -  1.00x        -         -
data@zfs-auto-snap_frequent-2019-11-11-0800                          -     0B         -       -              -          -  1.00x        -         -
data@zfs-auto-snap_frequent-2019-11-11-0830                          -     0B         -       -              -          -  1.00x        -         -
data@zfs-auto-snap_frequent-2019-11-11-0845                          -     0B         -       -              -          -  1.00x        -         -
data/enc                                                          241M  1.76T        0B    192K             0B      1.76T  1.00x     128K         -
data/enc@zfs-auto-snap_daily-2019-11-10-1131                         -     0B         -       -              -          -  1.00x        -         -
data/enc@zfs-auto-snap_daily-2019-11-10-1132                         -     0B         -       -              -          -  1.00x        -         -
data/enc@zfs-auto-snap_hourly-2019-11-10-1217                        -     0B         -       -              -          -  1.00x        -         -
data/enc@zfs-auto-snap_hourly-2019-11-10-1317                        -     0B         -       -              -          -  1.00x        -         -
data/enc@zfs-auto-snap_hourly-2019-11-10-1417                        -     0B         -       -              -          -  1.00x        -         -
data/enc@zfs-auto-snap_hourly-2019-11-10-1517                        -     0B         -       -              -          -  1.00x        -         -
data/enc@zfs-auto-snap_hourly-2019-11-10-1617                        -     0B         -       -              -          -  1.00x        -         -
data/enc@zfs-auto-snap_hourly-2019-11-10-1717                        -     0B         -       -              -          -  1.00x        -         -
data/enc@zfs-auto-snap_hourly-2019-11-10-1817                        -     0B         -       -              -          -  1.00x        -         -
data/enc@zfs-auto-snap_hourly-2019-11-10-1917                        -     0B         -       -              -          -  1.00x        -         -
data/enc@zfs-auto-snap_hourly-2019-11-10-2017                        -     0B         -       -              -          -  1.00x        -         -
data/enc@zfs-auto-snap_hourly-2019-11-10-2117                        -     0B         -       -              -          -  1.00x        -         -
data/enc@zfs-auto-snap_hourly-2019-11-10-2217                        -     0B         -       -              -          -  1.00x        -         -
data/enc@zfs-auto-snap_hourly-2019-11-10-2317                        -     0B         -       -              -          -  1.00x        -         -
data/enc@zfs-auto-snap_hourly-2019-11-11-0017                        -     0B         -       -              -          -  1.00x        -         -
data/enc@zfs-auto-snap_hourly-2019-11-11-0117                        -     0B         -       -              -          -  1.00x        -         -
data/enc@zfs-auto-snap_hourly-2019-11-11-0217                        -     0B         -       -              -          -  1.00x        -         -
data/enc@zfs-auto-snap_hourly-2019-11-11-0317                        -     0B         -       -              -          -  1.00x        -         -
data/enc@zfs-auto-snap_hourly-2019-11-11-0417                        -     0B         -       -              -          -  1.00x        -         -
data/enc@zfs-auto-snap_hourly-2019-11-11-0517                        -     0B         -       -              -          -  1.00x        -         -
data/enc@zfs-auto-snap_daily-2019-11-11-0525                         -     0B         -       -              -          -  1.00x        -         -
data/enc@zfs-auto-snap_hourly-2019-11-11-0617                        -     0B         -       -              -          -  1.00x        -         -
data/enc@zfs-auto-snap_hourly-2019-11-11-0717                        -     0B         -       -              -          -  1.00x        -         -
data/enc@zfs-auto-snap_frequent-2019-11-11-0745                      -     0B         -       -              -          -  1.00x        -         -
data/enc@zfs-auto-snap_frequent-2019-11-11-0800                      -     0B         -       -              -          -  1.00x        -         -
data/enc@zfs-auto-snap_frequent-2019-11-11-0830                      -     0B         -       -              -          -  1.00x        -         -
data/enc@zfs-auto-snap_frequent-2019-11-11-0845                      -     0B         -       -              -          -  1.00x        -         -

 

[pauet]

and the other part (messages limited to 1000)

root@kenia:~# zfs list -o space,compressratio,recordsize,volblocksize -r -t all
NAME                                                             AVAIL   USED  USEDSNAP  USEDDS  USEDREFRESERV  USEDCHILD  RATIO  RECSIZE  VOLBLOCK
data                                                              241M  1.76T        0B     96K             0B      1.76T  1.00x     128K         -

data/enc/vm-104-disk-0                                            109G   625G      773M    515G           109G         0B  1.00x        -        8K
data/enc/vm-104-disk-0@zfs-auto-snap_frequent-2019-11-11-0845        -   773M         -       -              -          -  1.00x        -         -
data/enc/vm-105-disk-0                                            718G  1.15T     3.28G    452G           717G         0B  1.00x        -        8K
data/enc/vm-105-disk-0@zfs-auto-snap_hourly-2019-11-10-1617          -   178M         -       -              -          -  1.00x        -         -
data/enc/vm-105-disk-0@zfs-auto-snap_hourly-2019-11-10-1717          -  38.3M         -       -              -          -  1.00x        -         -
data/enc/vm-105-disk-0@zfs-auto-snap_hourly-2019-11-10-1817          -  6.17M         -       -              -          -  1.00x        -         -
data/enc/vm-105-disk-0@zfs-auto-snap_hourly-2019-11-10-1917          -  35.6M         -       -              -          -  1.00x        -         -
data/enc/vm-105-disk-0@zfs-auto-snap_hourly-2019-11-10-2017          -   113M         -       -              -          -  1.00x        -         -
data/enc/vm-105-disk-0@zfs-auto-snap_hourly-2019-11-10-2117          -  1.41M         -       -              -          -  1.00x        -         -
data/enc/vm-105-disk-0@zfs-auto-snap_hourly-2019-11-10-2217          -  1.38M         -       -              -          -  1.00x        -         -
data/enc/vm-105-disk-0@zfs-auto-snap_hourly-2019-11-10-2317          -  18.1M         -       -              -          -  1.00x        -         -
data/enc/vm-105-disk-0@zfs-auto-snap_hourly-2019-11-11-0017          -  6.42M         -       -              -          -  1.00x        -         -
data/enc/vm-105-disk-0@zfs-auto-snap_hourly-2019-11-11-0117          -  6.80M         -       -              -          -  1.00x        -         -
data/enc/vm-105-disk-0@zfs-auto-snap_hourly-2019-11-11-0217          -  79.6M         -       -              -          -  1.00x        -         -
data/enc/vm-105-disk-0@zfs-auto-snap_hourly-2019-11-11-0317          -  9.48M         -       -              -          -  1.00x        -         -
data/enc/vm-105-disk-0@zfs-auto-snap_hourly-2019-11-11-0417          -     0B         -       -              -          -  1.00x        -         -
data/enc/vm-105-disk-0@zfs-auto-snap_hourly-2019-11-11-0517          -     0B         -       -              -          -  1.00x        -         -
data/enc/vm-105-disk-0@zfs-auto-snap_daily-2019-11-11-0525           -     0B         -       -              -          -  1.00x        -         -
data/enc/vm-105-disk-0@zfs-auto-snap_hourly-2019-11-11-0617          -     0B         -       -              -          -  1.00x        -         -
data/enc/vm-105-disk-0@zfs-auto-snap_hourly-2019-11-11-0717          -  2.16M         -       -              -          -  1.00x        -         -
data/enc/vm-105-disk-0@zfs-auto-snap_frequent-2019-11-11-0745        -  2.23M         -       -              -          -  1.00x        -         -
data/enc/vm-105-disk-0@zfs-auto-snap_frequent-2019-11-11-0800        -  1.53M         -       -              -          -  1.00x        -         -
data/enc/vm-105-disk-0@zfs-auto-snap_frequent-2019-11-11-0830        -     8K         -       -              -          -  1.00x        -         -
data/enc/vm-105-disk-0@zfs-auto-snap_frequent-2019-11-11-0845        -     8K         -       -              -          -  1.00x        -         -

 

[fabian]

you are using zvols with reservation and snapshots, so your zvols will always take up their full size + any data only referenced by snapshots. e.g., vm-105-disk-0 seems to currently take up:
- 3G for data only referenced by one or more snapshots
- 452G for data currently referenced by the dataset (and potentially also by one or more snapshots)
- 717G for reserved space (needed to allow you to successfully overwrite the whole zvol)