[SOLVED] ZFS Datastore size fixed to usage, free storage not reflected

M

MaeveFirstborn

New Member
Aug 21, 2024
8
3
3
We have a Zpool called sata-mirror-10T. We have two drives (each 10T) in a ZFS Raid1 configuration as the dataset for Proxmox Backup Server. It's mostly full. I noticed that as I cleared out some VMs, the utilization percentage stayed fixed at 99%. This puzzled me until I noticed that the size of the Zpool itself also decreased. When I cleared out some VM snapshots, after a garbage collection of several hundred gigabytes, the size of the datastore apparently shrunk. At the start, there was 9.6TB usage. It's now at 8.6TB. PBS indicates that the total size is now just above 8.6TB, instead of 9.85 which is what we expected.
Code: 
root@PBS:~# zfs list
NAME                      USED  AVAIL  REFER  MOUNTPOINT
sata-mirror-10T          8.92T  45.4G    96K  none
sata-mirror-10T/Backups  8.92T  45.4G  7.83T  /mnt/datastore/Backups

root@PBS:~# zpool list
NAME              SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP    HEALTH  ALTROOT
sata-mirror-10T  9.09T  8.92T   173G        -         -    80%    98%  1.00x    ONLINE  -
Does anyone have any insight as to what's happening here? I suspect it has to do with the subvolume /Backups.
 
A 10TB disk is brutto. For zfs netto do -9% so getting 9.1TB (as shown in zpool list 9.09TB).
Brutto size in never reported in a os as that isn't "useable" because formatted size is required.
In pve or pbs the disk is shown in TiB (which is less) instead of TB.
So everythink looks ok ... until your pool is full but that's not a technical problem ... :)
 
I should clarify, at a previous point in time (we moved this zpool from one system to another) it had 9.85T actually available to be filled, which it nearly was. I then pruned and garbage collected almost a terabyte of data. Instead of actually freeing storage, the max size *shrank* with it. 1735236787319.png
 
For more detailed information you could run zpool list -v + zfs list -o space plus zpool get all + zfs get all sata-mirror-10T/Backups .

"FRAG 80%" is really bad... and this happened because the pool was/is by far too full.

Though it is not clear to me yet why "max size" shrank...
 
Code: 
root@PBS:~# zpool list -v
NAME              SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP    HEALTH  ALTROOT
sata-mirror-10T  9.09T  8.92T   173G        -         -    80%    98%  1.00x    ONLINE  -
  mirror-0       9.09T  8.92T   173G        -         -    80%  98.1%      -    ONLINE
    sdc          9.10T      -      -        -         -      -      -      -    ONLINE
    sdb          9.10T      -      -        -         -      -      -      -    ONLINE

Code: 
root@PBS:~# zfs list -o space
NAME                     AVAIL   USED  USEDSNAP  USEDDS  USEDREFRESERV  USEDCHILD
sata-mirror-10T          45.4G  8.92T       72K     96K             0B      8.92T
sata-mirror-10T/Backups  45.4G  8.92T     1.10T   7.83T             0B         0B

Code: 
root@PBS:~# zpool get all
NAME             PROPERTY                       VALUE                          SOURCE
sata-mirror-10T  size                           9.09T                          -
sata-mirror-10T  capacity                       98%                            -
sata-mirror-10T  altroot                        -                              default
sata-mirror-10T  health                         ONLINE                         -
sata-mirror-10T  guid                           10860928085258184117           -
sata-mirror-10T  version                        -                              default
sata-mirror-10T  bootfs                         -                              default
sata-mirror-10T  delegation                     on                             default
sata-mirror-10T  autoreplace                    off                            default
sata-mirror-10T  cachefile                      -                              default
sata-mirror-10T  failmode                       wait                           default
sata-mirror-10T  listsnapshots                  off                            default
sata-mirror-10T  autoexpand                     on                             local
sata-mirror-10T  dedupratio                     1.00x                          -
sata-mirror-10T  free                           173G                           -
sata-mirror-10T  allocated                      8.92T                          -
sata-mirror-10T  readonly                       off                            -
sata-mirror-10T  ashift                         12                             local
sata-mirror-10T  comment                        -                              default
sata-mirror-10T  expandsize                     -                              -
sata-mirror-10T  freeing                        0                              -
sata-mirror-10T  fragmentation                  80%                            -
sata-mirror-10T  leaked                         0                              -
sata-mirror-10T  multihost                      off                            default
sata-mirror-10T  checkpoint                     -                              -
sata-mirror-10T  load_guid                      10751362249844564116           -
sata-mirror-10T  autotrim                       off                            default
sata-mirror-10T  compatibility                  off                            default
sata-mirror-10T  bcloneused                     0                              -
sata-mirror-10T  bclonesaved                    0                              -
sata-mirror-10T  bcloneratio                    1.00x                          -
sata-mirror-10T  feature@async_destroy          enabled                        local
sata-mirror-10T  feature@empty_bpobj            active                         local
sata-mirror-10T  feature@lz4_compress           active                         local
sata-mirror-10T  feature@multi_vdev_crash_dump  enabled                        local
sata-mirror-10T  feature@spacemap_histogram     active                         local
sata-mirror-10T  feature@enabled_txg            active                         local
sata-mirror-10T  feature@hole_birth             active                         local
sata-mirror-10T  feature@extensible_dataset     active                         local
sata-mirror-10T  feature@embedded_data          active                         local
sata-mirror-10T  feature@bookmarks              enabled                        local
sata-mirror-10T  feature@filesystem_limits      enabled                        local
sata-mirror-10T  feature@large_blocks           enabled                        local
sata-mirror-10T  feature@large_dnode            enabled                        local
sata-mirror-10T  feature@sha512                 enabled                        local
sata-mirror-10T  feature@skein                  enabled                        local
sata-mirror-10T  feature@edonr                  enabled                        local
sata-mirror-10T  feature@userobj_accounting     active                         local
sata-mirror-10T  feature@encryption             enabled                        local
sata-mirror-10T  feature@project_quota          active                         local
sata-mirror-10T  feature@device_removal         enabled                        local
sata-mirror-10T  feature@obsolete_counts        enabled                        local
sata-mirror-10T  feature@zpool_checkpoint       enabled                        local
sata-mirror-10T  feature@spacemap_v2            active                         local
sata-mirror-10T  feature@allocation_classes     enabled                        local
sata-mirror-10T  feature@resilver_defer         enabled                        local
sata-mirror-10T  feature@bookmark_v2            enabled                        local
sata-mirror-10T  feature@redaction_bookmarks    enabled                        local
sata-mirror-10T  feature@redacted_datasets      enabled                        local
sata-mirror-10T  feature@bookmark_written       enabled                        local
sata-mirror-10T  feature@log_spacemap           active                         local
sata-mirror-10T  feature@livelist               enabled                        local
sata-mirror-10T  feature@device_rebuild         enabled                        local
sata-mirror-10T  feature@zstd_compress          enabled                        local
sata-mirror-10T  feature@draid                  enabled                        local
sata-mirror-10T  feature@zilsaxattr             disabled                       local
sata-mirror-10T  feature@head_errlog            disabled                       local
sata-mirror-10T  feature@blake3                 disabled                       local
sata-mirror-10T  feature@block_cloning          disabled                       local
sata-mirror-10T  feature@vdev_zaps_v2           disabled                       local

Code: 
root@PBS:~# zfs get all sata-mirror-10T/Backups
NAME                     PROPERTY              VALUE                          SOURCE
sata-mirror-10T/Backups  type                  filesystem                     -
sata-mirror-10T/Backups  creation              Thu Jun  3 10:47 2021          -
sata-mirror-10T/Backups  used                  8.92T                          -
sata-mirror-10T/Backups  available             45.4G                          -
sata-mirror-10T/Backups  referenced            7.83T                          -
sata-mirror-10T/Backups  compressratio         1.03x                          -
sata-mirror-10T/Backups  mounted               yes                            -
sata-mirror-10T/Backups  quota                 none                           default
sata-mirror-10T/Backups  reservation           none                           default
sata-mirror-10T/Backups  recordsize            128K                           default
sata-mirror-10T/Backups  mountpoint            /mnt/datastore/backups         local
sata-mirror-10T/Backups  sharenfs              off                            default
sata-mirror-10T/Backups  checksum              on                             default
sata-mirror-10T/Backups  compression           lz4                            inherited from sata-mirror-10T
sata-mirror-10T/Backups  atime                 off                            inherited from sata-mirror-10T
sata-mirror-10T/Backups  devices               on                             default
sata-mirror-10T/Backups  exec                  on                             default
sata-mirror-10T/Backups  setuid                on                             default
sata-mirror-10T/Backups  readonly              off                            default
sata-mirror-10T/Backups  zoned                 off                            default
sata-mirror-10T/Backups  snapdir               hidden                         default
sata-mirror-10T/Backups  aclmode               discard                        default
sata-mirror-10T/Backups  aclinherit            restricted                     default
sata-mirror-10T/Backups  createtxg             8045                           -
sata-mirror-10T/Backups  canmount              on                             default
sata-mirror-10T/Backups  xattr                 on                             default
sata-mirror-10T/Backups  copies                1                              default
sata-mirror-10T/Backups  version               5                              -
sata-mirror-10T/Backups  utf8only              off                            -
sata-mirror-10T/Backups  normalization         none                           -
sata-mirror-10T/Backups  casesensitivity       sensitive                      -
sata-mirror-10T/Backups  vscan                 off                            default
sata-mirror-10T/Backups  nbmand                off                            default
sata-mirror-10T/Backups  sharesmb              off                            default
sata-mirror-10T/Backups  refquota              none                           default
sata-mirror-10T/Backups  refreservation        none                           default
sata-mirror-10T/Backups  guid                  17527741978476971169           -
sata-mirror-10T/Backups  primarycache          all                            default
sata-mirror-10T/Backups  secondarycache        all                            default
sata-mirror-10T/Backups  usedbysnapshots       1.10T                          -
sata-mirror-10T/Backups  usedbydataset         7.83T                          -
sata-mirror-10T/Backups  usedbychildren        0B                             -
sata-mirror-10T/Backups  usedbyrefreservation  0B                             -
sata-mirror-10T/Backups  logbias               latency                        default
sata-mirror-10T/Backups  objsetid              4208                           -
sata-mirror-10T/Backups  dedup                 off                            default
sata-mirror-10T/Backups  mlslabel              none                           default
sata-mirror-10T/Backups  sync                  standard                       default
sata-mirror-10T/Backups  dnodesize             legacy                         default
sata-mirror-10T/Backups  refcompressratio      1.03x                          -
sata-mirror-10T/Backups  written               3.44G                          -
sata-mirror-10T/Backups  logicalused           9.18T                          -
sata-mirror-10T/Backups  logicalreferenced     8.04T                          -
sata-mirror-10T/Backups  volmode               default                        default
sata-mirror-10T/Backups  filesystem_limit      none                           default
sata-mirror-10T/Backups  snapshot_limit        none                           default
sata-mirror-10T/Backups  filesystem_count      none                           default
sata-mirror-10T/Backups  snapshot_count        none                           default
sata-mirror-10T/Backups  snapdev               hidden                         default
sata-mirror-10T/Backups  acltype               off                            default
sata-mirror-10T/Backups  context               none                           default
sata-mirror-10T/Backups  fscontext             none                           default
sata-mirror-10T/Backups  defcontext            none                           default
sata-mirror-10T/Backups  rootcontext           none                           default
sata-mirror-10T/Backups  relatime              on                             default
sata-mirror-10T/Backups  redundant_metadata    all                            default
sata-mirror-10T/Backups  overlay               on                             default
sata-mirror-10T/Backups  encryption            off                            default
sata-mirror-10T/Backups  keylocation           none                           default
sata-mirror-10T/Backups  keyformat             none                           default
sata-mirror-10T/Backups  pbkdf2iters           0                              default
sata-mirror-10T/Backups  special_small_blocks  0                              default
sata-mirror-10T/Backups  snapshots_changed     Fri Nov 22 12:10:06 2024       -
sata-mirror-10T/Backups  prefetch              all                            default
 
MaeveFirstborn said:
sata-mirror-10T/Backups usedbysnapshots 1.10T -
sata-mirror-10T/Backups usedbydataset 7.83T -
Click to expand...
So there is still 1.10 TB occupied by snapshots. Is that correct? Is that expected? Are you aware of this? Do you want to keep those?

To drill down the next level: zfs list -t snapshot lists all of them.

If - and only if - you want to get rid of a listed snapshot you may run zfs destroy dataset@snapshotlabel. The import bit is that "@" in the middle. If your "destroy" does not contain an "@namewhatever" it will destroy the dataset in itself. Caution: there is no "are you sure?" question asked!
 
  • Like
Reactions: waltar
AHA! I see what happened! You're a lifesaver! I used a snapshot when I created a snapshot on this pool to clone it at one point. The snapshot is roughly the size of the storage I'm expecting to see freed up. Destroying the snapshot fixed the strange percentage pinning behavior. Could I get some context as to the way snapshot size correlates to the rest of this? Why wasn't this more obvious? I should say I'm relatively new to ZFS.
 
  • Like
Reactions: UdoB
MaeveFirstborn said:
Could I get some context as to the way snapshot size correlates to the rest of this?
Click to expand...

What specifically do you want to know? Search for it, you should be able to find articles describing the whole architecture, including how snapshots work.

PVE uses ZFS snapshots when you create a snapshot of a VM while using ZFS-backed storage. (But a PVE-snapshots are NOT bound to ZFS. See https://pve.proxmox.com/pve-docs/chapter-pvesm.html#_storage_types)

We are talking about PBS. PBS does NOT use any ZFS snapshot capabilities. The backup datastore is filesystem agnostic.

But yeah, you can manually create as many snapshots of datasets as you wish. The will exist until they are removed via the same way :)

Deleting files will not free up actual data storage capacity while snapshots referencing that content do exist. That's the whole point of it! :)


(More specifically: not files but blocks are managed on that level. But for simplicity... the difference does not matter.)
 
  • Like
Reactions: waltar
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom