ZFS usable size in container limited to 9GB

mvleening

New Member
Feb 3, 2024
2
0
1
Hello,

I have a fresh installation of PVE 8.1.4 with a ZFS pool:

Code:
#> zpool status
  pool: storage
 state: ONLINE
  scan: scrub repaired 0B in 00:03:56 with 0 errors on Sun Jan 14 00:27:57 2024
config:

    NAME        STATE     READ WRITE CKSUM
    storage     ONLINE       0     0     0
      raidz2-0  ONLINE       0     0     0
        sdb     ONLINE       0     0     0
        sdc     ONLINE       0     0     0
        sdd     ONLINE       0     0     0
        sde     ONLINE       0     0     0
        sdf     ONLINE       0     0     0

errors: No known data errors

Created a CT with a root volume of 50GB. Everything appears to work okay, just that when I try to copy files onto the disk the used size stays stuck at 9GB

Code:
Filesystem                 Size  Used Avail Use% Mounted on
storage/subvol-102-disk-0   50G  9.0G   42G  18% /

Even though I copied 27GB onto the disk without any error, the total used space stays stuck at 9GB

Therefore, I suspect it has something to do with the pool, but of course am not sure, hence me being here :-)

Container config:

Code:
#> pct config 102
arch: amd64
cores: 4
features: nesting=1
hostname: test-restore
memory: 1024
net0: name=eth0,bridge=vmbr0,firewall=1,hwaddr=BC:24:11:09:6B:CB,ip=dhcp,ip6=dhcp,type=veth
ostype: ubuntu
rootfs: storage:subvol-102-disk-0,size=50G
swap: 512
unprivileged: 1

What could I check to figure out the problem? There are no discernible quota on the pool or the volumes.

I've also tried to create a file in the volume on the node itself

Code:
root@proxmox:/storage/subvol-102-disk-0/data# dd if=/dev/zero of=1g.img bs=1 count=0 seek=1G
0+0 records in
0+0 records out
0 bytes copied, 0.000182609 s, 0.0 kB/s

Which looks like it does not work, but also gives no error and the total used space stays at 9.0G, however the file is created:

Code:
0 bytes copied, 0.000182609 s, 0.0 kB/s
root@proxmox:/storage/subvol-102-disk-0/tmp# ls -la
total 16
-rw-r--r--  1 root   root   1073741824 Feb  3 12:20 1g.img

Here's the output of zfs get all storage/subvol-102-disk-0 where you can see that the value of logicalused/logicalreferenced is 28.8G - which is the amount of data I wanted to put on the disk

Code:
NAME                       PROPERTY              VALUE                       SOURCE
storage/subvol-102-disk-0  type                  filesystem                  -
storage/subvol-102-disk-0  creation              Fri Feb  2 17:00 2024       -
storage/subvol-102-disk-0  used                  8.98G                       -
storage/subvol-102-disk-0  available             41.0G                       -
storage/subvol-102-disk-0  referenced            8.98G                       -
storage/subvol-102-disk-0  compressratio         3.60x                       -
storage/subvol-102-disk-0  mounted               yes                         -
storage/subvol-102-disk-0  quota                 none                        default
storage/subvol-102-disk-0  reservation           none                        default
storage/subvol-102-disk-0  recordsize            128K                        default
storage/subvol-102-disk-0  mountpoint            /storage/subvol-102-disk-0  default
storage/subvol-102-disk-0  sharenfs              off                         default
storage/subvol-102-disk-0  checksum              on                          default
storage/subvol-102-disk-0  compression           lz4                         inherited from storage
storage/subvol-102-disk-0  atime                 off                         inherited from storage
storage/subvol-102-disk-0  devices               on                          default
storage/subvol-102-disk-0  exec                  on                          default
storage/subvol-102-disk-0  setuid                on                          default
storage/subvol-102-disk-0  readonly              off                         default
storage/subvol-102-disk-0  zoned                 off                         default
storage/subvol-102-disk-0  snapdir               hidden                      default
storage/subvol-102-disk-0  aclmode               discard                     default
storage/subvol-102-disk-0  aclinherit            restricted                  default
storage/subvol-102-disk-0  createtxg             829751                      -
storage/subvol-102-disk-0  canmount              on                          default
storage/subvol-102-disk-0  xattr                 sa                          local
storage/subvol-102-disk-0  copies                1                           default
storage/subvol-102-disk-0  version               5                           -
storage/subvol-102-disk-0  utf8only              on                          -
storage/subvol-102-disk-0  normalization         formD                       -
storage/subvol-102-disk-0  casesensitivity       sensitive                   -
storage/subvol-102-disk-0  vscan                 off                         default
storage/subvol-102-disk-0  nbmand                off                         default
storage/subvol-102-disk-0  sharesmb              off                         default
storage/subvol-102-disk-0  refquota              50G                         local
storage/subvol-102-disk-0  refreservation        none                        default
storage/subvol-102-disk-0  guid                  15629937967884957536        -
storage/subvol-102-disk-0  primarycache          all                         default
storage/subvol-102-disk-0  secondarycache        all                         default
storage/subvol-102-disk-0  usedbysnapshots       0B                          -
storage/subvol-102-disk-0  usedbydataset         8.98G                       -
storage/subvol-102-disk-0  usedbychildren        0B                          -
storage/subvol-102-disk-0  usedbyrefreservation  0B                          -
storage/subvol-102-disk-0  logbias               latency                     default
storage/subvol-102-disk-0  objsetid              6792                        -
storage/subvol-102-disk-0  dedup                 off                         default
storage/subvol-102-disk-0  mlslabel              none                        default
storage/subvol-102-disk-0  sync                  standard                    default
storage/subvol-102-disk-0  dnodesize             legacy                      default
storage/subvol-102-disk-0  refcompressratio      3.60x                       -
storage/subvol-102-disk-0  written               8.98G                       -
storage/subvol-102-disk-0  logicalused           28.8G                       -
storage/subvol-102-disk-0  logicalreferenced     28.8G                       -
storage/subvol-102-disk-0  volmode               default                     default
storage/subvol-102-disk-0  filesystem_limit      none                        default
storage/subvol-102-disk-0  snapshot_limit        none                        default
storage/subvol-102-disk-0  filesystem_count      none                        default
storage/subvol-102-disk-0  snapshot_count        none                        default
storage/subvol-102-disk-0  snapdev               hidden                      default
storage/subvol-102-disk-0  acltype               posix                       local
storage/subvol-102-disk-0  context               none                        default
storage/subvol-102-disk-0  fscontext             none                        default
storage/subvol-102-disk-0  defcontext            none                        default
storage/subvol-102-disk-0  rootcontext           none                        default
storage/subvol-102-disk-0  relatime              on                          default
storage/subvol-102-disk-0  redundant_metadata    all                         default
storage/subvol-102-disk-0  overlay               on                          default
storage/subvol-102-disk-0  encryption            off                         default
storage/subvol-102-disk-0  keylocation           none                        default
storage/subvol-102-disk-0  keyformat             none                        default
storage/subvol-102-disk-0  pbkdf2iters           0                           default
storage/subvol-102-disk-0  special_small_blocks  0                           default

Weirdest thing is that the folder contains the same amount of files as the source, which leads me to believe that this might be a disk usage reporting problem, rather than a physical files on the disk problem

I hope someone has had this before or could lead me in the right direction.

Thanks in advance!
 
it says compress ratio 3.6.
that means you copied very compressable data onto the volume and it simply doesnt use any more data.

dd with /dev/zero as source just generates a ton of zeros in a file which is infinitely compressible.

try copying something incompressible there such as a highly compressed video. or use /dev/random as source for your dd tests.
you should see the used space increase.
9GB x 3.6 = 32.4 GB
if zfs wouldnt use compression your data would actually consume 32.4GB of space.
 
Last edited:
Aaaah, it's dawning on me! Sometimes one needs just a little nudge :-)

Thank you so much for your insights.
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!