Why my ZFS dataset is consuming much more disk space?

D

decibel83

Renowned Member
Oct 15, 2008
210
1
83
Hi,
my virtual machine has two ZFS drives, one 60 Gb and one 18 Tb:

Code: 
root@gus3:~# grep zfs /etc/pve/qemu-server/3001.conf
virtio0: local-zfs:vm-3001-disk-0,size=60G
virtio1: local-zfs:vm-3001-disk-1,size=18T

I have 4 10 Tb disks on this server and I'm running out of disk space, so I'm understanding why:

Code: 
root@gus3:~# zfs list -o space
NAME                       AVAIL   USED  USEDSNAP  USEDDS  USEDREFRESERV  USEDCHILD
rpool                      83.8G  25.4T        0B    140K             0B      25.4T
rpool/ROOT                 83.8G  3.09G        0B    140K             0B      3.09G
rpool/ROOT/pve-1           83.8G  3.09G        0B   3.09G             0B         0B
rpool/data                 83.8G  25.4T        0B    140K             0B      25.4T
rpool/data/vm-3001-disk-0  83.8G  55.8G      806M   55.0G             0B         0B
rpool/data/vm-3001-disk-1  83.8G  25.4T      312G   25.1T             0B         0B
rpool/swap                 87.0G  16.5G        0B   13.3G          3.19G         0B

Why the rpool/data/vm-3001-disk-1 dataset is taking 25,4 Tb instead of 18 Tb?
 
LnxBil said:
For starters: Metadata and Snapshots and Compression.
Click to expand...

7 Tb (about 40%!) of metadata??

Could you please post the output of:

Code: 
zfs list -o space,lused,refer,lrefer,compressratio
Click to expand...

Code: 
root@gus3:~# zfs list -o space,lused,refer,lrefer,compressratio
NAME                       AVAIL   USED  USEDSNAP  USEDDS  USEDREFRESERV  USEDCHILD  LUSED     REFER  LREFER  RATIO
rpool                       214G  25.3T        0B    140K             0B      25.3T  18.0T      140K     40K  1.03x
rpool/ROOT                  214G  12.5G        0B    140K             0B      12.5G  14.4G      140K     40K  1.17x
rpool/ROOT/pve-1            214G  12.5G        0B   12.5G             0B         0B  14.4G     12.5G   14.4G  1.17x
rpool/data                  214G  25.1T        0B    140K             0B      25.1T  17.8T      140K     40K  1.03x
rpool/data/vm-3001-disk-0   214G  55.0G        0B   55.0G             0B         0B  57.9G     55.0G   57.9G  1.53x
rpool/data/vm-3001-disk-1   214G  25.1T        0B   25.1T             0B         0B  17.8T     25.1T   17.8T  1.03x
rpool/swap                  217G  16.5G        0B   13.3G          3.19G         0B  13.4G     13.3G   13.4G  1.47x

and zpool status -v.
Click to expand...

Code: 
root@gus3:~# zpool status -v


  pool: rpool


 state: DEGRADED


status: One or more devices are faulted in response to persistent errors.


        Sufficient replicas exist for the pool to continue functioning in a


        degraded state.


action: Replace the faulted device, or use 'zpool clear' to mark the device


        repaired.


  scan: scrub repaired 0B in 8 days 08:05:57 with 0 errors on Mon Mar 16 08:29:58 2020


config:





        NAME        STATE     READ WRITE CKSUM


        rpool       DEGRADED     0     0     0


          raidz1-0  DEGRADED     0     0     0


            sda3    FAULTED      0     0    71  too many errors


            sdb3    ONLINE       0     0     0


            sdc3    ONLINE       0     0     0


            sdd3    ONLINE       0     0     0





errors: No known data errors

I know, my pool is degraded: I'm moving it to a new server.
 
decibel83 said:
Code: 
NAME                       AVAIL   USED  USEDSNAP  USEDDS  USEDREFRESERV  USEDCHILD  LUSED     REFER  LREFER  RATIO
[...]
rpool/data/vm-3001-disk-0   214G  55.0G        0B   55.0G             0B         0B  57.9G     55.0G   57.9G  1.53x
rpool/data/vm-3001-disk-1   214G  25.1T        0B   25.1T             0B         0B  17.8T     25.1T   17.8T  1.03x
Click to expand...

Usually, the lrefer property is bigger than refer due to compression and copies, so could you please also post

Code: 
zfs get all rpool/data/vm-3001-disk-1| sort
 
LnxBil said:
Usually, the lrefer property is bigger than refer due to compression and copies, so could you please also post
Click to expand...

Code: 
root@gus3:~# zfs get all rpool/data/vm-3001-disk-1| sort
NAME                       PROPERTY              VALUE                  SOURCE
rpool/data/vm-3001-disk-1  available             376G                   -
rpool/data/vm-3001-disk-1  checksum              on                     default
rpool/data/vm-3001-disk-1  compression           on                     inherited from rpool
rpool/data/vm-3001-disk-1  compressratio         1.03x                  -
rpool/data/vm-3001-disk-1  context               none                   default
rpool/data/vm-3001-disk-1  copies                1                      default
rpool/data/vm-3001-disk-1  createtxg             580                    -
rpool/data/vm-3001-disk-1  creation              Sun Jan  6 12:41 2019  -
rpool/data/vm-3001-disk-1  dedup                 off                    default
rpool/data/vm-3001-disk-1  defcontext            none                   default
rpool/data/vm-3001-disk-1  encryption            off                    default
rpool/data/vm-3001-disk-1  fscontext             none                   default
rpool/data/vm-3001-disk-1  guid                  17071553500721009767   -
rpool/data/vm-3001-disk-1  keyformat             none                   default
rpool/data/vm-3001-disk-1  keylocation           none                   default
rpool/data/vm-3001-disk-1  logbias               latency                default
rpool/data/vm-3001-disk-1  logicalreferenced     17.8T                  -
rpool/data/vm-3001-disk-1  logicalused           17.8T                  -
rpool/data/vm-3001-disk-1  mlslabel              none                   default
rpool/data/vm-3001-disk-1  objsetid              147                    -
rpool/data/vm-3001-disk-1  pbkdf2iters           0                      default
rpool/data/vm-3001-disk-1  primarycache          all                    default
rpool/data/vm-3001-disk-1  readonly              off                    default
rpool/data/vm-3001-disk-1  redundant_metadata    all                    default
rpool/data/vm-3001-disk-1  refcompressratio      1.03x                  -
rpool/data/vm-3001-disk-1  referenced            25.1T                  -
rpool/data/vm-3001-disk-1  refreservation        none                   default
rpool/data/vm-3001-disk-1  reservation           none                   default
rpool/data/vm-3001-disk-1  rootcontext           none                   default
rpool/data/vm-3001-disk-1  secondarycache        all                    default
rpool/data/vm-3001-disk-1  snapdev               hidden                 default
rpool/data/vm-3001-disk-1  snapshot_count        none                   default
rpool/data/vm-3001-disk-1  snapshot_limit        none                   default
rpool/data/vm-3001-disk-1  sync                  standard               inherited from rpool
rpool/data/vm-3001-disk-1  type                  volume                 -
rpool/data/vm-3001-disk-1  used                  25.1T                  -
rpool/data/vm-3001-disk-1  usedbychildren        0B                     -
rpool/data/vm-3001-disk-1  usedbydataset         25.1T                  -
rpool/data/vm-3001-disk-1  usedbyrefreservation  0B                     -
rpool/data/vm-3001-disk-1  usedbysnapshots       0B                     -
rpool/data/vm-3001-disk-1  volblocksize          8K                     default
rpool/data/vm-3001-disk-1  volmode               default                default
rpool/data/vm-3001-disk-1  volsize               18T                    local
rpool/data/vm-3001-disk-1  written               25.1T                  -
 
yes, such an overhead is not unheard of when using zvols with small volblocksize and raidz. you need to write parity information for each small data block after all.
 
fabian said:
yes, such an overhead is not unheard of when using zvols with small volblocksize and raidz. you need to write parity information for each small data block after all.
Click to expand...

Thanks Fabian!
I cannot understand how I can solve this problem: now I'm creating a new virtual machine on another server to solve the disk problem and I don't want to replicate this problem.
Could you send me some hints on what I have to do to avoid this on the new server?
I've just created a new disk from the Proxmox GUI, and that's all.
Thanks!
 
possible solutions are:
- don't use raidz, but mirrors
- use a bigger volblocksize (but please read up on the downsides!) and tune your VMs accordingly
- use a slightly bigger volblocksize, live with a medium overhead
 
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom
Back