sparse and compression

[Miguel]

Dear all,


Is it advisable to use sparse on ZFS pools performance wise? And
compression? Which kind of compression?

Can I change a zpool to sparse on the fly or do I need to turn off all
VMs before doing so?

Why a virtual disk shows as 60G when originally It was 36 Gb in raw format?

NAME USED AVAIL REFER MOUNTPOINT
rpool/data/vm-102-disk-1 60.0G 51.3G 20.9G -

Thanks!

Miguel

 

[brwainer]

"Refer" is the amount that the current item actually uses. "Used" includes all space taken up by snapshots for that volume. So you have 20.9GB for the VM at present, plus 39.1GB from snapshots.
Use "zfs get compressratio -t volume" to see compression ratio, and "zfs list -t all" to see your snapshots. There is a lot more available via the list and get subcommands.

 

[Miguel]

Thanks for answering!

NAME

NAME                       USED  AVAIL  REFER  MOUNTPOINT
rpool                      194G  75.6G   104K  /rpool
rpool/ROOT                1.56G  75.6G    96K  /rpool/ROOT
rpool/ROOT/pve-1          1.56G  75.6G  1.56G  /
rpool/data                 188G  75.6G    96K  /rpool/data
rpool/data/vm-100-disk-1  99.1G  75.6G  99.1G  -
rpool/data/vm-102-disk-1  37.1G  91.3G  21.4G  -
rpool/data/vm-102-disk-2  51.6G  97.9G  29.3G  -
rpool/swap                4.25G  78.6G  1.25G  -

I have found that I don´t have sparse enabled but I do have compression. vm-100-disk-1 refers to the actual data used (but the disk was 134 Gb). However vm-102 disks used data as you can see is different than refer data. I have used dd if=/dev/zero on both VMs to clean up unused space but I don´t see any result on vm-102. Both VMs have same settings scsi driver.

 

[brwainer]

When you "erase" all the unused space with zeroes, that only affects the current/live version of the volume. The old bits that were overwritten are still going to be kept in your snapshots - otherwise, how would you recover that data?. Check your snapshots with a command like "zfs list -t snapshot -r rpool/data/vm-102-disk-1". For snapshots, the "Used" and "Refer" are a bit different than for volumes. "Used" is the amount used by that individual snapshot, while "Refer" is the amount of space that the volume took up at that time, and would take up if you restored this snapshot. Data that is "used" by more than one snapshot doesn't show up in the "Used" amount for either snapshot, but is included in the amount Used by the volume overall.

 

[Miguel]

When I run that command I get No datasets available

 

[brwainer]

how about just "zfs list -t snapshot"?

 

[Miguel]

I tried that too, same thing.

 

[brwainer]

How about “zfs list -t all -o space” and “zfs get compressratio”? The first should help show exactly where that space is being used, the latter is a fairly obvious command.

 

[Miguel]

root@server:~# zfs list -t all -o space
NAME                      AVAIL   USED  USEDSNAP  USEDDS  USEDREFRESERV  USEDCHILD
rpool                     72.4G   197G        0B    104K             0B       197G
rpool/ROOT                72.4G  1.56G        0B     96K             0B      1.56G
rpool/ROOT/pve-1          72.4G  1.56G        0B   1.56G             0B         0B
rpool/data                72.4G   191G        0B     96K             0B       191G
rpool/data/vm-100-disk-1  72.4G   102G        0B    102G             0B         0B
rpool/data/vm-102-disk-1  88.0G  37.1G        0B   21.5G          15.6G         0B
rpool/data/vm-102-disk-2  94.6G  51.6G        0B   29.5G          22.1G         0B
rpool/swap                75.5G  4.25G        0B   1.25G          3.00G         0B

root@server:~# zfs get compressratio
NAME                      PROPERTY       VALUE  SOURCE
rpool                     compressratio  1.20x  -
rpool/ROOT                compressratio  1.65x  -
rpool/ROOT/pve-1          compressratio  1.65x  -
rpool/data                compressratio  1.20x  -
rpool/data/vm-100-disk-1  compressratio  1.19x  -
rpool/data/vm-102-disk-1  compressratio  1.32x  -
rpool/data/vm-102-disk-2  compressratio  1.14x  -
rpool/swap                compressratio  1.02x  -

Thanks for following this up!

 

[brwainer]

Ah here we go. See the values in “USEDREFRESERV” column? That indicates that you have created minimum size reservations on those volumes, and they haven’t used all of their reserved space yet. This document should help https://docs.oracle.com/cd/E19253-01/819-5461/gazvb/index.html - in particular, try the command “zfs get reservation,refreserve -r rpool/data”

 

[Miguel]

If you meant the property reservation, here we go:

root@server:~# zfs get reservation,refreservation -r rpool/data
NAME                      PROPERTY        VALUE      SOURCE
rpool/data                reservation     none       default
rpool/data                refreservation  none       default
rpool/data/vm-100-disk-1  reservation     none       default
rpool/data/vm-100-disk-1  refreservation  none       default
rpool/data/vm-102-disk-1  reservation     none       default
rpool/data/vm-102-disk-1  refreservation  37.1G      local
rpool/data/vm-102-disk-2  reservation     none       default
rpool/data/vm-102-disk-2  refreservation  51.6G      local

 

[Miguel]

Why those different values? I didn´t do anything different when I imported the raw disks

 

[brwainer]

Ok right, the names of properties can vary a bit between ZFS implementations (most of the documentation that exists is for the original made by Sun, as opposed to OpenZFS and ZFSonLinux). So, notice how your refreservation on two of those volumes is the same as the amount they are “using”? That’s your problem. If you want to change that, run “zfs set refreservation=(x)G (volume)” or “zfs set refreservation=none (volume)”. As for how they got there, what system did you import these from? How did you import them? They are either from the source (most likely) or were created by the import process (I think this is unlikely)

 

[Miguel]

Both two VMs come from the same proxmox server. If I set to zfs set refreservation=none it will decrease the space used or there isn´t a way back?

 

[brwainer]

Setting refreservation=none will remove it, and should see the Used size decrease to equal to the Refer. As to how you ended up with these, I have no clue.

 

[brwainer]

I just realized - referservation is how things are handled when you don’t use sparse/thin-provision on ZFS. So when you said before that you didn’t have sparse enabled, that is why. The setting really only affects when a new virtual disk is crreated, and unlike other storage systems, ZFS lets you undo it.

 

[Miguel]

Many thanks! You have been very helpful!

root@server:~# zfs get reservation,refreservation -r rpool/data
NAME                      PROPERTY        VALUE      SOURCE
rpool/data                reservation     none       default
rpool/data                refreservation  none       default
rpool/data/vm-100-disk-1  reservation     none       default
rpool/data/vm-100-disk-1  refreservation  none       default
rpool/data/vm-102-disk-1  reservation     none       default
rpool/data/vm-102-disk-1  refreservation  37.1G      local
rpool/data/vm-102-disk-2  reservation     none       default
rpool/data/vm-102-disk-2  refreservation  51.6G      local
root@server:~# zfs set refreservation=none rpool/data/vm-102-disk-1
root@server:~# zfs set refreservation=none rpool/data/vm-102-disk-2
root@server:~# zfs get reservation,refreservation -r rpool/data
NAME                      PROPERTY        VALUE      SOURCE
rpool/data                reservation     none       default
rpool/data                refreservation  none       default
rpool/data/vm-100-disk-1  reservation     none       default
rpool/data/vm-100-disk-1  refreservation  none       default
rpool/data/vm-102-disk-1  reservation     none       default
rpool/data/vm-102-disk-1  refreservation  none       local
rpool/data/vm-102-disk-2  reservation     none       default
rpool/data/vm-102-disk-2  refreservation  none       local
root@server:~# df -h
Filesystem        Size  Used Avail Use% Mounted on
udev               16G     0   16G   0% /dev
tmpfs             3.2G  318M  2.9G  10% /run
rpool/ROOT/pve-1  112G  1.6G  111G   2% /
tmpfs              16G   37M   16G   1% /dev/shm
tmpfs             5.0M     0  5.0M   0% /run/lock
tmpfs              16G     0   16G   0% /sys/fs/cgroup
rpool             111G  128K  111G   1% /rpool
rpool/ROOT        111G  128K  111G   1% /rpool/ROOT
rpool/data        111G  128K  111G   1% /rpool/data
/dev/fuse          30M   16K   30M   1% /etc/pve