[SOLVED] ZFS Dedup: On for Pool, Off for Container Disk?

[Proxygen]

I have a ZFS pool where I put (only) containers which will have mostly identical data (95% or more). I am trying to save some disk space by using ZFS (as each container is 30GB and I will run a couple dozen of them)

This is the current setup:

# zfs get all | egrep 'dedup|checksum|compression'
zfs_containers2                    checksum              on                                  default
zfs_containers2                    compression           lz4                                 local
zfs_containers2                    dedup                 on                                  local
zfs_containers2/subvol-101-disk-0  checksum              on                                  default
zfs_containers2/subvol-101-disk-0  compression           lz4                                 inherited from zfs_containers2
zfs_containers2/subvol-101-disk-0  dedup                 on                                  inherited from zfs_containers2

WITHIN the VM there is very little data to dedup, especially given the HW costs of doing so.

In this situation, is it best practice to turn dedup off for zfs_containers2/subvol-101-disk-0 but leave it on for zfs_containers2 ?

 

[Nemesiz]

No point. It will effect only single fs/volume with no effecting children data`s

 

[LnxBil]

Besides dedup, you can create a master volume from which you clone all other volumes if the OS is very similar and only differs in data. Unfortunately, this will only be valid until you start updating your machines.

 

[Nemesiz]

Another solution is to use single ZFS file system with dedup and use it as directory mount point in Proxmox. Use VM disk as images (raw,qcow2,vmdk)

 

[Proxygen]

Another solution is to use single ZFS file system with dedup and use it as directory mount point in Proxmox. Use VM disk as images (raw,qcow2,vmdk)

In my case they are containers, I don't need full VMs.

 

[Proxygen]

Besides dedup, you can create a master volume from which you clone all other volumes if the OS is very similar and only differs in data. Unfortunately, this will only be valid until you start updating your machines.

Not sure I follow this option, but the OS AND data will be nearly identical.

 

[Proxygen]

No point. It will effect only single fs/volume with no effecting children data`s

So setting dedup=off on zfs_containers2/subvol-101-disk-0 does not effect the disk for ct 101?

 

[LnxBil]

So setting dedup=off on zfs_containers2/subvol-101-disk-0 does not effect the disk for ct 101?

Only after setting it, all previously written blocks are still in it. I have no ballpark number if this improves the read/write time a lot, because every I/O is still routed through the dedup table.

 

[Proxygen]

thanks LnxBil, that's what I thought. i can zfs send/receive or just move it in and out of ZFS.

i'm going to disable for the VM disk and enable for the pool.

 

[Nemesiz]

If you will set dedup for pool it will be inherited for lowers levels of fs/volumes. Except if you set for everyone manually.

 

[Proxygen]

yes. i'm thinking how could i set dedup off for the volume BEFORE i fill it up with actual data...

 

[Nemesiz]

Are you sure you need dedup? It require RAM .... a lot.

 

[Proxygen]

yes, positive. as mentioned, 95% of the data is identical and the data set is over 20GB

 

[LnxBil]

Are you sure you need dedup? It require RAM .... a lot.

Yes, I also thought I have enough RAM, but I hadn't. Pool was totally unusable.

@Gaia: How much RAM do you have?

i can zfs send/receive

Could work, because the data is written again. Be sure that the destination has dedup off. This can be tricky if you create the destination on receive, you need to disable dedup for the pool to have dedup off for newly created filesystems. I don't know if I would go that road and fiddle around with dedup per volume and use send/receive.

or just move it in and out of ZFS.

Just create a new filesystem in ZFS with dedup off, stop the source container, sync all your files with rsync to the new filesystem, rename your old container with suffix old, rename the old to the previous name, start your container and enjoy. Afterwards you should see with zfs list that both filesystems should differ in space usage.

 

[Proxygen]

Yes, I also thought I have enough RAM, but I hadn't. Pool was totally unusable.

How much RAM do you have?

32GB, could add more.

Just create a new filesystem in ZFS with dedup off, stop the source container, sync all your files with rsync to the new filesystem, rename your old container with suffix old, rename the old to the previous name, start your container and enjoy. Afterwards you should see with zfs list that both filesystems should differ in space usage.

duh, that's the way to go!

 

[LnxBil]

32GB, could add more.

I tried with 64 GB and that was not enough for me, but depends on your pool. If your pool is small or not filled up much, you will not have a problem. In small setups, it can work very well, but keep in mind, that your ARC is extremely important if you use dedup. The dedup table should stay in RAM all the time if you do not want to have terrible I/O performance.

 

[Proxygen]

I tried with 64 GB and that was not enough for me, but depends on your pool. If your pool is small or not filled up much, you will not have a problem. In small setups, it can work very well, but keep in mind, that your ARC is extremely important if you use dedup. The dedup table should stay in RAM all the time if you do not want to have terrible I/O performance.

thanks for the tip. i could use L2ARC on NVME SSD too

 

[LnxBil]

i could use L2ARC on NVME SSD too

Yes, that'll work. Counter intuitively, this will in general even worse the performance of your pool, because you will have even less ARC to use in the first place. The only way to get it fast is to throw main memory at it. Yet if your dedup table fits into your fast L2ARC, it's a good start and will improve dedup, but keep in mind, that L2ARC is flushed on restart.