ZFS dedup on a Proxmox host in 2026 — has fast dedup changed the recommendation?

I setup ZFS fast dedup on 14x DRAID2 spinning disks pool, with a mirrored Special SSD vdev sometime last year.

No issues except for (1) unfixable metadata permanent error that basically meant I had to create a new just-big-enough RAID0 pool, + copy the data back and forth with rclone to fix it as zfs send would abort. After that, the issue has not reoccurred. Dual-booting proxmox and IIRC Linux Mint
 
On dedup_table_quota — I asked about this in the first post, so here's the measurement.

Test box, not the production pool: PVE 9.2.5, zfs-2.4.3-pve1, loopback pool, compression=off, recordsize=128K. Same 1.25 GiB corpus written twice — once with dedup=on and no quota, once with dedup=on and dedup_table_quota=256K.

no dedup, baseline 1285.8 MiB allocated
dedup=on, no quota 840.9 MiB dedupratio 1.53x DDT 6684 entries, 3.27M on disk
dedup=on, quota 256K 1287.2 MiB dedupratio 1.00x dedup_table_size 1.95M

Two things there are worth reading twice. The table grew to 1.95M against a 256K quota — it's a soft target, not a cap. And the capped run deduplicated nothing at all while still paying for that table, ending up marginally worse than never enabling dedup.

zpool status -D explains it: all 3604 entries sat at refcnt 1. The unique part of the corpus happened to be written first, filled the table, and by the time the duplicate files arrived no new entries could be created — so every duplicate was written as a fresh block.

Writes never errored, so the "degrades gracefully" part holds. But the practical consequence is stronger than the documentation suggests: the Klara PVE guide says new writes simply bypass dedup once the quota is reached, which is accurate about the mechanism and quiet about the outcome. With an undersized quota, whether you get any benefit at all depends on the order your data lands in. Unique data arriving first poisons the table and leaves you with the cost and none of the saving.

Sizing the quota from a zdb -S histogram before enabling anything looks like the only way to avoid that.
 
  • Like
Reactions: Onslow and waltar
" Doing it wrong " and posting "results" doesn't mean much. Nobody in real life is going to try to impose a 256KB quota on dedup, this isn't the 1980s.

32GB RAM on the server. I gave my ~30TB 4TBx14-disk DRAID2 fast-dedup a quota of 16GB + special device mirror. Even with the pool ~70% full it hasn't gotten anywhere near that. IIRC it's only using somewhere under 3GB for dedup table.

Had One-off issue, permanent error in metadata that was probably from dual-booting different ZFS versions. Scrub wasn't fixing it at all. Created new "just big enough" temporary RAID0 pool, copied data over with rclone in parallel, recreated DRAID pool, copied data back. Metadata error gone.
 
  • Like
Reactions: Johannes S
Kingneutron said:
" Doing it wrong " and posting "results" doesn't mean much. Nobody in real life is going to try to impose a 256KB quota on dedup, this isn't the 1980s.

32GB RAM on the server. I gave my ~30TB 4TBx14-disk DRAID2 fast-dedup a quota of 16GB + special device mirror. Even with the pool ~70% full it hasn't gotten anywhere near that. IIRC it's only using somewhere under 3GB for dedup table.

Had One-off issue, permanent error in metadata that was probably from dual-booting different ZFS versions. Scrub wasn't fixing it at all. Created new "just big enough" temporary RAID0 pool, copied data over with rclone in parallel, recreated DRAID pool, copied data back. Metadata error gone.
Click to expand...
Fair on the 256K — nobody would run that, and I didn't suggest anyone should. It was picked to make the table actually fill on a 1.25 GiB corpus. Worth separating the units while we're here, though: dedup_table_quota caps the on-disk DDT size, not RAM — dedup_table_size is what it's measured against. RAM pressure comes later, through ARC caching the thing.

The finding wasn't "256K is too small", though. It was that the quota wasn't enforced: the table reached 1.95M against a 256K cap, 7.6× over. It's a soft limit, so the overshoot scales with how fast you dirty data between transaction groups — at any quota value, 16 GB included. Whether you'd ever notice is a separate question.

Your pool is more interesting to me than my test box, because it would settle something this thread has left open. @alexskysilk put the cost at ~2 GB per deduplicated TB in #20 — at ~21 TB of data that's ~42 GB of table. You're reporting under 3 GB for the same amount of data. That's an order of magnitude apart and one of them has to be wrong.

If you can post zpool status -D and the recordsize of the deduped datasets, that would be the first real fast-dedup sizing figure in this thread — mine came from a corpus small enough to be a toy, and daanw's is from the 0.8–2.0 era. A 1M recordsize would explain the whole gap on its own; if it's 128K, something more interesting is going on.

Unrelated, since it keeps not coming up: I also have the block cloning half measured on 2.4.3 and haven't posted it — BRT sharing does not survive zfs send (the replica re-expands by exactly what was saved), and zfs list shows no change at all after a reflink pass, only zpool list does. That belongs in its own post rather than bolted onto this one.
 
uzumo said:
If you're worried about memory usage or the fact that you can't turn it off, this would probably make you happy when it comes out.

When multiple methods for deduplication emerge, it becomes unclear whether the term “ZFS deduplication” refers to inline or post-processing...

https://github.com/openzfs/zfs/pull/18745
Click to expand...
@uzumo that ambiguity is worth nailing down, because the two things have almost nothing in common operationally.

Inline (dedup=on) decides at write time, costs a DDT in RAM forever, and can't be undone for data already written. Out-of-band runs after the fact, reads what's already on disk, collapses matches through block cloning, and costs nothing once it has finished — but it only sees what it can find by reading, and it can't touch anything that's already been written since.

On the pool I measured, that difference is 1.51 versus 1.44 — about 5%. Which is the whole argument in one number: on a pool that's already 86% full, inline's extra 5% would require rewriting 6.29 TiB first, and there's 1003 GiB free to do it in.
 
@IsThisThingOn that commit is the merge of the PR @uzumo linked back in #21. It went in yesterday, 20 August, committed by Behlendorf, closing both the PR and an issue that had been open since 2020.

To answer the question directly: no, dedup doesn't have to be live. That's exactly what this is. FIDEDUPERANGE is the out-of-band path — you find the identical ranges yourself, afterwards, and ask the kernel to collapse them.

What it adds over just calling cp --reflink is the part that matters. A userspace tool that hashes two files and then reflinks them has a window between the hash and the clone; if the source changes inside that window you've written the wrong content over the destination. The merged code takes a read lock on the source and a write lock on the destination, compares, clones, and only then releases — the comment in zfs_vnops.c puts it as the cloned blocks being exactly the bytes that were compared. If they differ, nothing happens and you're told so. No userspace tool can offer that.

One practical caveat, because a commit link makes it look available: it's on master only. The version string there reads 2.4.99, so it's headed for 2.5.0 — no tag, no backport to zfs-2.4-release, no milestone. PVE ships 2.4.x, so there is nothing to use today. ‍

And correcting myself from two posts up: I said out-of-band dedup "costs nothing once it has finished", which is too generous. The BRT table persists on disk for as long as the clones exist, and is cached in ARC while it's in use. Far cheaper than a DDT — it holds only the blocks actually shared, not every block in the pool — but not free.
 
You must log in or register to reply here.
Top Bottom
Back