ZFS dedup on a Proxmox host in 2026 — has fast dedup changed the recommendation?

I setup ZFS fast dedup on 14x DRAID2 spinning disks pool, with a mirrored Special SSD vdev sometime last year.

No issues except for (1) unfixable metadata permanent error that basically meant I had to create a new just-big-enough RAID0 pool, + copy the data back and forth with rclone to fix it as zfs send would abort. After that, the issue has not reoccurred. Dual-booting proxmox and IIRC Linux Mint
 
On dedup_table_quota — I asked about this in the first post, so here's the measurement.

Test box, not the production pool: PVE 9.2.5, zfs-2.4.3-pve1, loopback pool, compression=off, recordsize=128K. Same 1.25 GiB corpus written twice — once with dedup=on and no quota, once with dedup=on and dedup_table_quota=256K.

no dedup, baseline 1285.8 MiB allocated
dedup=on, no quota 840.9 MiB dedupratio 1.53x DDT 6684 entries, 3.27M on disk
dedup=on, quota 256K 1287.2 MiB dedupratio 1.00x dedup_table_size 1.95M

Two things there are worth reading twice. The table grew to 1.95M against a 256K quota — it's a soft target, not a cap. And the capped run deduplicated nothing at all while still paying for that table, ending up marginally worse than never enabling dedup.

zpool status -D explains it: all 3604 entries sat at refcnt 1. The unique part of the corpus happened to be written first, filled the table, and by the time the duplicate files arrived no new entries could be created — so every duplicate was written as a fresh block.

Writes never errored, so the "degrades gracefully" part holds. But the practical consequence is stronger than the documentation suggests: the Klara PVE guide says new writes simply bypass dedup once the quota is reached, which is accurate about the mechanism and quiet about the outcome. With an undersized quota, whether you get any benefit at all depends on the order your data lands in. Unique data arriving first poisons the table and leaves you with the cost and none of the saving.

Sizing the quota from a zdb -S histogram before enabling anything looks like the only way to avoid that.
 
  • Like
Reactions: Onslow and waltar
" Doing it wrong " and posting "results" doesn't mean much. Nobody in real life is going to try to impose a 256KB quota on dedup, this isn't the 1980s.

32GB RAM on the server. I gave my ~30TB 4TBx14-disk DRAID2 fast-dedup a quota of 16GB + special device mirror. Even with the pool ~70% full it hasn't gotten anywhere near that. IIRC it's only using somewhere under 3GB for dedup table.

Had One-off issue, permanent error in metadata that was probably from dual-booting different ZFS versions. Scrub wasn't fixing it at all. Created new "just big enough" temporary RAID0 pool, copied data over with rclone in parallel, recreated DRAID pool, copied data back. Metadata error gone.
 
  • Like
Reactions: Johannes S
Kingneutron said:
" Doing it wrong " and posting "results" doesn't mean much. Nobody in real life is going to try to impose a 256KB quota on dedup, this isn't the 1980s.

32GB RAM on the server. I gave my ~30TB 4TBx14-disk DRAID2 fast-dedup a quota of 16GB + special device mirror. Even with the pool ~70% full it hasn't gotten anywhere near that. IIRC it's only using somewhere under 3GB for dedup table.

Had One-off issue, permanent error in metadata that was probably from dual-booting different ZFS versions. Scrub wasn't fixing it at all. Created new "just big enough" temporary RAID0 pool, copied data over with rclone in parallel, recreated DRAID pool, copied data back. Metadata error gone.
Click to expand...
Fair on the 256K — nobody would run that, and I didn't suggest anyone should. It was picked to make the table actually fill on a 1.25 GiB corpus. Worth separating the units while we're here, though: dedup_table_quota caps the on-disk DDT size, not RAM — dedup_table_size is what it's measured against. RAM pressure comes later, through ARC caching the thing.

The finding wasn't "256K is too small", though. It was that the quota wasn't enforced: the table reached 1.95M against a 256K cap, 7.6× over. It's a soft limit, so the overshoot scales with how fast you dirty data between transaction groups — at any quota value, 16 GB included. Whether you'd ever notice is a separate question.

Your pool is more interesting to me than my test box, because it would settle something this thread has left open. @alexskysilk put the cost at ~2 GB per deduplicated TB in #20 — at ~21 TB of data that's ~42 GB of table. You're reporting under 3 GB for the same amount of data. That's an order of magnitude apart and one of them has to be wrong.

If you can post zpool status -D and the recordsize of the deduped datasets, that would be the first real fast-dedup sizing figure in this thread — mine came from a corpus small enough to be a toy, and daanw's is from the 0.8–2.0 era. A 1M recordsize would explain the whole gap on its own; if it's 128K, something more interesting is going on.

Unrelated, since it keeps not coming up: I also have the block cloning half measured on 2.4.3 and haven't posted it — BRT sharing does not survive zfs send (the replica re-expands by exactly what was saved), and zfs list shows no change at all after a reflink pass, only zpool list does. That belongs in its own post rather than bolted onto this one.
 
uzumo said:
If you're worried about memory usage or the fact that you can't turn it off, this would probably make you happy when it comes out.

When multiple methods for deduplication emerge, it becomes unclear whether the term “ZFS deduplication” refers to inline or post-processing...

https://github.com/openzfs/zfs/pull/18745
Click to expand...
@uzumo that ambiguity is worth nailing down, because the two things have almost nothing in common operationally.

Inline (dedup=on) decides at write time, costs a DDT in RAM forever, and can't be undone for data already written. Out-of-band runs after the fact, reads what's already on disk, collapses matches through block cloning, and costs nothing once it has finished — but it only sees what it can find by reading, and it can't touch anything that's already been written since.

On the pool I measured, that difference is 1.51 versus 1.44 — about 5%. Which is the whole argument in one number: on a pool that's already 86% full, inline's extra 5% would require rewriting 6.29 TiB first, and there's 1003 GiB free to do it in.
 
  • Like
Reactions: Kingneutron
@IsThisThingOn that commit is the merge of the PR @uzumo linked back in #21. It went in yesterday, 20 August, committed by Behlendorf, closing both the PR and an issue that had been open since 2020.

To answer the question directly: no, dedup doesn't have to be live. That's exactly what this is. FIDEDUPERANGE is the out-of-band path — you find the identical ranges yourself, afterwards, and ask the kernel to collapse them.

What it adds over just calling cp --reflink is the part that matters. A userspace tool that hashes two files and then reflinks them has a window between the hash and the clone; if the source changes inside that window you've written the wrong content over the destination. The merged code takes a read lock on the source and a write lock on the destination, compares, clones, and only then releases — the comment in zfs_vnops.c puts it as the cloned blocks being exactly the bytes that were compared. If they differ, nothing happens and you're told so. No userspace tool can offer that.

One practical caveat, because a commit link makes it look available: it's on master only. The version string there reads 2.4.99, so it's headed for 2.5.0 — no tag, no backport to zfs-2.4-release, no milestone. PVE ships 2.4.x, so there is nothing to use today. ‍

And correcting myself from two posts up: I said out-of-band dedup "costs nothing once it has finished", which is too generous. The BRT table persists on disk for as long as the clones exist, and is cached in ARC while it's in use. Far cheaper than a DDT — it holds only the blocks actually shared, not every block in the pool — but not free.
 
  • Like
Reactions: Kingneutron
You can still use `block_cloning`, but since it’s extremely dangerous without `FIDEDUPERANGE`, I’m just waiting for now.

The program below was developed by Mic92 during the discussion in #18745. It’s designed to work with `FIDEDUPERANGE` on ZFS systems that support it, so I’m hoping that `FIDEDUPERANGE` will eventually be implemented in Proxmox VE’s ZFS as well.

https://github.com/Mic92/zfs-dedup

It's good to see some progress.
 
  • Like
Reactions: DeQuzzie and Johannes S
IsThisThingOn said:
I thought I might share this with you guys. Could be interesting. I personally never really understood why dedup has to be "live".
But to be honest, I don't know much about dedup.

https://github.com/openzfs/zfs/commit/afb4010325a52413c0ad813885eed50738a8380b
Click to expand...
It doesn't, really. Live dedup is just the way it came out as implemented in the original Solaris code.

PBS does after-the-fact dedup on filesystems other than ZFS. It gets run by the garbage-collection job

https://search.brave.com/search?q=pbs+dedup&conversation=09782fd2a4ea3aa8ed402373279d2701b05d

.

Live dedup kinda makes it so you don't have to have excessive amounts of free space; it introduces a short delay in write-out I/O tho, and benefits from more RAM.

Offline dedup is by nature a 2-pass system where the backup data gets written out, and then deduped while doing a 2nd read of the data. Could cause more fragmentation on spinners.

.

https://search.brave.com/search?q=b...ersation=0978736db1a575865dd7958f15a5515ec2d6

https://search.brave.com/search?q=n...ersation=0978e769e6c32bce1495b53457337281598a
 
  • Like
Reactions: Johannes S
DeQuzzie said:
Fair on the 256K — nobody would run that, and I didn't suggest anyone should. It was picked to make the table actually fill on a 1.25 GiB corpus. Worth separating the units while we're here, though: dedup_table_quota caps the on-disk DDT size, not RAM — dedup_table_size is what it's measured against. RAM pressure comes later, through ARC caching the thing.

.

Your pool is more interesting to me than my test box, because it would settle something this thread has left open. @alexskysilk put the cost at ~2 GB per deduplicated TB in #20 — at ~21 TB of data that's ~42 GB of table. You're reporting under 3 GB for the same amount of data. That's an order of magnitude apart and one of them has to be wrong.

If you can post zpool status -D and the recordsize of the deduped datasets, that would be the first real fast-dedup sizing figure in this thread — mine came from a corpus small enough to be a toy, and daanw's is from the 0.8–2.0 era. A 1M recordsize would explain the whole gap on its own; if it's 128K, something more interesting is going on.
Click to expand...
I'll try to post something later tonight or this weekend, after the primetime electric rates go down...
 
  • Like
Reactions: Johannes S
Kingneutron said:
I'll try to post something later tonight or this weekend, after the primetime electric rates go down...
Click to expand...

> zpool status -D and the recordsize of the deduped datasets

zpool list $zp; zfs list $zp


Code: 
NAME       SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP    HEALTH  ALTROOT
zshelf15  47.5T  19.7T  27.8T        -         -     0%    41%  1.85x    ONLINE  -
NAME       USED  AVAIL  REFER  MOUNTPOINT
zshelf15  25.1T  19.6T   285K  /zshelf15

zshelf15/dedupfast                                 23.9T  19.6T  10.9T  /zshelf15/dedupfast
zshelf15/dedupfast/from-beelink                     254G  19.6T   254G  /zshelf15/dedupfast/from-beelink
zshelf15/dedupfast/from-macmini                    2.89T  19.6T  2.89T  /zshelf15/dedupfast/from-macmini
zshelf15/dedupfast/from-qotom-proxmox              9.79T  19.6T  9.79T  /zshelf15/dedupfast/from-qotom-proxmox
zshelf15/dedupfast/notshrcompr-zshelf15            41.7G  19.6T  41.7G  /zshelf15/dedupfast/notshrcompr-zshelf15


I may have been going off an old memory before re-creating the pool. Right now it's 7.76G on disk, 5.39G in core - but when I copied the data back to clear the metadata error I moved more stuff into the dedup datasets.

This is tertiary backup for the house, only gets turned on about once a month or less to update things. 4TB all-SAS disks used from ebay

More info here:

https://pastebin.com/tr7nTscj

Update - the Special mirror dev has ~191G alloc / 62.9G free
 
Last edited:
  • Like
Reactions: DeQuzzie and Johannes S
@Kingneutron thanks for going back and checking — retracting your own number in public is rarer than it should be.

Your figures are the most useful thing in the thread right now, because they answer what I asked in #25 and they match neither estimate that was floating around. 7.76G of table against 19.7T allocated is about 0.4 GB per TB — roughly five times cheaper than the ~2 GB/TB @alexskysilk quoted in #20. A large recordsize is almost certainly what's doing that; if the pastebin has it, that one number settles the disagreement for good.

The thing I'd actually flag is sitting in your own numbers. Table 7.76G, quota 16G, pool at 41% capacity. If the table keeps growing at the rate it has so far, you reach the quota somewhere around 85% pool capacity — before the pool is full, on a pool you're evidently still filling.

What happens then is milder than my test made it look, and the difference is worth spelling out. My 1.00x came from the table filling with unique blocks before any duplicates arrived, so it never held anything useful. Yours is already full of entries earning you 1.85x — hitting the cap won't take that away. New blocks simply stop being added, so the ratio stops improving from wherever it stands. Better to know that before it happens than after. And going by my test the cap is soft — mine ran 7.6× past its quota — so don't be surprised if dedup_table_size drifts past 16G rather than stopping there.

One number for anyone sizing a machine from this thread: it's the in-core figure that matters, not the on-disk one. 5.39G resident for a single pool on a 32GB host.

Separate point, on #30: PBS doesn't dedupe after the fact. It's inline — the client chunks data by content and skips uploading any chunk whose digest is already in the datastore, at backup time. Garbage collection is mark-and-sweep space reclamation: it removes chunks nothing references any more, it never deduplicates. Which makes PBS an argument for the opposite of what it was doing in that post — it's inline dedup, just implemented outside ZFS.
 
You must log in or register to reply here.
Top Bottom
Back