ZFS dedup on a Proxmox host in 2026 — has fast dedup changed the recommendation?

[DeQuzzie]

Hi,

The standing advice for years has been to leave dedup=off on ZFS pools under Proxmox: the DDT lives in RAM, the write path suffers, and you can't turn it back off for data already written. OpenZFS 2.3 ships fast dedup, which is supposed to address most of that — quota'd DDT, pruning, dedup logs.

Has that changed anything on the Proxmox side? Concretely, for a pool that holds both guest disks and a large dataset of ordinary files with a lot of duplicates among them:

is dedup=on still a "don't", or is it now a defensible option for the file dataset only, with the guest disks on a dataset that has it off?
is there any RAM sizing guidance that isn't the old 5 GB per TB rule of thumb?
dedup_table_quota — does anyone have it in production on a PVE host, and what happens to the write path when the table hits the quota?
Everything I can find on this is either pre-2.3 or not Proxmox-specific.

 

[daanw]

From my experience, even the old ZFS dedup was doable as long as you had a nvme special vdev included.
The dedup memory myth is pretty much like the general ZFS memory myth. Adding that special vdev definitely had more impact than adding more RAM.

Actual numbers obviously depend on your workload and the resulting dedup factor whether it is worth the trouble and extra hardware at all.
With Proxmox in mind, if you have hundreds of virtual machines sharing 90% of their blocks and are limited on storage it might be interesting to check it out.

 

[IsThisThingOn]

I thinkg this is en excellent read on why it might could be considered now.

I stil think it is not a good idea. Why? Storage is cheap, performance is not. While it is faster, it still comes with a IMHO extreme performance penalty.

The other thing is that I don't think it would actually save that much.
But still, I think that in real life the dedublication would even be with 1000 VMs not that big.
And also working with ZFS snapshots (cloning a base image VM) will probably yield way better results without any performance implications. But to be fair, that saved storage will go down with usage and updates so it probably also depends on how long your VMs lifteime is.

 

[UdoB]

Your statement...

Storage is cheap, performance is not.

...seems to come from an age more than 10 months ago. Storage is expensive now. And CPU performance on my machines is plenty - all cores are nearly idle with 4 to 14 percent usage in this very moment.

That said... I am definitely not going to enable DDUP on any production machine. But I have a fitting test machine and it is tempting to try it out...

Edit: actual sefttest postponed - one would need to specify an adequate test-setup first and run reproducible tests before and after...

 

[DeQuzzie]

Thanks, both angles are useful.

On the special vdev: that matches what I'd expect for the DDT lookups, but the part I can't find numbers for is the write path — does putting the DDT on NVMe actually remove the penalty, or just move it? If anyone has before/after write latency at a fixed queue depth, that would settle it.

One thing I should have been clearer about in the first post: the duplicates I care about aren't blocks shared between VM images. It's a bulk dataset of ordinary files — archives, media, build outputs — where a good share of the files are byte-identical whole copies of each other. Linked clones from a base image don't help with that, and the dedup factor doesn't decay with guest updates either. So I'd be looking at dedup=on on that one dataset only, guest disks staying on a dataset with it off. Does anyone run it that way?

@UdoB on the test setup — for "would it even save anything", zdb -S <pool> gives the dedup histogram and the ratio you would have got, without enabling anything. It walks every block so it's slow and wants RAM, but it answers the question before you commit to a one-way setting. Beyond that I'd measure write latency with fio at a fixed queue depth before and after, and watch the DDT size against dedup_table_quota — which is the other thing nobody has mentioned yet: what actually happens to writes when the table hits the quota?

 

[Onslow]

FWIW:
"ZFS Fast Dedup for Proxmox VE 9.x: Technical Implementation Guide"
https://klarasystems.com/articles/zfs-fast-dedup-for-proxmox-ve-9x/

 

[IsThisThingOn]

Storage is expensive now. And CPU performance on my machines is plenty

I am not talking about CPU performance but storage performance
I would argue the statement "storage is cheap, performance is not" has become even more true in the last 10 months.
Even though storage has got more expensive.

 

[UdoB]

I am not talking about CPU performance but storage performance

Oh, a classic misunderstanding...

 

[DeQuzzie]

Thanks @Onslow — that one actually answers the quota question from my first post:

"Once the quota is reached, new writes for dedup-enabled datasets bypass deduplication and are written as unique blocks, preventing unbounded DDT growth."

So it degrades to plain writes rather than stalling. Good to have that in writing somewhere.

Two other useful things in there: it drops the old 5 GB/TB rule in favour of "DDT no more than 50% of max ARC, or 85% of the dedup vdev if you have one", and it puts a floor under the whole question — deploy above 3:1, don't bother below 2:1.

What it doesn't have is a single measurement. No benchmark, no before/after, no latency figure, nothing on recordsize or volblocksize, and — @daanw — nothing on whether putting the DDT on a fast special vdev removes the write penalty or merely relocates it. That one is still open and it's the number I'd most like to see.

The design pattern it recommends is the split I asked about: one dataset with dedup=on, general guest storage without it on zstd. So the shape is endorsed — my case just has ordinary files rather than images on the dedup side, and I still haven't heard from anyone actually running it that way.

@UdoB — this is why I keep coming back to zdb -S. It is the "adequate test setup" for the first half of your question: it gives the histogram and the ratio you would have got, on your real data, with nothing enabled and nothing irreversible. It walks every block so it's slow and wants RAM, but if it comes back at 1.2:1 then the second half of the test never needs to happen. Rob Norris does exactly this on his own laptop in the post @IsThisThingOn linked and gets 1.00x.

One gap worth naming: for a Feb 2026 article about saving space on ZFS 2.3, it never mentions block cloning once. Where the duplicates are whole identical files rather than blocks shared across images, cloning gets most of the win with no DDT, no RAM budget and no one-way switch — and by the article's own 2:1/3:1 gating that kind of data is a poor inline-dedup candidate anyway. Different mechanism for a different shape of duplication, and the guide skips it entirely.

 

[waltar]

If doing dedup at all there's an additional special dedup vdev available which should be considered to be configured too.

 

[daanw]

What it doesn't have is a single measurement. No benchmark, no before/after, no latency figure, nothing on recordsize or volblocksize, and — @daanw — nothing on whether putting the DDT on a fast special vdev removes the write penalty or merely relocates it. That one is still open and it's the number I'd most like to see.

My use case was hot storage of archive material with a modest 1.3 dedup ratio. RAIDZ3 pool of 12x10TB SAS disks with 128GB RAM, 1M recordsize/zstd dataset. Real world write speeds of about 750 MB/s on a non-dedup dataset. Dedup had a factor ~7 penalty compared to a non-dedup dataset with the DDT still being loaded from the harddisks, even causing kernel timeouts on large operations. Additionally, the system severely suffered from write throttles down to KB/s which we were able to relieve up to 10-20 MB/s by adding 128GB more RAM. In other words, unworkable.

By adding a mirrored 1TB nvme special vdev the dedup penalty was practically gone, staying close to the non-dedup write speeds, no timeouts and hardly any throttling.

Keep in mind that this was all in the old dedup ZFS 0.8 - 2.0 era.

If doing dedup at all there's an additional special dedup vdev available which should be considered to be configured too.

Only consider a separate Dedup VDEV if you have a specific need to isolate DDT I/O from metadata/small file I/O, or if the Special VDEV is too small to hold both metadata and the DDT.

 

[Onslow]

What it doesn't have is a single measurement. No benchmark, no before/after, no latency figure, nothing on recordsize or volblocksize, and — @daanw — nothing on whether putting the DDT on a fast special vdev removes the write penalty or merely relocates it. That one is still open and it's the number I'd most like to see.

Hi, @DeQuzzie

A measurement (a general one, not particular for Proxmox) is in the article linked by other poster.
Raw performance, throughput, performance delta, dedup ratio, write and read latency - in various workloads:
"Introducing OpenZFS Fast Dedup"
https://klarasystems.com/articles/introducing-openzfs-fast-dedup/

A small section about "Dedup VDEV" (probably not answering your questions ) is in the article
"Pool and VDEV Topology for Proxmox Workloads"
https://klarasystems.com/articles/pool-and-vdev-topology-for-proxmox-workloads/

"Dedup VDEV: Locates the dedup table on a specific device, ensuring it has all of the available IOPS from that device and does not contend with other workloads. The dedup table is memory-hungry and can be redirected to the dedup (or failing that, special) VDEVs, however Klara generally discourages dedup for generic VM stores because it is often not worth the small storage savings, compared to the cost in RAM and CPU unless there is a very strong, proven dedup use-case."

And an article not about deduplication, but may be worth mentioning in Proxmox context:
"Pool and VDEV Topology for Proxmox Workloads"
https://klarasystems.com/articles/pool-and-vdev-topology-for-proxmox-workloads/

I'm not affiliated with that site, just have read a few articles there

 

[IsThisThingOn]

Different mechanism for a different shape of duplication, and the guide skips it entirely.

Actual data is pretty sparse on that topic. Sooo..... if you would fire up a test environment and let us know your results, that would probably very much be welcomed here in this forum

 

[DeQuzzie]

I'm currently working on this. I have a 2*HDD mirror with 8 TB of raw, mixed data.

 

[IsThisThingOn]

Just a wild guess: Mixed data is not that interesting. The result will be so low (1-1,5) that it does not make sense.

 

[waltar]

In my opinion even mixed data makes sense to experiment of as the result would show to others how this feature can or cannot be useful to them.

 

[IsThisThingOn]

I am 99% sure, for mixed data, the result will be a disaster and not worth it (<2)

 

[daanw]

If it is "worth it" depends on a lot of things.

My use case was archiving as much as possible data on single server with all disk slots used.
A 1.3 dedup ratio is not much, but on archiving for example 100TB of data it results in quite a bit of space saving and as a bonus a reduced replication size (zfs send -w) to a second server. "Worth it" in this case was pretty much defined by space saving while keeping operations performant enough to do incremental backups and restores, which succeeded by adding the special vdev to the pool.

Chipping in with @waltar; since deploying (fast) dedup is not that common, sharing any real world experiences is definitely "worth it". Curious to hear about the results.

 

[DeQuzzie]

Results, as promised.

Hardware and data: PVE 9.1.18, kernel 7.0.2-5-pve, zfs-2.4.2-pve1. The pool is a two-way mirror of 7200 rpm drives (Exos ST12000NM0127 + SkyHawk ST8000VX004), 7.27T size / 6.29T allocated, 86% capacity, 3% fragmentation, no special vdev. Compression is on (lz4/on, zstd-3 on one dataset, off on one), recordsize 128K except one dataset at 1M. The content is exactly the "mixed data" I described: archives, media, backups, ISOs. 3,860,159 files, 8.00 TiB apparent size / 6.27 TiB actually allocated.
Number one, block level. zdb -S tank — a DDT simulation, nothing has to be enabled:

dedup = 1.51, compress = 1.07, copies = 1.00
Total 36.8M blocks 4.29T allocated <- 56.0M blocks 6.48T referenced

Number two, file level. Walk every file, group by size, blake2b only for files that have a size twin: 1,016,158 groups of byte-identical whole files, about 1976 GiB of actually allocated space reclaimable, ratio ≈1.44.
On the argument in this thread. @IsThisThingOn was right about the number — 1.51, below two, exactly as predicted. But @daanw is right about how to read it. He measured 1.3 on his archive box, I got 1.51, and on my pool that is roughly 2 TiB of real space against 1003 GiB currently free. It would free up more than the pool has free right now. Arguing about "worth it" in terms of the ratio alone doesn't work, because 1.5 on 10 GiB and 1.5 on 8 TB are different conversations.
What I don't think anyone has posted before: the gap between file-level and block-level was only ~5% (1.44 vs 1.51, in comparable units). Practically that means, for data shaped like this, simply collapsing whole identical files with block cloning gets about 95% of what inline dedup would — with no DDT, no memory budget, and no one-way pool setting. The difference comes from blocks repeated inside different files, not from whole files.
You can see exactly where it comes from in the histogram. The bulk of the saving (1.28 TiB) is refcnt 2 — ordinary pairs of copies, which whole-file matching catches too. The high-refcount tail is what whole-file matching can never reach:
refcnt 64 636 MiB allocated -> 63.0 GiB referenced
refcnt 128 549 MiB -> 89.8 GiB
refcnt 64K 1 MiB -> 76.8 GiB <- one block, 76,800 references

That last line is a single 1 MiB block in the dataset with recordsize 1M, referenced nearly 77,000 times.
A methodology trap I fell into — worth warning others about. My first pass computed "reclaimable" from st_size and gave 2143 GiB, ratio 1.35. That's wrong: reflink frees allocated blocks, not apparent bytes. On this pool the difference is large — 8.00 TiB apparent vs 6.27 TiB allocated, i.e. 21.6% (holes plus compression). What surprised me was the direction of the correction: the duplicate files turned out to be three times denser than average (7.78% vs 21.6% — all the sparseness sits in files that have no twin), so the file-level number didn't drop, it rose from 1.35 to ≈1.44. If you repeat this, account with st_blocks, otherwise you get an answer that's wrong in both directions at once.
Caveats, so nobody takes this for more than it is. The file-level figure in allocated bytes is estimated from the density of size-candidates rather than measured directly — I failed to save the group list, which I'm kicking myself for. The scan ran against a live filesystem, not a frozen snapshot. And zdb -S covers the whole pool including snapshots, while the file walk only covered live files.
For anyone who wants to repeat this — timings and rakes. The metadata census of 3.86M files took 13 minutes. Hashing 3.94 TiB of candidates took 17 hours 15 minutes on the HDD mirror; throughput fell from 265 to 15 MiB/s as it moved from large files to small ones, and the tail bottomed out at 142 IOPS, i.e. pure 7200 rpm seek mechanics. zdb -S took only 40 minutes because the metadata was still warm in ARC after the scan — from cold it will be longer.

 

[alexskysilk]

If the discussion is strictly on a "worth it" basis, its worth noting by what metric. Ram costs more then storage. Fast dedup is less hungry but thats still ~2G per deduplicated TB- although you could substitute a special device for SOME ram usage.