Proxmox is destroying my NVMEs

[J-Rod]

100% agree with the enterprise drive sentiment. I bought a pile of used Intel DC SATA SSDs (mostly S3700, some Dell & IBM branded) over a decade ago and they are all still at 99% or 100% with many hundreds of TBs written. I think they will outlive me!

There is some pain in this area for home lab, as M.2 NVMe enterprise drives are rare, due to the fact that U.2 is sort of the standard. There are some adapters available, but it's just not as easy as SATA (or SAS).

 

[IsThisThingOn]

Enterprise operating system/hypervisors are just begging for enterprise drives.

You guys keep repeating that claim, but that does not make it any more true. And it isn't a source.

My hypervisor with many Linux VMs, that have wastful docker applications inside, and my other Proxmox Server with many Windows Hosts, both suffer less writes than my MacBook Air that I use to mostly to surf on my sofa. And both hypervisors run totally fine with prosumer drives.
At the currenty rate, they will last for 24 years. That is more than enough for me.

That is why I highly disagree with that broad and wrong generalisation. You can say that if you don't want to think about it, just get enterprise drives. I agree with that. But there is nothing inherantly to hypervisors nor ZFS that automatically requires enterprise drives.

 

[6equj5]

At the currenty rate, they will last for 24 years. That is more than enough for me.

So true. The WD Red SN700 here have in 4.7 years 420 TB r/w and that makes a 20% wear out. At least the next 5 years I don't have to worry and by then it's a little unlikely anyway they are not retired.

Another WD Black SN850: ~ 140 TB r/w in 5.3 years and still 0% wear out.

 

[J-Rod]

ZFS defaults generate over double the writes compared to EXT4 and XFS: https://www.youtube.com/watch?v=V7V3kmJDHTA

Also, ZFS mirroring and replication are crucial PVE features, even for home labs.

In total: 8X the writes.

The WD Red and Black SSDs mentioned in the above post have an endurance rating of approximately 0.7 drive writes per day for 5 years. The DC S3700 I mentioned has an endurance of 10 drive writes per day and the PCIe DC P3700 is 17.

Of course every forum participant leverages PVE differently, but supporting consumer/prosumer versus enterprise will likely result in more pain than joy.

 

[6equj5]

... supporting consumer/prosumer versus enterprise will likely result in more pain than joy.

My experience does just not confirm that. And of course do I utilize ZFS. Ok, it's "just" at home.

The DC S3700 you mentioned are SATA with approx 500 MB/s. Obviously a certain bottle neck in today's hardware.

 

[UdoB]

Off-topic: I am erasing an NVMe consumer device in this very moment. "dd" is writing 1MB blocks with a speed of 130 MB/s --> slower than rotating rust :-(

(( Edit: at the end the overall rate was 102 MB/s - on a 1 TB Kingston NVMe... ))

 

[IsThisThingOn]

ZFS defaults generate over double the writes compared to EXT4 and XFS

That is again a broad statement, that is simply wrong.
Yes changes are CoW. Yes there is ZIL for sync writes. Yes there is a 16k blocksize.

Also, ZFS mirroring and replication are crucial PVE features, even for home labs.

Which has nothing to do with ZFS. A ext4 hardware RAID mirror behaves exactly the same (well besides for rebuilds)

Of course every forum participant leverages PVE differently, but supporting consumer/prosumer versus enterprise will likely result in more pain than joy.

SATA instead of NVME would bring me no joy.

Just saw that I even have a 870 EVO in production on the mostly Windows VMs Proxmox. Even that cheap thing will last 4 years. And that that is only the rated TBW. Most likely it will run even longer.
I never run a mirror of the same drives, though, maybe that is why I don't worry about SSDs.

 

[6equj5]

Yes there is ZIL for sync writes.

Sync writes are just another waste of personal life time. And absolutely no joy.

 

[J-Rod]

So what part of ElectronicsWizardry's YouTube video is wrong?

Drive and host redundancy are both very valuable features even for home lab, so mirroring plus replication will be a frequently suggested PVE configuration, leading to more writes overall.

Disabling the ZIL is a terrible idea and is really only necessary when using consumer SSDs.

SATA throughput isn't a meaningful bottleneck in this context, since a typical virtualization host usage pattern is just a lot of random small reads/writes. There are other factors, such as the realistic throughput in/out of the host via the network as well as ZFS compression which can allow throughput to exceed the disk interface speed by a fair margin. I also question the validity of the bottleneck argument in general since many consumer SSDs have a limited amount of full speed capacity (such as 10%), and once that buffer is filled, speeds drop. Furthermore, significant/continuous writes to a consumer SSD will be avoided if you want it to last.

Ultimately, there are directions a PVE user can go that will either greatly increase or greatly reduce their odds of an unfortunate event. Shall we discuss ECC memory next?

 

[IsThisThingOn]

So what part of ElectronicsWizardry's YouTube video is wrong?

I did never said it was wrong. I did not even watch the vid.
I only said your statment

ZFS defaults generate over double the writes compared to EXT4 and XFS

is wrong. Which it is.

Disabling the ZIL is a terrible idea and is really only necessary when using consumer SSDs.

Is is a terrible idea, that is why nobody sane would recommend that.

can go that will either greatly increase or greatly reduce their odds of an unfortunate event.

No. You are again making implications that are not true.

You imply for example with your broad generalisation, that using a brand new 4TB 990 Pro combined with a WD black whatever in a mirror has greater odds of an unfortunate event than using two DC S3700. Which is simply not true. On the contrary. You can't base that risk factor simply on

if (drive == "enterprise") {
    risk = "false";
} else {
    risk = "true";
}

 

[6equj5]

Disabling the ZIL is a terrible idea

You don't need ZIL if you don't do sync writes. And unless you enforce sync writes on you ZFS datasets there are none. Just check it out.

a typical virtualization host usage pattern is just a lot of random small reads/writes.

My experience is, it's not "a lot" but pretty much next to zilch. But even if so, you have the NVMEs (special vdev / special_small_blocks). So no big deal.

 

[IsThisThingOn]

And unless you enforce sync writes on you ZFS datasets there are none.

that isn't true.

 

[UdoB]

You don't need ZIL if you don't do sync writes.

I am not sure if I read this sentence correctly.

There is always a ZIL and it is especially used for async writes. It is in inner part of the ZFS logic and it is not possible to disable it, afaik (edit: that's wrong). The Intent-Log absorbes "normal" writes and data is written to disk every five seconds --> asynchron.

If there is no ZIL in Ram then it is placed on the target device, which usually is... a bad idea. See e.g. https://www.truenas.com/docs/references/zilandslog/

 

[rulezz]

Off-topic: I am erasing an NVMe consumer device in this very moment. "dd" is writing 1MB blocks with a speed of 130 MB/s --> slower than rotating rust :-(

I wonder why you use dd to erase, wouldn't built-in sanitize erase be more efficient?

Back in the days I heard that some HA related PVE components were thrashing drives, but I don't remember exactly which ones. I even saw recommendations to disable them, that makes perfect sense for homelab

 

[leesteken]

Secure erase by the drive controller itself is probably much more efficient. You might need to hot-plug the drive to enable it. As this is not Proxmox specific, various Linux secure erase guides on the internet will apply.

EDIT: The drive does not even need to write much when doing a secure erase. It wipes the administration of used flash memory blocks and trims everything. Much faster and might not wear the drive at all. And definitely not as much as writing a full drive's worth of zeros.

 

[uberarchiver]

So true. The WD Red SN700 here have in 4.7 years 420 TB r/w and that makes a 20% wear out. At least the next 5 years I don't have to worry and by then it's a little unlikely anyway they are not retired.

Another WD Black SN850: ~ 140 TB r/w in 5.3 years and still 0% wear out.

I have to say that I lost a 500G AND a 1000G SN700 in short succession in 2025. Both had a few percent on them. All was great until it wasn't. No idea what happened. I did not even go through warranty as I was done with WD with that. Been running ZFS and they were log devices.

Now I have u.2 drives (refurbished, enterprise) on my tri-mode SAS-controller instead.

YMMV of course. Temps were ok as I am running Noctua Industrials all over my build. And the fan curves are loud.

 

[UdoB]

I wonder why you use dd to erase, wouldn't built-in sanitize erase be more efficient?

Yes, definitely. "dd" just happened to be my first idea - I am old school ;-)

 

[alexskysilk]

I am not sure if I read this sentence correctly.

There is always a ZIL and it is especially used for async writes.

Not to speak for OP, but I believe he meant external ZIL. which only make sense in specific workloads.

Why does this argument keep popping up?

yes, enterprise drives TYPICALLY have more endurance then consumer drives.
No, "homelab" workloads dont generate the type of load that would stress either.

Why do any of you care? you want to overprovision endurance you dont need- go nuts, no one stops you. but whats with the evangelism?

 

[UdoB]

Not to speak for OP, but I believe he meant external ZIL. which only make sense in specific workloads.

Then he should have said so; a SLOG is a different animal ;-)

Why do any of you care?
...
but whats with the evangelism?

Well... when I read about misconceptions I believe it is better to add some information. Of course everybody can use any devices he/she wants - and any filesystems. My only intention is to allow to make an informed decision.

If my statements are recognized as evangelism... I am actually not sure how to react... :-(

 

[alexskysilk]

The statement wasnt aimed at you specifically.

The misconceptions, as you put it, typically stem around generalities- which dont necessarily apply. an 18 wheeler hauls better and more efficiently but only when you have the load for it. if you're just using it to go to the groceries you're expending resources without benefit- it doesnt mean you start explaining how its better without defining the use case.