I installed proxmox in a raid1 using 2x4TB NVMEs. About 6 months later, while randomly browsing the proxmox web UI, I stumbled upon a page saying that the drives are at 6% wear. I dont even write to these drives other than whatever logs linux writes in the background by just existing. I only use the drives to install proxmox and the VM OSes, and use other drives like HDDs for things that "write" and for writing media to (jellyfin, immich etc). WTH is going on and how do I stop it? Another thread said it was "ZFS". Ok, so should I backup proxmox and all the VMs and reinstall without using ZFS? Should have a warning when selecting it during proxmox installation.
Proxmox is destroying my NVMEs
- Thread starter manguy
- Start date
I'd recommend you look into what actually causes your writes. Please also share some details about your storage setup and drive models
I don't have the numbers (check yours) but PVE writes quite a bit by itself. Logs, Graphs, pmxcfs, etc. Do you happen to know what the wear was 6 months ago?
Bash:
lsblk -o+FSTYPE,LABEL,MODEL
First of all, 6% isn't that bad, is it?
For half a year. That would mean that it runs for over 8 years.
That is pretty good. Longer than warranty.
What drive do you use? Consumer or prosumer drives or maybe even QLC? Tell us the drive model.
ZFS does not by design cause more wear than ext4.
But there are two small diffrences, that can have an impact.
I don't belong to the "you have to use enterprise SSDs" crowd. But I do belong to the "you can't use trash SSDs" crowd.
For half a year. That would mean that it runs for over 8 years.
That is pretty good. Longer than warranty.
What drive do you use? Consumer or prosumer drives or maybe even QLC? Tell us the drive model.
No.
ZFS does not by design cause more wear than ext4.
But there are two small diffrences, that can have an impact.
1. ZFS is CoW.
Every sync write, will write double (ZIL). But since almost nothing is a sync write, that should not matter.
If you do have many sync writes (DB), you want move ZIL to a SLOG enterprise drive with PLP to speed up performance anyway.
2. ZFS uses by default 16k ZVOL.
So your RAW VM disks also use 16k.
ext4 uses by default 64k QCOW2 disks.
Why should it matter if you write in 16k or 64k chunks?
Well, normally it should not. It does definetly not matter for HDD, but it can for SSDs. Why? Well, while SSDs report to use 4k sectors, the firmware controller does stuff differently behind in NAND. So it could be, that the drive internally uses 64k.
If you want to write a 64k file, ZFS would do 4 times a 16k write, each causing 64k (or 256k total), while the ext4 only does 64k.
So you have a 4 times write amplification.
BTW the other way round can also be true! If you want to do a 4k write, ZFS needs to change 16k. If the SSD does not have controller amplification, that can result in an only 16k write which is only 4x write ampflification.
ext4 qcow2 on the other hand, would write 64k, which is 16x write ampflification.
That is why enterprise drives often not only specify one TBW but two. One TBW for 64k writes, and one TBW for 4k writes. Again, none trash SSDs, don't really suffer from that.
Every sync write, will write double (ZIL). But since almost nothing is a sync write, that should not matter.
If you do have many sync writes (DB), you want move ZIL to a SLOG enterprise drive with PLP to speed up performance anyway.
2. ZFS uses by default 16k ZVOL.
So your RAW VM disks also use 16k.
ext4 uses by default 64k QCOW2 disks.
Why should it matter if you write in 16k or 64k chunks?
Well, normally it should not. It does definetly not matter for HDD, but it can for SSDs. Why? Well, while SSDs report to use 4k sectors, the firmware controller does stuff differently behind in NAND. So it could be, that the drive internally uses 64k.
If you want to write a 64k file, ZFS would do 4 times a 16k write, each causing 64k (or 256k total), while the ext4 only does 64k.
So you have a 4 times write amplification.
BTW the other way round can also be true! If you want to do a 4k write, ZFS needs to change 16k. If the SSD does not have controller amplification, that can result in an only 16k write which is only 4x write ampflification.
ext4 qcow2 on the other hand, would write 64k, which is 16x write ampflification.
That is why enterprise drives often not only specify one TBW but two. One TBW for 64k writes, and one TBW for 4k writes. Again, none trash SSDs, don't really suffer from that.
I don't belong to the "you have to use enterprise SSDs" crowd. But I do belong to the "you can't use trash SSDs" crowd.
Last edited:
Can you tell us please, witch Drive do you use and which ZFS config are your setup based on?