I installed proxmox in a raid1 using 2x4TB NVMEs. About 6 months later, while randomly browsing the proxmox web UI, I stumbled upon a page saying that the drives are at 6% wear. I dont even write to these drives other than whatever logs linux writes in the background by just existing. I only use the drives to install proxmox and the VM OSes, and use other drives like HDDs for things that "write" and for writing media to (jellyfin, immich etc). WTH is going on and how do I stop it? Another thread said it was "ZFS". Ok, so should I backup proxmox and all the VMs and reinstall without using ZFS? Should have a warning when selecting it during proxmox installation.
Proxmox is destroying my NVMEs
- Thread starter manguy
- Start date
I'd recommend you look into what actually causes your writes. Please also share some details about your storage setup and drive models
I don't have the numbers (check yours) but PVE writes quite a bit by itself. Logs, Graphs, pmxcfs, etc. Do you happen to know what the wear was 6 months ago?
Bash:
lsblk -o+FSTYPE,LABEL,MODEL
First of all, 6% isn't that bad, is it?
For half a year. That would mean that it runs for over 8 years.
That is pretty good. Longer than warranty.
What drive do you use? Consumer or prosumer drives or maybe even QLC? Tell us the drive model.
ZFS does not by design cause more wear than ext4.
But there are two small diffrences, that can have an impact.
I don't belong to the "you have to use enterprise SSDs" crowd. But I do belong to the "you can't use trash SSDs" crowd.
For half a year. That would mean that it runs for over 8 years.
That is pretty good. Longer than warranty.
What drive do you use? Consumer or prosumer drives or maybe even QLC? Tell us the drive model.
No.
ZFS does not by design cause more wear than ext4.
But there are two small diffrences, that can have an impact.
1. ZFS is CoW.
Every sync write, will write double (ZIL). But since almost nothing is a sync write, that should not matter.
If you do have many sync writes (DB), you want move ZIL to a SLOG enterprise drive with PLP to speed up performance anyway.
2. ZFS uses by default 16k ZVOL.
So your RAW VM disks also use 16k.
ext4 uses by default 64k QCOW2 disks.
Why should it matter if you write in 16k or 64k chunks?
Well, normally it should not. It does definetly not matter for HDD, but it can for SSDs. Why? Well, while SSDs report to use 4k sectors, the firmware controller does stuff differently behind in NAND. So it could be, that the drive internally uses 64k.
If you want to write a 64k file, ZFS would do 4 times a 16k write, each causing 64k (or 256k total), while the ext4 only does 64k.
So you have a 4 times write amplification.
BTW the other way round can also be true! If you want to do a 4k write, ZFS needs to change 16k. If the SSD does not have controller amplification, that can result in an only 16k write which is only 4x write ampflification.
ext4 qcow2 on the other hand, would write 64k, which is 16x write ampflification.
That is why enterprise drives often not only specify one TBW but two. One TBW for 64k writes, and one TBW for 4k writes. Again, none trash SSDs, don't really suffer from that.
Every sync write, will write double (ZIL). But since almost nothing is a sync write, that should not matter.
If you do have many sync writes (DB), you want move ZIL to a SLOG enterprise drive with PLP to speed up performance anyway.
2. ZFS uses by default 16k ZVOL.
So your RAW VM disks also use 16k.
ext4 uses by default 64k QCOW2 disks.
Why should it matter if you write in 16k or 64k chunks?
Well, normally it should not. It does definetly not matter for HDD, but it can for SSDs. Why? Well, while SSDs report to use 4k sectors, the firmware controller does stuff differently behind in NAND. So it could be, that the drive internally uses 64k.
If you want to write a 64k file, ZFS would do 4 times a 16k write, each causing 64k (or 256k total), while the ext4 only does 64k.
So you have a 4 times write amplification.
BTW the other way round can also be true! If you want to do a 4k write, ZFS needs to change 16k. If the SSD does not have controller amplification, that can result in an only 16k write which is only 4x write ampflification.
ext4 qcow2 on the other hand, would write 64k, which is 16x write ampflification.
That is why enterprise drives often not only specify one TBW but two. One TBW for 64k writes, and one TBW for 4k writes. Again, none trash SSDs, don't really suffer from that.
I don't belong to the "you have to use enterprise SSDs" crowd. But I do belong to the "you can't use trash SSDs" crowd.
Last edited:
Can you tell us please, witch Drive do you use and which ZFS config are your setup based on?
I had quite fast increase of wear out, I was suspecting that logs are the problem. By simply using log2ram the situation has improved a lot. Of course this decision will make troubleshooting more difficult if my host crashes.
Proxmox logs a lot, and when it is amplified by (many small) sync writes due to ZFS (or BTRFS), especially QLC drives can wear fast. Lots of older threads about this with work-around like non-persistent logging, disabling unused services, using LVM(-thin) instead or using drives with much higher write endurance or (enterprise) PLP.
Proxmox is wasteful, as they in effect are catering to enterprise users and an assumption is made enterprise storage is used, so its unlikely these issues will change, examples of waste including the logging, the scratch storage when uploading ISO's, every single update on the screen, graph updates etc. is logged to an access log.
I have two thoughts, first where does this 6% come from, do you have a full SMART stat output? I am assuming at the moment it is write cycles, but it also could be reallocated sector usage.
Second there is ways massively decrease writes. But of course they are unsupported, so I wont give step by step, but merely hints that you can research,
Frontend in front of pveproxy to buffer log writes.
Redirect web access writes to /dev/null.
Both of these likely mitigated with your ramdisk though.
Making drives behave same as eneterprise drives, ZFS can be told to not send flush commands to the storage device, this does not disable ZIL or force only async writes, it just stops forcing the disk to do immediate flushes (same way enterprise drives behave).
Small i/o of 32k or less can also have a duplicated write removed with some tuning.
I have two thoughts, first where does this 6% come from, do you have a full SMART stat output? I am assuming at the moment it is write cycles, but it also could be reallocated sector usage.
Second there is ways massively decrease writes. But of course they are unsupported, so I wont give step by step, but merely hints that you can research,
Frontend in front of pveproxy to buffer log writes.
Redirect web access writes to /dev/null.
Both of these likely mitigated with your ramdisk though.
Making drives behave same as eneterprise drives, ZFS can be told to not send flush commands to the storage device, this does not disable ZIL or force only async writes, it just stops forcing the disk to do immediate flushes (same way enterprise drives behave).
Small i/o of 32k or less can also have a duplicated write removed with some tuning.