I installed proxmox in a raid1 using 2x4TB NVMEs. About 6 months later, while randomly browsing the proxmox web UI, I stumbled upon a page saying that the drives are at 6% wear. I dont even write to these drives other than whatever logs linux writes in the background by just existing. I only use the drives to install proxmox and the VM OSes, and use other drives like HDDs for things that "write" and for writing media to (jellyfin, immich etc). WTH is going on and how do I stop it? Another thread said it was "ZFS". Ok, so should I backup proxmox and all the VMs and reinstall without using ZFS? Should have a warning when selecting it during proxmox installation.
Proxmox is destroying my NVMEs
- Thread starter manguy
- Start date
I'd recommend you look into what actually causes your writes. Please also share some details about your storage setup and drive models
I don't have the numbers (check yours) but PVE writes quite a bit by itself. Logs, Graphs, pmxcfs, etc. Do you happen to know what the wear was 6 months ago?
Bash:
lsblk -o+FSTYPE,LABEL,MODELFirst of all 6% isn't that bad, is it? For half a year. That would mean that it runs for over 8 years.
That is pretty good.
What drive do you use? Consumer or prosumer drives or maybe even QLC? Please tell us your drives.
1. ZFS is CoW. Every sync write, will write double. But since almost nothing is a sync write, that should not matter. If you do have many sync writes (DB), you want SLOG enterprise drive with PLP to speed up performance anyway.
2. ZFS uses by default 16k ZVOL where you put your VM RAW disks. ext4 uses by default 64k QCOW2 disks.
Why should it matter if you write in 16k or 64k chunks? Well, normally it should not. It does definetly not matter for HDD, but it can for SSDs. Why? Well, while SSDs report to use 4k sectors, the firmware controller does stuff differently behind in NAND. So it could be, that the drive internally uses 64k. A ZFS 16k write would than cause massive write amplification. That is why enterprise drives often not only specify one TBW but two. One TBW for 64k writes, and one TBW for 4k writes. Again, none trash SSDs, don't really suffer from that.
I don't belong to the "you have to use enterprise SSDs" crowd. But I do belong to the "you can't use trash SSDs" crowd.
That is pretty good.
What drive do you use? Consumer or prosumer drives or maybe even QLC? Please tell us your drives.
No. ZFS does not inherantly cause more wear than ext4. But there are two small diffrences, that can have an impact.
1. ZFS is CoW. Every sync write, will write double. But since almost nothing is a sync write, that should not matter. If you do have many sync writes (DB), you want SLOG enterprise drive with PLP to speed up performance anyway.
2. ZFS uses by default 16k ZVOL where you put your VM RAW disks. ext4 uses by default 64k QCOW2 disks.
Why should it matter if you write in 16k or 64k chunks? Well, normally it should not. It does definetly not matter for HDD, but it can for SSDs. Why? Well, while SSDs report to use 4k sectors, the firmware controller does stuff differently behind in NAND. So it could be, that the drive internally uses 64k. A ZFS 16k write would than cause massive write amplification. That is why enterprise drives often not only specify one TBW but two. One TBW for 64k writes, and one TBW for 4k writes. Again, none trash SSDs, don't really suffer from that.
I don't belong to the "you have to use enterprise SSDs" crowd. But I do belong to the "you can't use trash SSDs" crowd.