I installed proxmox in a raid1 using 2x4TB NVMEs. About 6 months later, while randomly browsing the proxmox web UI, I stumbled upon a page saying that the drives are at 6% wear. I dont even write to these drives other than whatever logs linux writes in the background by just existing. I only use the drives to install proxmox and the VM OSes, and use other drives like HDDs for things that "write" and for writing media to (jellyfin, immich etc). WTH is going on and how do I stop it? Another thread said it was "ZFS". Ok, so should I backup proxmox and all the VMs and reinstall without using ZFS? Should have a warning when selecting it during proxmox installation.
Proxmox is destroying my NVMEs
- Thread starter manguy
- Start date
I'd recommend you look into what actually causes your writes. Please also share some details about your storage setup and drive models
I don't have the numbers (check yours) but PVE writes quite a bit by itself. Logs, Graphs, pmxcfs, etc. Do you happen to know what the wear was 6 months ago?
Bash:
lsblk -o+FSTYPE,LABEL,MODEL
First of all, 6% isn't that bad, is it?
For half a year. That would mean that it runs for over 8 years.
That is pretty good. Longer than warranty.
What drive do you use? Consumer or prosumer drives or maybe even QLC? Tell us the drive model.
ZFS does not by design cause more wear than ext4.
But there are two small diffrences, that can have an impact.
I don't belong to the "you have to use enterprise SSDs" crowd. But I do belong to the "you can't use trash SSDs" crowd.
For half a year. That would mean that it runs for over 8 years.
That is pretty good. Longer than warranty.
What drive do you use? Consumer or prosumer drives or maybe even QLC? Tell us the drive model.
No.
ZFS does not by design cause more wear than ext4.
But there are two small diffrences, that can have an impact.
1. ZFS is CoW.
Every sync write, will write double (ZIL). But since almost nothing is a sync write, that should not matter.
If you do have many sync writes (DB), you want move ZIL to a SLOG enterprise drive with PLP to speed up performance anyway.
2. ZFS uses by default 16k ZVOL.
So your RAW VM disks also use 16k.
ext4 uses by default 64k QCOW2 disks.
Why should it matter if you write in 16k or 64k chunks?
Well, normally it should not. It does definetly not matter for HDD, but it can for SSDs. Why? Well, while SSDs report to use 4k sectors, the firmware controller does stuff differently behind in NAND. So it could be, that the drive internally uses 64k.
If you want to write a 64k file, ZFS would do 4 times a 16k write, each causing 64k (or 256k total), while the ext4 only does 64k.
So you have a 4 times write amplification.
BTW the other way round can also be true! If you want to do a 4k write, ZFS needs to change 16k. If the SSD does not have controller amplification, that can result in an only 16k write which is only 4x write ampflification.
ext4 qcow2 on the other hand, would write 64k, which is 16x write ampflification.
That is why enterprise drives often not only specify one TBW but two. One TBW for 64k writes, and one TBW for 4k writes. Again, none trash SSDs, don't really suffer from that.
Every sync write, will write double (ZIL). But since almost nothing is a sync write, that should not matter.
If you do have many sync writes (DB), you want move ZIL to a SLOG enterprise drive with PLP to speed up performance anyway.
2. ZFS uses by default 16k ZVOL.
So your RAW VM disks also use 16k.
ext4 uses by default 64k QCOW2 disks.
Why should it matter if you write in 16k or 64k chunks?
Well, normally it should not. It does definetly not matter for HDD, but it can for SSDs. Why? Well, while SSDs report to use 4k sectors, the firmware controller does stuff differently behind in NAND. So it could be, that the drive internally uses 64k.
If you want to write a 64k file, ZFS would do 4 times a 16k write, each causing 64k (or 256k total), while the ext4 only does 64k.
So you have a 4 times write amplification.
BTW the other way round can also be true! If you want to do a 4k write, ZFS needs to change 16k. If the SSD does not have controller amplification, that can result in an only 16k write which is only 4x write ampflification.
ext4 qcow2 on the other hand, would write 64k, which is 16x write ampflification.
That is why enterprise drives often not only specify one TBW but two. One TBW for 64k writes, and one TBW for 4k writes. Again, none trash SSDs, don't really suffer from that.
I don't belong to the "you have to use enterprise SSDs" crowd. But I do belong to the "you can't use trash SSDs" crowd.
Last edited:
Can you tell us please, witch Drive do you use and which ZFS config are your setup based on?
I had quite fast increase of wear out, I was suspecting that logs are the problem. By simply using log2ram the situation has improved a lot. Of course this decision will make troubleshooting more difficult if my host crashes.
Proxmox logs a lot, and when it is amplified by (many small) sync writes due to ZFS (or BTRFS), especially QLC drives can wear fast. Lots of older threads about this with work-around like non-persistent logging, disabling unused services, using LVM(-thin) instead or using drives with much higher write endurance or (enterprise) PLP.
Proxmox is wasteful, as they in effect are catering to enterprise users and an assumption is made enterprise storage is used, so its unlikely these issues will change, examples of waste including the logging, the scratch storage when uploading ISO's, every single update on the screen, graph updates etc. is logged to an access log.
I have two thoughts, first where does this 6% come from, do you have a full SMART stat output? I am assuming at the moment it is write cycles, but it also could be reallocated sector usage.
Second there is ways massively decrease writes. But of course they are unsupported, so I wont give step by step, but merely hints that you can research,
Frontend in front of pveproxy to buffer log writes.
Redirect web access writes to /dev/null.
Both of these likely mitigated with your ramdisk though.
Making drives behave same as eneterprise drives, ZFS can be told to not send flush commands to the storage device, this does not disable ZIL or force only async writes, it just stops forcing the disk to do immediate flushes (same way enterprise drives behave).
Small i/o of 32k or less can also have a duplicated write removed with some tuning.
I have two thoughts, first where does this 6% come from, do you have a full SMART stat output? I am assuming at the moment it is write cycles, but it also could be reallocated sector usage.
Second there is ways massively decrease writes. But of course they are unsupported, so I wont give step by step, but merely hints that you can research,
Frontend in front of pveproxy to buffer log writes.
Redirect web access writes to /dev/null.
Both of these likely mitigated with your ramdisk though.
Making drives behave same as eneterprise drives, ZFS can be told to not send flush commands to the storage device, this does not disable ZIL or force only async writes, it just stops forcing the disk to do immediate flushes (same way enterprise drives behave).
Small i/o of 32k or less can also have a duplicated write removed with some tuning.
my results are quite ok - I used 2 nvme (in zfs mirror, 2TB each) only on this machine since new and only as Proxmox and result is (for both is the same):
so 14% after 2 years and 8 months
considering I have 7x the life, it means I still have ... many years ahead
and I have 20+ VMs and LXC - some are writing a lot (databases etc)
I don't think it is Proxmox itself that eats your SSD
Code:
└──| nvme smart-log /dev/nvme0n1
Smart Log for NVME device:nvme0n1 namespace-id:ffffffff
critical_warning : 0
temperature : 32 °C (305 K)
available_spare : 100%
available_spare_threshold : 5%
percentage_used : 14%
endurance group critical warning summary: 0
Data Units Read : 560212626 (286.83 TB)
Data Units Written : 400418594 (205.01 TB)
host_read_commands : 10698443601
host_write_commands : 4831943127
controller_busy_time : 13955
power_cycles : 92
power_on_hours : 23741
unsafe_shutdowns : 39
media_errors : 0
num_err_log_entries : 0
Warning Temperature Time : 0
Critical Composite Temperature Time : 0
Temperature Sensor 1 : 32 °C (305 K)
Thermal Management T1 Trans Count : 0
Thermal Management T2 Trans Count : 0
Thermal Management T1 Total Time : 0
Thermal Management T2 Total Time : 0
└──|so 14% after 2 years and 8 months
considering I have 7x the life, it means I still have ... many years ahead
and I have 20+ VMs and LXC - some are writing a lot (databases etc)
I don't think it is Proxmox itself that eats your SSD
Code:
smartctl -a /dev/nvme0n1
smartctl 7.5 2025-04-30 r5714 [x86_64-linux-7.0.14-25-pve] (local build)
Copyright (C) 2002-25, Bruce Allen, Christian Franke, www.smartmontools.org
=== START OF INFORMATION SECTION ===
Model Number: CT2000T500SSD8
Serial Number: 2348454C1ED5
Firmware Version: P8CR002
PCI Vendor/Subsystem ID: 0xc0a9
IEEE OUI Identifier: 0x00a075
Total NVM Capacity: 2,000,398,934,016 [2.00 TB]
Unallocated NVM Capacity: 0
Controller ID: 0
NVMe Version: 2.0
Number of Namespaces: 1
Namespace 1 Size/Capacity: 2,000,398,934,016 [2.00 TB]
Namespace 1 Formatted LBA Size: 512
Namespace 1 IEEE EUI-64: 00a075 01454c1ed5
Local Time is: Sat Oct 24 17:49:34 2026 BST
Firmware Updates (0x14): 2 Slots, no Reset required
Optional Admin Commands (0x0017): Security Format Frmw_DL Self_Test
Optional NVM Commands (0x00d7): Comp Wr_Unc DS_Mngmt Sav/Sel_Feat Timestmp Verify
Log Page Attributes (0x3e): Cmd_Eff_Lg Ext_Get_Lg Telmtry_Lg Pers_Ev_Lg Log0_FISE_MI
Maximum Data Transfer Size: 128 Pages
Warning Comp. Temp. Threshold: 86 Celsius
Critical Comp. Temp. Threshold: 88 Celsius
Namespace 1 Features (0x08): No_ID_Reuse
Supported Power States
St Op Max Active Idle RL RT WL WT Ent_Lat Ex_Lat
0 + 8.25W - - 0 0 0 0 0 0
1 + 4.00W - - 1 1 1 1 0 0
2 + 3.00W - - 2 2 2 2 0 0
3 - 0.0250W - - 3 3 3 3 2000 3000
4 - 0.0035W - - 4 4 4 4 5000 35000
Supported LBA Sizes (NSID 0x1)
Id Fmt Data Metadt Rel_Perf
0 + 512 0 1
1 - 4096 0 0
=== START OF SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED
SMART/Health Information (NVMe Log 0x02, NSID 0xffffffff)
Critical Warning: 0x00
Temperature: 32 Celsius
Available Spare: 100%
Available Spare Threshold: 5%
Percentage Used: 14%
Data Units Read: 560,212,689 [286 TB]
Data Units Written: 400,423,050 [205 TB]
Host Read Commands: 10,698,446,426
Host Write Commands: 4,832,010,272
Controller Busy Time: 13,955
Power Cycles: 92
Power On Hours: 23,741
Unsafe Shutdowns: 39
Media and Data Integrity Errors: 0
Error Information Log Entries: 0
Warning Comp. Temperature Time: 0
Critical Comp. Temperature Time: 0
Temperature Sensor 1: 32 Celsius
Error Information (NVMe Log 0x01, 16 of 255 entries)
No Errors Logged
Self-test Log (NVMe Log 0x06, NSID 0xffffffff)
Self-test status: No self-test in progress
Num Test_Description Status Power_on_Hours Failing_LBA NSID Seg SCT Code
0 Short Completed without error 15078 - - - - -
Code:
zpool status rpool
pool: rpool
state: ONLINE
status: Some supported and requested features are not enabled on the pool.
The pool can still be used, but some features are unavailable.
action: Enable all features using 'zpool upgrade'. Once this is done,
the pool may no longer be accessible by software that does not support
the features. See zpool-features(7) for details.
scan: scrub repaired 0B in 00:04:05 with 0 errors on Sun Oct 11 00:28:07 2026
config:
NAME STATE READ WRITE CKSUM
rpool ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
nvme-CT2000T500SSD8_2348454C1ED5-part3 ONLINE 0 0 0
nvme-CT2000T500SSD8_2348454C098C-part3 ONLINE 0 0 0
errors: No known data errors
┌──[18:15:39]─[root@monkey:/home/pi]
└──| zpool list -v rpool
NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT
rpool 992G 231G 761G - - 47% 23% 1.00x ONLINE -
mirror-0 992G 231G 761G - - 47% 23.2% - ONLINE -
nvme-CT2000T500SSD8_2348454C1ED5-part3 999G - - - - - - - ONLINE -
nvme-CT2000T500SSD8_2348454C098C-part3 999G - - - - - - - ONLINE -
┌──[18:15:45]─[root@monkey:/home/pi]
└──| zpool get all rpool
NAME PROPERTY VALUE SOURCE
rpool size 992G -
rpool capacity 23% -
rpool altroot - default
rpool health ONLINE -
rpool guid 15821386157924201392 -
rpool version - default
rpool bootfs rpool/ROOT/pve-1 local
rpool delegation on default
rpool autoreplace off default
rpool cachefile - default
rpool failmode wait default
rpool listsnapshots off default
rpool autoexpand off default
rpool dedupratio 1.00x -
rpool free 761G -
rpool allocated 231G -
rpool readonly off -
rpool ashift 12 local
rpool comment - default
rpool expandsize - -
rpool freeing 0 -
rpool fragmentation 47% -
rpool leaked 0 -
rpool multihost off default
rpool checkpoint - -
rpool load_guid 17570024439417582607 -
rpool autotrim off default
rpool compatibility off default
rpool bcloneused 113M -
rpool bclonesaved 113M -
rpool bcloneratio 2.00x -
rpool dedup_table_size 0 -
rpool dedup_table_quota auto default
rpool last_scrubbed_txg 3079343 -
rpool feature@async_destroy enabled local
rpool feature@empty_bpobj active local
rpool feature@lz4_compress active local
rpool feature@multi_vdev_crash_dump enabled local
rpool feature@spacemap_histogram active local
rpool feature@enabled_txg active local
rpool feature@hole_birth active local
rpool feature@extensible_dataset active local
rpool feature@embedded_data active local
rpool feature@bookmarks enabled local
rpool feature@filesystem_limits enabled local
rpool feature@large_blocks enabled local
rpool feature@large_dnode enabled local
rpool feature@sha512 enabled local
rpool feature@skein enabled local
rpool feature@edonr enabled local
rpool feature@userobj_accounting active local
rpool feature@encryption enabled local
rpool feature@project_quota active local
rpool feature@device_removal enabled local
rpool feature@obsolete_counts enabled local
rpool feature@zpool_checkpoint enabled local
rpool feature@spacemap_v2 active local
rpool feature@allocation_classes enabled local
rpool feature@resilver_defer enabled local
rpool feature@bookmark_v2 enabled local
rpool feature@redaction_bookmarks enabled local
rpool feature@redacted_datasets enabled local
rpool feature@bookmark_written enabled local
rpool feature@log_spacemap active local
rpool feature@livelist enabled local
rpool feature@device_rebuild enabled local
rpool feature@zstd_compress enabled local
rpool feature@draid enabled local
rpool feature@zilsaxattr active local
rpool feature@head_errlog active local
rpool feature@blake3 enabled local
rpool feature@block_cloning active local
rpool feature@vdev_zaps_v2 active local
rpool feature@redaction_list_spill enabled local
rpool feature@raidz_expansion enabled local
rpool feature@fast_dedup enabled local
rpool feature@longname enabled local
rpool feature@large_microzap enabled local
rpool feature@dynamic_gang_header disabled local
rpool feature@block_cloning_endian disabled local
rpool feature@physical_rewrite disabled local
┌──[18:15:59]─[root@monkey:/home/pi]
└──| zfs get all rpool
NAME PROPERTY VALUE SOURCE
rpool type filesystem -
rpool creation Fri Apr 17 5:38 2026 -
rpool used 231G -
rpool available 731G -
rpool referenced 96K -
rpool compressratio 1.35x -
rpool mounted yes -
rpool quota none default
rpool reservation none default
rpool recordsize 128K default
rpool mountpoint /rpool default
rpool sharenfs off default
rpool checksum on default
rpool compression on local
rpool atime on local
rpool devices on default
rpool exec on default
rpool setuid on default
rpool readonly off default
rpool zoned off default
rpool snapdir hidden default
rpool aclmode discard default
rpool aclinherit restricted default
rpool createtxg 1 -
rpool canmount on default
rpool xattr sa default
rpool copies 1 default
rpool version 5 -
rpool utf8only off -
rpool normalization none -
rpool casesensitivity sensitive -
rpool vscan off default
rpool nbmand off default
rpool sharesmb off default
rpool refquota none default
rpool refreservation none default
rpool guid 1147838214942478856 -
rpool primarycache all default
rpool secondarycache all default
rpool usedbysnapshots 0B -
rpool usedbydataset 96K -
rpool usedbychildren 231G -
rpool usedbyrefreservation 0B -
rpool logbias latency default
rpool objsetid 54 -
rpool dedup off default
rpool mlslabel none default
rpool sync standard local
rpool dnodesize legacy default
rpool refcompressratio 1.00x -
rpool written 96K -
rpool logicalused 310G -
rpool logicalreferenced 42K -
rpool volmode default default
rpool filesystem_limit none default
rpool snapshot_limit none default
rpool filesystem_count none default
rpool snapshot_count none default
rpool snapdev hidden default
rpool acltype off default
rpool context none default
rpool fscontext none default
rpool defcontext none default
rpool rootcontext none default
rpool relatime on local
rpool redundant_metadata all default
rpool overlay on default
rpool encryption off default
rpool keylocation none default
rpool keyformat none default
rpool pbkdf2iters 0 default
rpool special_small_blocks 0 default
rpool prefetch all default
rpool direct standard default
rpool longname off default
rpool defaultuserquota 0 -
rpool defaultgroupquota 0 -
rpool defaultprojectquota 0 -
rpool defaultuserobjquota 0 -
rpool defaultgroupobjquota 0 -
rpool defaultprojectobjquota 0 -I use log2ram, + installed rsyslog and fwd to an rsyslog-target LXC with 2 vcpu, 384MB RAM (mostly needed for apt upgrades), and a 32G vdisk backed by compressed ZFS. It's tracking (11) instances and using less than 1TB of disk so far. You could easily put the vdisk on a spinner ZFS disk.
YMMV, but that should mitigate your concern about not having logs if a node crashes
Note that since your ZFS mirror is comprised of the same make/model, they will both likely wear out around the same time due to identical wear patterns.
You can get around this a couple of different ways; replace one of the nvmes with a different brand/model, or get another of the same model and replace one that has 6% wear. Since it will have more lifetime out of the box, one of the drives should (almost definitely) fail before the other - and you'll have a spare on hand.
You could also reinstall to a smaller ZFS rpool, say 256-384G, install log2ram and rsyslog (and fwd your logs to some other node, preferably with compressed storage), and use the rest as lvm-thin if you like. Disabling cluster services for a single-node as well will get you pretty far when it comes to ssd wearout. A smaller rpool size also gives you more replacement options, since you don't need a full 2-4TB just to boot the system.
Don't forget you still need backups. A mirror is NOT a backup, and a while back one of our members had both disks in the mirror fail.
https://github.com/kneutron/ansitest/tree/master/proxmox
Look into the bkpcrit script, point it to separate disk / NAS, run it nightly in cron. There are also scripts to mirror your rpool to iscsi and recover from that if necessary, so you would likely be covered (system still running from iscsi) even if both nvmes failed. Do some DR testing.
You can get around this a couple of different ways; replace one of the nvmes with a different brand/model, or get another of the same model and replace one that has 6% wear. Since it will have more lifetime out of the box, one of the drives should (almost definitely) fail before the other - and you'll have a spare on hand.
You could also reinstall to a smaller ZFS rpool, say 256-384G, install log2ram and rsyslog (and fwd your logs to some other node, preferably with compressed storage), and use the rest as lvm-thin if you like. Disabling cluster services for a single-node as well will get you pretty far when it comes to ssd wearout. A smaller rpool size also gives you more replacement options, since you don't need a full 2-4TB just to boot the system.
Don't forget you still need backups. A mirror is NOT a backup, and a while back one of our members had both disks in the mirror fail.
https://github.com/kneutron/ansitest/tree/master/proxmox
Look into the bkpcrit script, point it to separate disk / NAS, run it nightly in cron. There are also scripts to mirror your rpool to iscsi and recover from that if necessary, so you would likely be covered (system still running from iscsi) even if both nvmes failed. Do some DR testing.
Last edited:
your pool rpool has a fragmentation of 47%, so you can only get random 4k read speed.
do you know fio (fio - flexible I/O tester)?
do you know fio (fio - flexible I/O tester)?
apt install fioI search for some tests and i found for your Crucial T500 2TB
random 4k read speed, 70 ~ 80 MByte/s 4k/ 1Q per NVMe.
# https://www.kitguru.net/components/ssd-drives/simon-crisp/crucial-t500-2tb-review/all/1/
# https://www.techpowerup.com/ssd-specs/crucial-t500-2-tb.d1768
# https://hardware-helden.de/crucial-t500-im-test/
random 4k read speed, 70 ~ 80 MByte/s 4k/ 1Q per NVMe.
# https://www.kitguru.net/components/ssd-drives/simon-crisp/crucial-t500-2tb-review/all/1/
# https://www.techpowerup.com/ssd-specs/crucial-t500-2-tb.d1768
# https://hardware-helden.de/crucial-t500-im-test/
Last edited:
It causes more wear when you do a snyc write and when you change a file, yes.
Source?
You mention pveproxy, but that does not cause siginificant writes.
Either way, both isn't really worth mentioning in normal conditions. We should wait for OP to answer the open questions.
Last edited: