Hey after i watched a video from picloud where he tested his new system i did the same to my running proxmox and i was shocked.
why are my iops so incredible low and my rpool (ZFS)
The nvme are rated for:
- read: 7400 mb/s
- write: 6500MB/s SLC Cache
How can i make it faster (its in production and i cant loose data and yes im using proxmox backup server)
PS: i included everything i did, sorry if its to much. atleast i tried to format it ^^
My Setup:
- 13900k
- 64gb ddr5 @4800
- z790 arous elite ax
- 2x Lexar NM790 2TB NVME
- 1x SanDisk SDSSDP256G
- 1x Crucial CT525MX
- 3x Toshiba MG08 16TB
why are my iops so incredible low and my rpool (ZFS)
The nvme are rated for:
- read: 7400 mb/s
- write: 6500MB/s SLC Cache
How can i make it faster (its in production and i cant loose data and yes im using proxmox backup server)
PS: i included everything i did, sorry if its to much. atleast i tried to format it ^^
My Setup:
- 13900k
- 64gb ddr5 @4800
- z790 arous elite ax
- 2x Lexar NM790 2TB NVME
- 1x SanDisk SDSSDP256G
- 1x Crucial CT525MX
- 3x Toshiba MG08 16TB
Code:
root@proxmox:~# zpool status
pool: rpool
state: ONLINE
scan: scrub repaired 4K in 00:03:36 with 0 errors on Sun Apr 13 00:27:37 2025
config:
NAME STATE READ WRITE CKSUM
rpool ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
nvme-eui.0000000941126208caf25b0612002824-part3 ONLINE 0 0 0
nvme-eui.0000000941126208caf25b0612002725-part3 ONLINE 0 0 0
cache
sdb ONLINE 0 0 0
errors: No known data errors
pool: vault
state: ONLINE
scan: scrub repaired 0B in 09:11:33 with 0 errors on Sun Apr 13 09:35:35 2025
config:
NAME STATE READ WRITE CKSUM
vault ONLINE 0 0 0
raidz1-0 ONLINE 0 0 0
ata-TOSHIBA_MG08ACA16TE_31C0A09KFVGG ONLINE 0 0 0
ata-TOSHIBA_MG08ACA16TE_31C0A08VFVGG ONLINE 0 0 0
ata-TOSHIBA_MG08ACA16TE_4310A01CFVGG ONLINE 0 0 0
errors: No known data errors
Code:
root@proxmox:~# fio --name=randread --ioengine=libaio --rw=randread --bs=4k --size=4G --numjobs=4 --iodepth=32 --runtime=60 --group_reporting --directory=/rpool/bench
randread: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=32
...
fio-3.33
Starting 4 processes
Jobs: 4 (f=4): [r(4)][100.0%][r=6422KiB/s][r=1605 IOPS][eta 00m:00s]
randread: (groupid=0, jobs=4): err= 0: pid=2681941: Thu May 8 00:38:39 2025
read: IOPS=1646, BW=6588KiB/s (6746kB/s)(386MiB/60004msec)
slat (usec): min=2, max=353524, avg=2425.52, stdev=4416.52
clat (usec): min=2, max=689843, avg=75251.88, stdev=35459.27
lat (msec): min=3, max=693, avg=77.68, stdev=36.18
clat percentiles (msec):
| 1.00th=[ 15], 5.00th=[ 32], 10.00th=[ 45], 20.00th=[ 59],
| 30.00th=[ 67], 40.00th=[ 72], 50.00th=[ 77], 60.00th=[ 81],
| 70.00th=[ 84], 80.00th=[ 88], 90.00th=[ 94], 95.00th=[ 102],
| 99.00th=[ 190], 99.50th=[ 239], 99.90th=[ 609], 99.95th=[ 667],
| 99.99th=[ 676]
bw ( KiB/s): min= 2000, max=14760, per=99.90%, avg=6581.71, stdev=587.59, samples=476
iops : min= 500, max= 3690, avg=1645.43, stdev=146.90, samples=476
lat (usec) : 4=0.01%, 10=0.01%
lat (msec) : 2=0.01%, 4=0.03%, 10=0.23%, 20=1.84%, 50=11.14%
lat (msec) : 100=81.28%, 250=5.07%, 500=0.27%, 750=0.13%
cpu : usr=0.14%, sys=2.25%, ctx=64031, majf=0, minf=178
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=99.9%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
issued rwts: total=98823,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=32
Run status group 0 (all jobs):
READ: bw=6588KiB/s (6746kB/s), 6588KiB/s-6588KiB/s (6746kB/s-6746kB/s), io=386MiB (405MB), run=60004-60004msec
Code:
root@proxmox:~# lspci -vv | grep -i -A20 nvme
02:00.0 Non-Volatile memory controller: Phison Electronics Corporation PS5027-E27T PCIe4 NVMe Controller (DRAM-less) (rev 01) (prog-if 02 [NVM Express])
Subsystem: Phison Electronics Corporation PS5027-E27T PCIe4 NVMe Controller (DRAM-less)
Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 64 bytes
Interrupt: pin A routed to IRQ 16
IOMMU group: 15
Region 0: Memory at 44100000 (64-bit, non-prefetchable) [size=16K]
Capabilities: [80] Express (v2) Endpoint, MSI 00
DevCap: MaxPayload 512 bytes, PhantFunc 0, Latency L0s unlimited, L1 unlimited
ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ SlotPowerLimit 75W
DevCtl: CorrErr+ NonFatalErr+ FatalErr+ UnsupReq+
RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+ FLReset-
MaxPayload 256 bytes, MaxReadReq 512 bytes
DevSta: CorrErr- NonFatalErr- FatalErr- UnsupReq- AuxPwr- TransPend-
LnkCap: Port #1, Speed 16GT/s, Width x4, ASPM L1, Exit Latency L1 unlimited
ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+
LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 16GT/s, Width x4
TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
DevCap2: Completion Timeout: Range ABCD, TimeoutDis+ NROPrPrP- LTR+
--
Kernel driver in use: nvme
Kernel modules: nvme
03:00.0 Non-Volatile memory controller: Phison Electronics Corporation PS5027-E27T PCIe4 NVMe Controller (DRAM-less) (rev 01) (prog-if 02 [NVM Express])
Subsystem: Phison Electronics Corporation PS5027-E27T PCIe4 NVMe Controller (DRAM-less)
Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 64 bytes
Interrupt: pin A routed to IRQ 16
IOMMU group: 16
Region 0: Memory at 43f00000 (64-bit, non-prefetchable) [size=16K]
Capabilities: [80] Express (v2) Endpoint, MSI 00
DevCap: MaxPayload 512 bytes, PhantFunc 0, Latency L0s unlimited, L1 unlimited
ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ SlotPowerLimit 25W
DevCtl: CorrErr+ NonFatalErr+ FatalErr+ UnsupReq+
RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+ FLReset-
MaxPayload 256 bytes, MaxReadReq 512 bytes
DevSta: CorrErr- NonFatalErr- FatalErr- UnsupReq- AuxPwr- TransPend-
LnkCap: Port #1, Speed 16GT/s, Width x4, ASPM L1, Exit Latency L1 unlimited
ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+
LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 16GT/s, Width x4
TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
DevCap2: Completion Timeout: Range ABCD, TimeoutDis+ NROPrPrP- LTR+
--
Kernel driver in use: nvme
Kernel modules: nvme
Code:
nvme1n1 Lexar SSD NM790 2TB 1.9T 0
├─nvme1n1p1 1007K 0
├─nvme1n1p2 1G 0
└─nvme1n1p3 1.9T 0
nvme0n1 Lexar SSD NM790 2TB 1.9T 0
├─nvme0n1p1 1007K 0
├─nvme0n1p2 1G 0
└─nvme0n1p3 1.9T 0
Code:
smartctl 7.3 2022-02-28 r5338 [x86_64-linux-6.8.12-8-pve] (local build)
Copyright (C) 2002-22, Bruce Allen, Christian Franke, www.smartmontools.org
=== START OF INFORMATION SECTION ===
Model Number: Lexar SSD NM790 2TB
Serial Number: PMC669R0008570P271
Firmware Version: ERFMD0.0
PCI Vendor/Subsystem ID: 0x1987
IEEE OUI Identifier: 0x6479a7
Controller ID: 0
NVMe Version: 1.4
Number of Namespaces: 1
Namespace 1 Size/Capacity: 2,048,408,248,320 [2.04 TB]
Namespace 1 Formatted LBA Size: 512
Namespace 1 IEEE EUI-64: 6479a7 9dbad00ba8
Local Time is: Thu May 8 00:44:48 2025 CEST
Firmware Updates (0x12): 1 Slot, no Reset required
Optional Admin Commands (0x0017): Security Format Frmw_DL Self_Test
Optional NVM Commands (0x005f): Comp Wr_Unc DS_Mngmt Wr_Zero Sav/Sel_Feat Timestmp
Log Page Attributes (0x1e): Cmd_Eff_Lg Ext_Get_Lg Telmtry_Lg Pers_Ev_Lg
Maximum Data Transfer Size: 64 Pages
Warning Comp. Temp. Threshold: 83 Celsius
Critical Comp. Temp. Threshold: 85 Celsius
Namespace 1 Features (0x08): No_ID_Reuse
Supported Power States
St Op Max Active Idle RL RT WL WT Ent_Lat Ex_Lat
0 + 5.70W - - 0 0 0 0 0 0
1 + 3.00W - - 1 1 1 1 0 0
2 + 1.80W - - 2 2 2 2 0 0
3 - 0.0500W - - 3 3 3 3 5000 2500
4 - 0.0050W - - 4 4 4 4 8000 40000
Supported LBA Sizes (NSID 0x1)
Id Fmt Data Metadt Rel_Perf
0 + 512 0 1
1 - 4096 0 0
=== START OF SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED
SMART/Health Information (NVMe Log 0x02)
Critical Warning: 0x00
Temperature: 34 Celsius
Available Spare: 100%
Available Spare Threshold: 5%
Percentage Used: 0%
Data Units Read: 1,399,439 [716 GB]
Data Units Written: 12,129,383 [6.21 TB]
Host Read Commands: 16,736,993
Host Write Commands: 190,014,644
Controller Busy Time: 350
Power Cycles: 11
Power On Hours: 1,750
Unsafe Shutdowns: 6
Media and Data Integrity Errors: 0
Error Information Log Entries: 0
Warning Comp. Temperature Time: 0
Critical Comp. Temperature Time: 0
Temperature Sensor 1: 34 Celsius
Error Information (NVMe Log 0x01, 16 of 255 entries)
No Errors Logged
Code:
NAME PROPERTY VALUE SOURCE
rpool type filesystem -
rpool creation Sun Feb 23 15:47 2025 -
rpool used 302G -
rpool available 1.51T -
rpool referenced 16.0G -
rpool compressratio 1.25x -
rpool mounted yes -
rpool quota none default
rpool reservation none default
rpool recordsize 128K default
rpool mountpoint /rpool default
rpool sharenfs off default
rpool checksum on default
rpool compression on local
rpool atime on local
rpool devices on default
rpool exec on default
rpool setuid on default
rpool readonly off default
rpool zoned off default
rpool snapdir hidden default
rpool aclmode discard default
rpool aclinherit restricted default
rpool createtxg 1 -
rpool canmount on default
rpool xattr on default
rpool copies 1 default
rpool version 5 -
rpool utf8only off -
rpool normalization none -
rpool casesensitivity sensitive -
rpool vscan off default
rpool nbmand off default
rpool sharesmb off default
rpool refquota none default
rpool refreservation none default
rpool guid 9749832602965354269 -
rpool primarycache all default
rpool secondarycache all default
rpool usedbysnapshots 0B -
rpool usedbydataset 16.0G -
rpool usedbychildren 286G -
rpool usedbyrefreservation 0B -
rpool logbias latency default
rpool objsetid 54 -
rpool dedup off default
rpool mlslabel none default
rpool sync standard local
rpool dnodesize legacy default
rpool refcompressratio 1.00x -
rpool written 16.0G -
rpool logicalused 377G -
rpool logicalreferenced 16.0G -
rpool volmode default default
rpool filesystem_limit none default
rpool snapshot_limit none default
rpool filesystem_count none default
rpool snapshot_count none default
rpool snapdev hidden default
rpool acltype off default
rpool context none default
rpool fscontext none default
rpool defcontext none default
rpool rootcontext none default
rpool relatime on local
rpool redundant_metadata all default
rpool overlay on default
rpool encryption off default
rpool keylocation none default
rpool keyformat none default
rpool pbkdf2iters 0 default
rpool special_small_blocks 0 default
rpool prefetch all default
Code:
nvme 57344 2
nvme_core 204800 3 nvme
nvme_auth 24576 1 nvme_core
Last edited: