[SOLVED] Suddenly high iowait

[devan0123]

I started seeing laggy performance and noticed relatively high iowait cpu usage. You can see from the screenshot it just recently started (and I have the same or very similar workload running)

My main storage is an HP EX920 NVME drive and I started researching nvme random i/o fsync performance. I'm seeing crazy low numbers on pveperf. I did a secure erase (nvme format -s1 /dev/nvmen0), rebooted to an ubuntu live USB to rule out proxmox, and have tried fio tests which are also really slow.

CPU BOGOMIPS: 108583.52
REGEX/SECOND: 2343554
HD SIZE: 928.51 GB (/dev/nvme0n1p1)
BUFFERED READS: 2847.27 MB/sec
AVERAGE SEEK TIME: 0.02 ms
FSYNCS/SECOND: 7.50

7.5 fsyncs/second is really really low, even for consumer hardware. Anyone have any suggestions on what else to try? Is there another way to secure erase an nvme ssd besides the nvme-cli command? Reads are very fast, so I'm thinking it's not a controller problem.


hdparm -tT /dev/nvme0n1

/dev/nvme0n1:

Timing cached reads: 11938 MB in 2.00 seconds = 5973.18 MB/sec
Timing buffered disk reads: 8494 MB in 3.00 seconds = 2831.24 MB/sec




I also set up temperature monitoring and installed another fan and it's not going above ~ 35C

 

[Alwin]

Run iostat or atop to see what your system is doing during the high iowait.

 

[devan0123]

Thanks for the reply, sorry I didn't update this thread. Turned out to be a bad drive. I popped a new nvme drive in, spent more time than I ever expected learning about grub to transfer my partitions over and boot off the new drive, and performance is back to normal.