[SOLVED] PVE 8 Samsung NVMe issues

[dusatvoj]

Hello,
I did an upgrade to PVE 8 and after that I started receiving errors on all NVMe drives I have in cluster (3 node cluster).

There are informations of my NVMes (6 same disks - 2 per node)

# smartctl -a /dev/nvme0n1
smartctl 7.3 2022-02-28 r5338 [x86_64-linux-6.2.16-4-pve] (local build)
Copyright (C) 2002-22, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Number:                       SAMSUNG MZQLB1T9HAJR-00007
Serial Number:                      S439NC0R703862
Firmware Version:                   EDA5402Q

There is just single status: `0x4004` but diferrent commands ...

Error Information (NVMe Log 0x01, 16 of 64 entries)
Num   ErrCount  SQId   CmdId  Status  PELoc          LBA  NSID    VS
  0        966     0  0x000f  0x4004      -            0     1     -
  1        965     0  0xc019  0x4004      -            0     1     -
  2        964     0  0xb008  0x4004      -            0     1     -
  3        963     0  0x000e  0x4004      -            0     1     -
  4        962     0  0x400f  0x4004      -            0     1     -
  5        961     0  0xa012  0x4004      -            0     1     -
  6        960     0  0x700f  0x4004      -            0     1     -
  7        959     0  0x600f  0x4004      -            0     1     -
  8        958     0  0x101c  0x4004      -            0     1     -
  9        957     0  0x2011  0x4004      -            0     1     -
 10        956     0  0xb01b  0x4004      -            0     1     -
 11        955     0  0x100b  0x4004      -            0     1     -
 12        954     0  0x500f  0x4004      -            0     1     -
 13        953     0  0xd00c  0x4004      -            0     1     -
 14        952     0  0xd017  0x4004      -            0     1     -
 15        951     0  0x100e  0x4004      -            0     1     -
... (48 entries not read)

Can you help me with debugging what's wrong?
Thank you.

 

[Ramalama]

Can you post an output of zpool status?

But the problem is actually, you will probably get sadly no help, because everyone i know tryes to avoid Samsung enterprise drives.
Not because they are bad, the are probably really great, it's just that they don't provide any firmware updates.

However in the worst case you're on your own, just wanted to prepare you for this.

 

[dusatvoj]

I'm not using ZFS. They are connected via ceph (OS is @ diferrent drives)

 

[dusatvoj]

[SOLVED] Follow-up / TL;DR

After further investigation, I managed to reproduce and resolve the continuously increasing NVMe SMART error counters.

• Root cause: smartmontools 7.4 was querying the NVMe self-test log using an incorrect namespace ID, resulting in Invalid Field in Command errors rather than actual media failures.
• Reproduction: Both smartctl -l selftest and smartctl -x --json=o incremented the error counter. Ceph's device-health monitoring also executes the latter command periodically (by default, daily).
• Fix: Upgraded smartmontools from 7.4-pve1 to 7.5-pve2, which uses the broadcast NSID (0xffffffff) for the self-test log.
• Verification: Repeated the same commands on all affected nodes. Error counters stopped increasing (delta=0). Media and Data Integrity Errors remained at zero.

The historical error counters remain stored on the drives, but they do not indicate NAND degradation in this case.

One caveat: the original 2023 report used smartctl 7.3, so I cannot conclusively attribute the earliest errors to this specific 7.4 bug. However, the subsequently reproducible issue is now resolved.

Upstream issue: https://github.com/smartmontools/smartmontools/issues/217

Thanks to everyone who responded!