Only 3 nvme disks available

[Metz]

Hello,

I've created a new proxmox environment with 4 nmve disk. But in the GUI and on fdisk I see only 3 disks. Syslog shows all 4 disks.

I changed the location of the disk and the setup on the pcie / nvme card and can see all serial numbres depending on the position of the nvme. But the GUI shows always only 3 disks.

Syslog shows all 4 disks

Sep 30 20:22:04 pve1 smartd[1789]: Device: /dev/nvme0, opened
Sep 30 20:22:04 pve1 smartd[1789]: Device: /dev/nvme0, ADATA SX8200PNP, S/N:2L17xx, FW:42xx
Sep 30 20:22:04 pve1 smartd[1789]: Device: /dev/nvme0, is SMART capable. Adding to "monitor" list.
Sep 30 20:22:04 pve1 smartd[1789]: Device: /dev/nvme0, state read from /var/lib/smartmontools/smartd.ADATA_SX8200PNP-2L17xx.nvme.state
Sep 30 20:22:04 pve1 smartd[1789]: Device: /dev/nvme1, opened
Sep 30 20:22:04 pve1 smartd[1789]: Device: /dev/nvme1, ADATA SX8200PNP, S/N:2N21xx, FW:42xx
Sep 30 20:22:04 pve1 smartd[1789]: Device: /dev/nvme1, is SMART capable. Adding to "monitor" list.
Sep 30 20:22:04 pve1 smartd[1789]: Device: /dev/nvme1, state read from /var/lib/smartmontools/smartd.ADATA_SX8200PNP-2N21xx.nvme.state
Sep 30 20:22:04 pve1 smartd[1789]: Device: /dev/nvme2, opened
Sep 30 20:22:04 pve1 smartd[1789]: Device: /dev/nvme2, ADATA SX8200PNP, S/N:2N21xx, FW:42xx
Sep 30 20:22:04 pve1 smartd[1789]: Device: /dev/nvme2, is SMART capable. Adding to "monitor" list.
Sep 30 20:22:04 pve1 smartd[1789]: Device: /dev/nvme2, state read from /var/lib/smartmontools/smartd.ADATA_SX8200PNP-2N21xx.nvme.state
Sep 30 20:22:04 pve1 smartd[1789]: Device: /dev/nvme3, opened
Sep 30 20:22:04 pve1 smartd[1789]: Device: /dev/nvme3, ADATA SX8200PNP, S/N:2N28xx, FW:32xx
Sep 30 20:22:04 pve1 smartd[1789]: Device: /dev/nvme3, is SMART capable. Adding to "monitor" list.
Sep 30 20:22:04 pve1 smartd[1789]: Device: /dev/nvme3, state read from /var/lib/smartmontools/smartd.ADATA_SX8200PNP-2N28xx.nvme.state
Sep 30 20:22:04 pve1 smartd[1789]: Monitoring 5 ATA/SATA, 0 SCSI/SAS and 4 NVMe devices

Sep 30 20:22:04 pve1 smartd[1789]: Device: /dev/nvme0, state written to /var/lib/smartmontools/smartd.ADATA_SX8200PNP-2L17xx.nvme.state
Sep 30 20:22:04 pve1 smartd[1789]: Device: /dev/nvme1, state written to /var/lib/smartmontools/smartd.ADATA_SX8200PNP-2N21xx.nvme.state
Sep 30 20:22:04 pve1 smartd[1789]: Device: /dev/nvme2, state written to /var/lib/smartmontools/smartd.ADATA_SX8200PNP-2N21xx.nvme.state
Sep 30 20:22:04 pve1 smartd[1789]: Device: /dev/nvme3, state written to /var/lib/smartmontools/smartd.ADATA_SX8200PNP-2N28xx.nvme.state

root@pve1:~# ls -l /dev/nvme*

crw------- 1 root root 241, 0 Sep 30 20:22 /dev/nvme0
brw-rw---- 1 root disk 259, 1 Sep 30 20:22 /dev/nvme0n1
crw------- 1 root root 241, 1 Sep 30 20:22 /dev/nvme1
brw-rw---- 1 root disk 259, 2 Sep 30 20:22 /dev/nvme1n1
crw------- 1 root root 241, 2 Sep 30 20:22 /dev/nvme2
crw------- 1 root root 241, 3 Sep 30 20:22 /dev/nvme3
brw-rw---- 1 root disk 259, 0 Sep 30 20:22 /dev/nvme3n1

root@pve1:~# fdisk -l | grep nvme

Disk /dev/nvme3n1: 1.86 TiB, 2048408248320 bytes, 4000797360 sectors
Disk /dev/nvme0n1: 1.86 TiB, 2048408248320 bytes, 4000797360 sectors
Disk /dev/nvme1n1: 1.86 TiB, 2048408248320 bytes, 4000797360 sectors

First I was thinking that's a hardware defect. But after 3 hours of changing and testing parts I do not believe that.

What do I miss / is there something else I can test?

BR Metz

 

[Metz]

root@pve1:~# lspci | grep ADA

84:00.0 Non-Volatile memory controller: ADATA Technology Co., Ltd. XPG SX8200 Pro PCIe Gen3x4 M.2 2280 Solid State Drive (rev 03)
85:00.0 Non-Volatile memory controller: ADATA Technology Co., Ltd. XPG SX8200 Pro PCIe Gen3x4 M.2 2280 Solid State Drive (rev 03)
86:00.0 Non-Volatile memory controller: ADATA Technology Co., Ltd. XPG SX8200 Pro PCIe Gen3x4 M.2 2280 Solid State Drive (rev 03)
87:00.0 Non-Volatile memory controller: ADATA Technology Co., Ltd. XPG SX8200 Pro PCIe Gen3x4 M.2 2280 Solid State Drive (rev 03)

lspci -vv | awk '/ASPM/{print $0}' RS= | grep --color -P '(^[a-z0-9:.]+|ASPM )'

84:00.0 Non-Volatile memory controller: ADATA Technology Co., Ltd. XPG SX8200 Pro PCIe Gen3x4 M.2 2280 Solid State Drive (rev 03) (prog-if 02 [NVM Express])
                LnkCap: Port #0, Speed 8GT/s, Width x4, ASPM L1, Exit Latency L1 <8us
                LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
85:00.0 Non-Volatile memory controller: ADATA Technology Co., Ltd. XPG SX8200 Pro PCIe Gen3x4 M.2 2280 Solid State Drive (rev 03) (prog-if 02 [NVM Express])
                LnkCap: Port #0, Speed 8GT/s, Width x4, ASPM L1, Exit Latency L1 <8us
                LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
86:00.0 Non-Volatile memory controller: ADATA Technology Co., Ltd. XPG SX8200 Pro PCIe Gen3x4 M.2 2280 Solid State Drive (rev 03) (prog-if 02 [NVM Express])
                LnkCap: Port #0, Speed 8GT/s, Width x4, ASPM L1, Exit Latency L1 <8us
                LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
87:00.0 Non-Volatile memory controller: ADATA Technology Co., Ltd. XPG SX8200 Pro PCIe Gen3x4 M.2 2280 Solid State Drive (rev 03) (prog-if 02 [NVM Express])
                LnkCap: Port #0, Speed 8GT/s, Width x4, ASPM L1, Exit Latency L1 <8us
                LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+

nvme list

Node                  Generic                        Model                                    Namespace Usage                      Format           FW Rev 
--------------------- --------------------- -------------------- ---------------------------------------- --------- -------------------------- ---------------- --------
/dev/nvme3n1          /dev/ng3n1                     ADATA SX8200PNP                          1           0.00   B /   2.05  TB    512   B +  0 B   
/dev/nvme1n1          /dev/ng1n1                     ADATA SX8200PNP                          1           2.04  TB /   2.05  TB    512   B +  0 B   
/dev/nvme0n1          /dev/ng0n1                     ADATA SX8200PNP                          1           0.00   B /   2.05  TB    512   B +  0 B

root@pve1:~# nvme list-subsys

nvme-subsys3 - NQN=nqn.2014.08.org.nvmexpress:1cc11cc12...       ADATA SX8200PNP
\
 +- nvme3 pcie 0000:87:00.0 live
nvme-subsys2 - NQN=nqn.2014.08.org.nvmexpress:1cc11cc12...        ADATA SX8200PNP
\
 +- nvme2 pcie 0000:86:00.0 live
nvme-subsys1 - NQN=nqn.2014.08.org.nvmexpress:1cc11cc12...        ADATA SX8200PNP
\
 +- nvme1 pcie 0000:84:00.0 live
nvme-subsys0 - NQN=nqn.2014.08.org.nvmexpress:1cc11cc12...        ADATA SX8200PNP
\
 +- nvme0 pcie 0000:85:00.0 live

 

[bbgeek17]

The terminology you are looking for is "nvme namespace".
https://narasimhan-v.github.io/2020/06/12/Managing-NVMe-Namespaces.html

Good luck

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox