only one of two nvme detected in linux (duplicate subnqn)

[Yury]

Hello!
I have two SSD M.2 ADATA 512GB SX6000 LITE (ASX6000LNP-512GT-C), but only one of them detected in linux.
Dmesg log
[ 0.720367] nvme nvme0: pci function 0000:01:00.0
[ 0.720415] nvme nvme1: pci function 0000:1f:00.0
[ 0.942706] nvme nvme1: ignoring ctrl due to duplicate subnqn (nqn.2018-05.com.example:nvme:nvm-subsystem-OUI00E04C).
[ 0.943017] nvme nvme1: Removing after probe failure status: -22
[ 0.953045] nvme nvme0: allocated 64 MiB host memory buffer.
On lanchpad i found new driver (ubuntu bionic linux 4.18) for nvme with fix, but don't know how to fix it on proxmox 5.4-1 (Debian 9).
Linux pve01 4.15.18-12-pve #1 SMP PVE 4.15.18-35 (Wed, 13 Mar 2019 08:24:42 +0100) x86_64 GNU/Linux

 

[fabian]

please file a bug on https://bugzilla.proxmox.com and include all relevant information like
- pveversion -v output
- potentially relevant bugs/discussions elsewhere (like launchpad)
- full logs

 

[Yury]

please file a bug on https://bugzilla.proxmox.com and include all relevant information like
- pveversion -v output
- potentially relevant bugs/discussions elsewhere (like launchpad)
- full logs

Seems like this is a manufacturer's firmware mistake. I try load with fedora 30 live, kernel 5.0 and have the same problems.

[root@localhost-live liveuser]# nvme list
Node SN Model Namespace Usage Format FW Rev
---------------- -------------------- ---------------------------------------- --------- -------------------------- ---------------- --------
/dev/nvme0n1 123456789012 ADATA SX6000LNP 1 512.11 GB / 512.11 GB 512 B + 0 B V9001c00
[root@localhost-live liveuser]#
[root@localhost-live liveuser]#
[root@localhost-live liveuser]#
[root@localhost-live liveuser]# dmesg | grep -i nvme
[ 5.573787] nvme nvme0: pci function 0000:01:00.0
[ 5.573870] nvme nvme1: pci function 0000:1f:00.0
[ 5.800282] nvme nvme1: allocated 64 MiB host memory buffer.
[ 5.800305] nvme nvme0: allocated 64 MiB host memory buffer.
[ 5.830503] nvme nvme0: 7/0/0 default/read/poll queues
[ 5.830635] nvme nvme1: 7/0/0 default/read/poll queues
[ 5.841630] nvme nvme0: duplicate IDs for nsid 1
[root@localhost-live liveuser]#
[root@localhost-live liveuser]#
[root@localhost-live liveuser]#
[root@localhost-live liveuser]# nvme list-subsys
nvme-subsys0 - NQN=nqn.2018-05.com.example:nvme:nvm-subsystem-OUI00E04C
\
+- nvme0 pcie 0000:01:00.0 live
+- nvme1 pcie 0000:1f:00.0 live
[root@localhost-live liveuser]#
[root@localhost-live liveuser]#
[root@localhost-live liveuser]# ls /dev/nvme* -lh
crw-------. 1 root root 236, 0 May 22 2019 /dev/nvme0
brw-rw----. 1 root disk 259, 0 May 22 2019 /dev/nvme0n1
crw-------. 1 root root 236, 1 May 22 2019 /dev/nvme1

 

[MishaN]

I see that you're the one other person on the whole Internet who's run into this problem with this particular SSD. I just ran into it the other day with my new Thinkpad laptop.

I've made a patch for this that should be going out in the Linux 5.3 kernel. It's a problem with the firmware, but something that can be dealt with by Linux by applying a quirk to this particular NVME. The patch makes the Linux kernel itself generate a subnqn for these SSDs.

(I can't post links as a new member):

lkml DOT org SLASH lkml/2019/7/15/57

Cheers!

 

[Alan99]

He's not the only other person who has issues with this drive(or similar drive from ADATA). There's at least a few more of us...
(Replace the DOT)
forums.unraidDOTnet/topic/81650-unraid-not-detecting-both-m2-drives/

 

[MishaN]

Ah. Well, fortunately it's easy to patch. If you find others, feel free to adapt my patch. `lspci -nn` will let you see the vendor ID and device ID. Just add another entry to the quirks table if it's the same duplicate subnqn issue.

Do you know if anyone's brought up the issue with ADATA? I'd really like for them to fix their lame firmware.

 

[Alan99]

@MishaN

I contacted ADATA "customer service" (in quotes since it is a joke). Their response email looks completely automated as it starts with "Dear Customer" and doesn't even have the name I entered when filling out the online form to contact them. The email starts out acknowledging I have an issue and then goes right into stating they are here to assist with the return of the product. The email did provide a few generic troubleshooting steps, but had absolutely no mention of the issue I explicitly detailed for them (again, since this is an automated, impersonal email.) I'll try replying to the email to see if that actually gets anywhere.

I'm not savvy with Linux. I'm not sure how to add an entry to the quirks table or if that's something possible in unRAID. I'm out of town and just left my server off for the meantime, so I'll have to look into this more when I get back.

 

[MishaN]

Looks like there's a resource on this: wiki(DOT)unraid(DOT)net/Building_a_custom_kernel

I'll send them (ADATA) an email later, too. I'm not sure if it's on Realtek or ADATA, but crappy firmware from Realtek is no surprise

And, since I got these SSDs as part of my laptop through Lenovo, I'll try bugging them, too, to put pressure on ADATA to fix this.

 

[Yury]

@MishaN

I contacted ADATA "customer service" (in quotes since it is a joke). Their response email looks completely automated as it starts with "Dear Customer" and doesn't even have the name I entered when filling out the online form to contact them. The email starts out acknowledging I have an issue and then goes right into stating they are here to assist with the return of the product. The email did provide a few generic troubleshooting steps, but had absolutely no mention of the issue I explicitly detailed for them (again, since this is an automated, impersonal email.) I'll try replying to the email to see if that actually gets anywhere.

I'm not savvy with Linux. I'm not sure how to add an entry to the quirks table or if that's something possible in unRAID. I'm out of town and just left my server off for the meantime, so I'll have to look into this more when I get back.

Hello! I spent almost two months to convince adata technical support that they have an error in the firmware of the ssd disk. They ignored me, but I turned to the distributor and adata support began to answer my questions. They connected two drives ourselves and made sure that they do not work. Appeal CS19051113. After that, I returned the drive to the store.
Adata promised to fix the problem, but did not name the dates.

 

[major]

I have two Adata gammix s5 drives and I'm experiencing the same issue.
Installation only detects one drive.
Are there any instructions on how to build a custom kernel to include the drives model in the kernel's quirks file for the installation iso?

 

[MishaN]

I have two Adata gammix s5 drives and I'm experiencing the same issue.
Installation only detects one drive.
Are there any instructions on how to build a custom kernel to include the drives model in the kernel's quirks file for the installation iso?

I don't know anything about Proxmox, but I do know that the 5.3-rc2 kernel has the patch you are looking for. If you can build the 5.3-rc2 kernel, you'll be all good. Sorry I don't have more info for you.

EDIT: Though, if these drives have a different device ID vs the SX6000LNP, this patch probably won't work.

Could you paste the output of "lspci -nn" please?

 

[major]

Certainly

83:00.0 Class 0108: 10ec:5762
82:00.0 Class 0108: 10ec:5762

 

[MishaN]

Certainly

83:00.0 Class 0108: 10ec:5762
82:00.0 Class 0108: 10ec:5762

That's the same as mine, so 5.3-rc2 (and onward) kernel currently has the patch, and maybe a later 5.2 kernel (after 5.2.6) will have it.

 

[mfg666]

06:00.0 Non-Volatile memory controller [0108]: Realtek Semiconductor Co., Ltd. Device [10ec:5762] (rev 01)
07:00.0 Non-Volatile memory controller [0108]: Realtek Semiconductor Co., Ltd. Device [10ec:5762] (rev 01)

nvme list
-bash: nvme: command not found

how can i add another entry to the quirks table ?

 

[MishaN]

06:00.0 Non-Volatile memory controller [0108]: Realtek Semiconductor Co., Ltd. Device [10ec:5762] (rev 01)
07:00.0 Non-Volatile memory controller [0108]: Realtek Semiconductor Co., Ltd. Device [10ec:5762] (rev 01)

nvme list
-bash: nvme: command not found

how can i add another entry to the quirks table ?

For the nvme command, you need to install the nvme-cli package. In Debian, at least, it's called nvme-cli. Not sure about other distributions. If it's apt-based, try "apt search nvme", if it's yum-based, try "yum search nvme"

As far as adding quirks, they're in this file in the Linux kernel source: https://github.com/torvalds/linux/blob/master/drivers/nvme/host/pci.c
Here is the actual code I added (lines 3030 thru 3031): https://github.com/torvalds/linux/blob/master/drivers/nvme/host/pci.c#L3030

Though, I've discovered that you actually want to base your patch against this repo: http://git.infradead.org/nvme.git -- the commits from here get merged into Linux.

The PCI vendor + device ID you pasted is covered in the patch I've submitted. You can build the latest 5.3-rc kernel and it will already have this patch. A new 5.2 kernel should come out soon with this patch as well.

I'm not familiar with Proxmox, you will probably have to build your own kernel for now. Pre 5.x kernels don't support this subnqn quirk at all, so, if you want to apply this patch yourself to another kernel version, you will need at least a 5.0.x kernel unless you want to backport all of the quirk.

Looks like Proxmox uses its own kernel, though? Looking at this post https://forum.proxmox.com/threads/pve-kernel-git-outdated.44474/

Please do note that you will need a kernel >= 5.0.x. 5.3-rc2 minimum to avoid having to patch the source code yourself.

 

[MishaN]

Also, if going through all this upsets you (as it should!), you may want to consider complaining to ADATA as well. See the comment @Yury posted above. He mentions a specific case number you'd want to reference with them. This doesn't solve the problem in the short term, but hopefully gets them to release a firmware update sooner -- this firmware update would make obsolete all of this quirky business

 

[MishaN]

Update: The 5.2.9 Linux kernel release has the fix.