2xNVMe datastore not showing anymore

[Andreas1138]

Hello,
I have the following configuration:

Lenovo ThinkStation P3
Intel Core i9-13900K
128GB of DDR5 RAM
2 x 250GB Samsung 870 EVO used for Proxmox OS (working fine)
2 x 4TB NVMe Samsung 990 Pro

Everthing worked fine since yesterday, when I am not able to see the NVMe datastore.

The interesting part of the log it's probably the following:

May 12 11:28:33 pve kernel: pcieport 0000:00:1b.0: AER: Multiple Corrected error received: 0000:00:1b.0
May 12 11:28:33 pve kernel: pcieport 0000:00:1b.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, (Receiver ID)
May 12 11:28:33 pve kernel: pcieport 0000:00:1b.0: device [8086:7ac4] error status/mask=00002001/00002000
May 12 11:28:33 pve kernel: pcieport 0000:00:1b.0: [ 0] RxErr (First)
May 12 11:29:06 pve kernel: nvme nvme1: controller is down; will reset: CSTS=0xffffffff, PCI_STATUS=0xffff
May 12 11:29:06 pve kernel: nvme nvme1: Does your device have a faulty power saving mode enabled?
May 12 11:29:06 pve kernel: nvme nvme1: Try "nvme_core.default_ps_max_latency_us=0 pcie_aspm=off" and report a bug
May 12 11:29:06 pve kernel: nvme 0000:02:00.0: Unable to change power state from D3cold to D0, device inaccessible
May 12 11:29:06 pve kernel: nvme nvme1: Disabling device after reset failure: -19
May 12 11:29:06 pve zed[1413188]: eid=17 class=data pool='datastore01' priority=3 err=6 flags=0xc001 bookmark=644:131:0:224198

I tried adding the line pcie_aspm=off to the grub line GRUB_CMDLINE_LINUX_DEFAULT with no resultes (after reboot).

May 13 19:06:22 pve smartd[1212]: Monitoring 2 ATA/SATA, 0 SCSI/SAS and 0 NVMe devices

I attach two log files, before and after reboot (with the added pcie_aspm=off line).
Thank you for your support.

 

[Kingneutron]

Make sure you have the latest firmware for the 990 Pros, there have been reported problems

https://duckduckgo.com/?q=990+pro+problems&ia=web

 

[Andreas1138]

Thanks for the reply. I guess I can't upgrade the firmware in this state. What course of action do you suggest?

Also, since they are two, is it too much to guess they both had problems?

 

[Kingneutron]

I haven't had to deal with it personally, but I would attach the drives one at a time to a Windows system and run the firmware upgrade there.

https://semiconductor.samsung.com/consumer-storage/support/tools/

 

[Andreas1138]

The firmware was already updated. If I turn off and on proxmox, everything works in order. Unfortunately a reboot is not enough.
SSDs are working fine.
I am worndering if it's something related to saving mode (if I look at the logs).The room is cold, I set the fans at max speed, I don't think it's a heat protection.