NVMe's dropping off

[nunodsrodrigues]

Hi,

I have 3 Samsung 990 PRO 2TB running the latest firmware and from time to time one of them disappears from the server and only a power cycle will bring it back. It happened with the 3 devices on different motherboard slots also.

After researching I saw many people facing the same issue with different nvme brands so I tried to add "nvme_core.default_ps_max_latency_us=0" to the grub but it happened again.

I don't know what else to try... at the beginning i was pointing to hardware failure but now I think it should be OS related since I know people with same config (not using proxmox) and they don't have any issue. All hardware is new, 5 months, and since then I faced this issue 4 times. The first time was on Jul 13 using kernel version 6.5.13-3-pve. Currently I'm using 6.8.12-2-pve.





Any help would be appreciated.

Regards.

 

[esi_y]

Hi,

I have 3 Samsung 990 PRO 2TB running the latest firmware and from time to time one of them disappears from the server and only a power cycle will bring it back. It happened with the 3 devices on different motherboard slots also.

After researching I saw many people facing the same issue with different nvme brands so I tried to add "nvme_core.default_ps_max_latency_us=0" to the grub but it happened again.

I don't know what else to try... at the beginning i was pointing to hardware failure but now I think it should be OS related since I know people with same config (not using proxmox) and they don't have any issue. All hardware is new, 5 months, and since then I faced this issue 4 times. The first time was on Jul 13 using kernel version 6.5.13-3-pve. Currently I'm using 6.8.12-2-pve.

View attachment 76884

Is this a system that happens to be idle at times? Every time I have something "disappear" on me the first thing I do is limit power saving features to see if it's not e.g. driver related:

https://nvmexpress.org/resource/technology-power-features/

 

[nunodsrodrigues]

Is this a system that happens to be idle at times? Every time I have something "disappear" on me the first thing I do is limit power saving features to see if it's not e.g. driver related:

https://nvmexpress.org/resource/technology-power-features/

The system has few VM's and dozens of docker containers running, not heavy load most of the time but also not idle.