Boot stuck at Loading Initial Ramdisk / disk failed

[paulmorabi]

Hi,

I'm using the latest Proxmox 6.2 and noticed this morning that some VM's had stalled and I couldn't SSH into Proxmox or access the web admin. I forced a reboot and now I am stuck at "Loading initial ramdisk ...".

I have several older 5.4.x and even 4.15 kernel versions which all gave the same result. I tried also to edit the boot parameters, remove "quiet" and add "nomodeset" parameter but never get further than the above. I then removed then all parameters related to iommu etc. and also tried rescue mode from the install CD and I get this:




I haven't changed anything recently in terms of hardware and other than Proxmox updates have not also changed anything on the software side.

I'm also not sure which device is being referred to here too as I can't map 1b:00.0 to anything as I can't get to a shell.

I'm using a Ryzen 2700 with two GPU's (RX580 and NVidia 1050ti). Is there anything I can do here? Is this a software or hardware issue?

Any help would be appreciated!

EDIT: I swapped in and out both of the GFX cards. Swapping the Nvidia produced the same as above but taking out the RX580 and I get this:



EDIT 2: Tried to boot a live CD. The SSD/NVME is listed and I can see the LVM volumes. They're all marked as active also. However, when I attempt to fsck them I get an error that there is nothing at /dev/pve/root or any other partition on the volume. Is there any way I can restore the data or get the disk the point where it is working so I can backup the VM's?

EDIT 3: NVME is reporting critical error 0x04 NVM subsystem reliability has been degraded. I guess this means it's completely dead?

Also, strangely, no Linux will boot while my second GFX card is connected and the NVME is failing.

 

[apoc]

I'd try to remove everything "unneeded" from the system and try most basic setup possible.
Are you booting from a USB stick by any chance?
Those little guys gave me a lot of strange errors over the time.
I was moving over from esxi and used to use USB thumb drives but proxmox works different and hence my experience with USB stick was very bad.

 

[paulmorabi]

Yeah, I gradually removed everything and figured out that it was the NVME which is dead. Looks like as it was sharing PCI lanes with the X4 slot, it's failure meant that I got the above errors.