[SOLVED] PVE Fails to load past 'Loading initial ramdisk...'

[Pragma808]

Hi all,

Recently my home PVE server has been unable to boot properly, hanging at 'Loading initial ramdisk...'. I've done some searches and followed what information I've been able to find, but as a rather inexperienced linux user, I'm at an impasse. It's been about a year since I set this server up, so I apologize for any lacking details, but I'm more than happy to report anything I can find out.

Some info:
- PVE version 5.x (not sure the exact version, but it is using linux 4.15.18-9-pve, as reported by grub).
- Boot device is an NVMe drive set up with LVM & ext2.
- Additional storage is a zRAID 5 consisting of 3 HDDs, a 2nd NVMe with windows installed on it, an optane write cache drive, and another ssd as a read cache. As far as I can recall, all boot critical information is on the boot SSD.
- A windows VM has been set up with GPU passthrough. The guide I followed had me blacklist all GPU drivers from loading, so I (stupidly) don't have a way to interact with the system directly, and in the past I've done everything over the network. I do have a 2nd GPU installed, but never could figure out how to whitelist is specifically.
- System does not boot to a point where it becomes reachable over the network.
- I've been able to boot directly into windows, where I've experienced intermittent issues with USB devices, but mostly functional. This could be a hardware issue, but I can't confirm that at this time.
- CPU: 9980xe w/ 128GB DDR4.

My troubleshooting steps thus far:
- Updated mobo BIOS. No change.
- Edited grub to remove 'quiet' option. Still hangs at 'Loading initial ramdisk' screen.
- Edited grub to remove 'quiet' and 'iommu=on' options. System now eventually loads past loading ramdisk screen, but only to display a black screen for ~1min before rebooting on its own.

Like I said before, I don't have a ton of experience with linux, so I'm basically stuck. Any help you all can provide would be greatly appreciated.

Thank you!

 

[wolfgang]

Hi,

try the nomodeset kernel parameter.
maybe then you get an error output.

 

[Pragma808]

Hi again. Apologies for the extended delay in my response. I finally had the chance to try the nomodeset kernel parameter, and was finally able to get a visual (THANK YOU).

The first thing I noticed was an error relating to a USB device, with some text warning that a cable might be bad. I shut the system down, unplugged everything but a keyboard, then tried again. That's when I got to this:



It looks like it's hanging during the ZFS mount job. I let the system run for ~2 hours, but it never gets past this. I never actually get to the command line, unfortunately. Again, being rather inexperienced with linux and ZFS, I'm stuck here. Any ideas on what I can do next?

Thank you!!

 

[wolfgang]

Try the kernel of PVE 6.0
you can download it manual with wget and then install it.

wget http://download.proxmox.com/debian/pve/dists/buster/pve-no-subscription/binary-amd64/pve-kernel-5.3.13-2-pve_5.3.13-2_amd64.deb
dpkg -i [URL='http://download.proxmox.com/debian/pve/dists/buster/pve-no-subscription/binary-amd64/pve-kernel-5.3.13-2-pve_5.3.13-2_amd64.deb']pve-kernel-5.3.13-2-pve_5.3.13-2_amd64.deb[/URL]

 

[Pragma808]

Seeing as I haven't been able to get to a command line, how can I do this? Or do I need to do a clean install with v6.0? Thanks again

 

[wolfgang]

Try to boot from an old kernel.
At the boot time, you can choose it in the Grub widows.

 

[Pragma808]

I will need to verify, but I believe there is only 1 option listed. I never upgraded the kernal after getting the system up and running. Noobie, mistake, I know!

Thanks again.

 

[wolfgang]

No you have to install it manually.

wget http://download.proxmox.com/debian/pve/dists/stretch/pve-no-subscription/binary-amd64/pve-kernel-4.15.18-9-pve_4.15.18-30_amd64.deb

dpkg -i pve-kernel-4.15.18-9-pve_4.15.18-30_amd64.deb

 

[Pragma808]

Okay, but I'm not sure how to get to a command line, as it's hanging on the ZFS mount job during startup. Are there any other kernel parameters I can try?

 

[wolfgang]

The grub has an advanced menu filed.
Under this menu-point, you see all installed kernels.
If there is only one kernel you must use e to edit the kernel parameter to prevent importing the zfs pool
edit the line which starts with Linux
and add "zfs.zfs_autoimport_disable=1" at the end of the line.
the commands are separated by a space and remove the quotes.

 

[Pragma808]

Hi again,

So even with zfs.zfs_autoimport_disable=1, I found that the system is still trying to import the pool. I've attached a pic of what my grub looks like before loading the OS (grub1.jpg). One thing that may or may not be worth noting is that I noticed an extra space (a double space) in the launch parameters after "ro". I have no idea if this makes a difference, but I tried removing it and loading the OS, but it didn't seem to have an effect (grub2.jpg).

Either way, the system still tries to load the ZFS pool (load_os.jpg). Based on the launch parameters I have in the grubX.jpg screenshots, does anything appear to be incorrect?

I did a google search for zfs.zfs_autoimport_disable=1 and found a github thread reporting similar behavior (https://github.com/openzfs/zfs/issues/2474). If this is the same issue they were experiencing, I believe it might be due to my pool being cached in initramfs. Unfortunately, I have no idea what to do about that, seeing as I haven't even been able to get to the command line yet.

Any thoughts?

Thanks again. I really appreciate the support!

 

[Pragma808]

Hi again. Any thoughts on this?

 

[Pragma808]

One last bump before I consider it a failed cause. Any thoughts, anyone?

 

[Stoiko Ivanov]

try booting with a root-delay set - see https://pve.proxmox.com/wiki/ZFS:_Tips_and_Tricks#Boot_fails_and_goes_into_busybox
maybe this could help get the system up

 

[chudak]

I don't use zfs and also see this problem on 5.4.34-1-pve kernel

 

[ilia987]

did you solve it ? i have similar issue at post-336153

 

[Pragma808]

did you solve it ? i have similar issue at post-336153

Unfortunately no, not really.

However, I did finally manage to get it to boot (without kernel parameters) by removing all my HDD's, though the webui would not load over the network.

 

[Pragma808]

I've come to suspect that there is an issue with one of the drives in my RAID, as it (appears to) spit out errors when loading ZFS.

Here's a screenshot of this:




At this point, having been without my system for many months now, I'm not quite sure what the next step is. I think maybe rebuilding PVE from scratch is a good idea, but I don't know how to recover the data on the drives.

When testing booting the system with certain drives plugged/unplugged, I found that it's likely just a single drive causing the issue. Fortunately, I have a spare drive ready that I could plug in, but don't know how to resilver it or get the array functioning again.

Any help anyone can provide would be greatly appreciated. Thanks again!

 

[r.jochum]

Hmm you could try to boot an Ubuntu Desktop Live Version and see if it works with that (use the latest available it contains ZFS).

If so, you could then mount the pve root chroot in it upgrade and see if it works after.

 

[Pragma808]

Hmm you could try to boot an Ubuntu Desktop Live Version and see if it works with that (use the latest available it contains ZFS).

If so, you could then mount the pve root chroot in it upgrade and see if it works after.

Is there a guide I can reference for this process?