[SOLVED] Help - Proxmox not starting with a /sbin/zpool import -N: not found error

[pookieman]

starting seeing this error on reboot - this is latest version of proxmox v7

 

[pookieman]

So I did some googling and found this video .. which seems to describe my situation

https://www.youtube.com/watch?v=kJB6BOtKKNU&t=1s

I tried the commands mentioned it didn't help. The import command runs fine. But when I type exit .. I'm back where I started..



It seems like 'rpool' is the pool that is my mirrored SSDs .. the 'Main' pool is where I keep my data

 

[pookieman]

this is what zpool status shows .. why it it only showing ... -part3 of the 2 disks

 

[pookieman]

so I type in exit .. took me back to prompt .. and I repeated a bunch of times.. then this ..



Is this pretty bad? I feel like the disks are corrupt

 

[Dunuin]

this is what zpool status shows .. why it it only showing ... -part3 of the 2 disks

That is normal. You can't directly boot from ZFS so you need other partitions that are not ZFS to boot from (ESP or grub). So your pool is just build using partition 3.

Did you try to boot from the PVE Iso in Rescue Mode?

 

[pookieman]

That is normal. You can't directly boot from ZFS so you need other partitions that are not ZFS to boot from (ESP or grub). So your pool is just build using partition 3.

Did you try to boot from the PVE Iso in Rescue Mode?

any suggestions what to try next?

 

[pookieman]

That is normal. You can't directly boot from ZFS so you need other partitions that are not ZFS to boot from (ESP or grub). So your pool is just build using partition 3.

Did you try to boot from the PVE Iso in Rescue Mode?

no .. let me try that

 

[pookieman]

Wow that actually worked .. it booted ..

but .. now I'm nervous .. do i unplug the USB key and reboot again?

 

[pookieman]

But .. I just rebooted the server .. removed the USB and back to where I started ..

If there was an issue with the zfs pool .. then how could proxmox be back .. it's installed on the paired SSDs

So I can boot into proxmox via the rescue option on the USB.. but not normally

 

[pookieman]

So something is clearly up with the boot process .. are there are any commands I can run to check the health of the boot drives?

 

[Dunuin]

Did you maybe update a old PVE6 that was using grub to PVE7? Short before the PVE7 release they switched to the proxmox-boot-tool because otherwise you wouldn't be able to boot after updating to PVE7. But so if your PVE was installed with PVE6.3 or older you might be uying the old bootloader.

Not sure why you can't boot from your drives but in rescue mode. You could try to boot in rescue mode and then write your bootloader and initramfs to the disk again if one if these is maybe corrupted.
And you could run a long smart test to see if one of your drives maybe causes problems.

 

[Dunuin]

So something is clearly up with the boot process .. are there are any commands I can run to check the health of the boot drives?

With fdisk -l you could check what drives are available. And you could start a smart test for example with smartctl -t long /dev/sda. That will then run in the background. You could use smartctl -a /dev/sda to see the results and how much of that test is finished.

You can write the initramfs with update-initramfs -u. And for the bootloader it depends if your installation is using systemd or grub to boot. For grub for example it should be grub-install <new disk>. And you could check if you are using the proxmox-boot-tool with this command proxmox-boot-tool status. And if you are using the proxmox-boot-tool with systemd boot you could use this proxmox-boot-tool format <new disk's ESP>, proxmox-boot-tool init <new disk's ESP>.
But if I remember right you first would need to chroot from the rescue mode into your old installation on the disks and you then need to bind-mount your /proc, /sys, /boot and /dev from rescue mode to your old installation.
Not sure how to do that with PVE. Only did that once with Debian 10 to rebuild my initramfs and grub bootloader.
Maybe someone of the staff can help before you try what I wrote, because if somethign is wrong you might damage your installation even more.

But doing smart tests shouldn't be able to cause problems.

 

[pookieman]

OK I've discovered something but I don't understand why it would cause the issue ..

I was trying to get PCIE pass through working for graphics card .. here is that thread https://forum.proxmox.com/threads/c...hrough-nvidia-geforce-9600.96659/#post-419368

As part of the steps I updated this file

/etc/kernel/cmdline

and I changed it to

root=ZFS=rpool/ROOT/pve-1 boot=zfs quiet intel_iommu=on iommu=pt vfio_iommu_type1.allow_unsafe_interrupts=1 pcie_acs_override=downstream video=efifb:eek:ff video=vesafb:eek:ff

then I ran this command

update-initramfs -u

that's when the trouble began ..

I just booted via USB and rescue .. edit the file back to

root=ZFS=rpool/ROOT/pve-1 boot=zfs quiet

then ran the update-initramfs -u

and everything is back .. OMG .. that's taken 4 hours of my life

I'm still no closer to getting the PCIE pass through working but I'm too tired to care right now.. getting a G&T - back on this tomorrow.. thanks for your help

 

[Dunuin]

and everything is back .. OMG .. that's taken 4 hours of my life

Good to hear.

I think with "pcie_acs_override=downstream" you force to break the IOMMU groups. That can cause alot of troubles and onboard stuff like NICs, SATA controller and so on may fail. If you have problems with your IOMMU groups you should first try to switch PCIe slots. Best would be to find a PCIe slot that is directly connected to your CPU and not to the mainboards chipset. Generally these got their own IOMMU group.

And for the future:
Backup your VMs/LXCs using Vzdump or PBS. And you can backup your complete system disks on block level by booting into a clonezilla usb stick. If then something happens you can easily restore your complete system disks (incl bootloader and everything) or just your VMs/LXCs from backups. That way you don't need to panic if something isn't working anymore.

 

[leesteken]

OK I've discovered something but I don't understand why it would cause the issue ..

As part of the steps I updated this file

/etc/kernel/cmdline

and I changed it to

root=ZFS=rpool/ROOT/pve-1 boot=zfs quiet intel_iommu=on iommu=pt vfio_iommu_type1.allow_unsafe_interrupts=1 pcie_acs_override=downstream video=efifb:eek:ff video=vesafb:eek:ff

Did you intend to write video=efifb:eek:ff video=vesafb:eek:ff (which cannot be correct) or did you type video=efifb:off video=vesafb:off and did the forum-software convert it into emoticons?
Maybe next time remove quiet to see more information on the screen (except ofcourse when using all the video off options. Those may cause it to display nothing at all).