Host will not boot

[mobrien118]

I have a Proxmox VE environment (HOST) that will not boot. This happened after boot was interrupted after installing additional memory. The additional memory was removed but the problem remains, impltying that the hardware is not the issue.

During a normal boot, the system outputs just the following, then no progress (even after a week of sitting):

  Booting Proxmox `VE GNU/Linux'

Loading Linux 6.8.4-2-pve ...
Loading initial ramdisk ...

A safe mode boot provides more verbose output, but ends here:

[    3.988973] systemd[1]: Finished systemd-remount-fs.service Remount Root and Kernel File Systems.
[  OK  ] Finished systemd-remount-fs.service - Remount Root and Kernel File Systems.
[    3.989640] systemd[1 ]: systemd-firstboot.service First Boot Wizard was skipped because of an unmet condition check (ConditionFirstBoot=yes).
[    3.990013] systemd[1]: systemd-pstore.service Platform Persistent Storage Archival was skipped because of an unmet condition check (ConditionDirectoryNotEmpty=/sys/fs/pstore).
[    3.991031] systemd[1]: Starting systemd-random-seed.service - Load/Save Random Seed...
         Starting systemd-random-seed.service Load/Save Random Seed...
[    3.992116] systemd[1]: Starting systemd-sysusers.service - Create Systen Users...
         Starting systemd-sysusers.service - Create System Users...
[    3.999874] systemd[1]: Finished systemd-sysusers.service Create System Users.

I was able to chroot into the system from a GParted LiveCD and run an apt update and apt upgrade that upgraded several packages, including the kernel and initramfs, I believe, so it looks like the file systems is OK in general and the OS seems to be intact.

Any ideas on what could be holding it up? Or ideas where to look?

 

[mobrien118]

What a mess this has turned into.

I gave up trying to get the system to boot. Swapped out the main hard drive and went for a fresh install.

Boots just fine off of the fresh install, but can't use my subscription because it is already assigned to the "other" (the same) server. Since it is unbootable, I can't go in to release it. I went ahead and added the old drive back into the system, but it is still booting off of the "new" one - which is good.

I also can't sign into my Proxmox account (using the last know password I had) and it won't send me a password reset. So, I can't contact support.

The system is complaining that there are 2 LVM groups named "pve" (rightly so) - so I have to figure out which is the old one and rename it, which, I assume, will make the old system even "less" bootable. On the brighter side, the old VM volumes are showing up without me doing anything.

Not looking forward to trying to set this up again and go through the hassle of getting PCI passthrough to work again, although I think I can streamline the process now since I know the dozen or so things that *didn't* work.

Anyone know how to get support if I can't log into my account and can't reset the password?

 

[leesteken]

Boots just fine off of the fresh install, but can't use my subscription because it is already assigned to the "other" (the same) server. Since it is unbootable, I can't go in to release it. I went ahead and added the old drive back into the system, but it is still booting off of the "new" one - which is good.

The system is complaining that there are 2 LVM groups named "pve" (rightly so) - so I have to figure out which is the old one and rename it, which, I assume, will make the old system even "less" bootable. On the brighter side, the old VM volumes are showing up without me doing anything.

You can reissue a new subscription key (and invalidating the old) via the Proxmox shop. I did so recently. You don't need to boot into the old system.

I also can't sign into my Proxmox account (using the last know password I had) and it won't send me a password reset. So, I can't contact support.

Anyone know how to get support if I can't log into my account and can't reset the password?

That's inconvenient and unexpected that Forgot Password does not work. Maybe e-mail the sales office ( https://www.proxmox.com/en/about/contact ), as they handle issues with subscriptions? Or use the contact form: https://shop.proxmox.com/submitticket.php?step=2&deptid=1

 

[mobrien118]

You can reissue a new subscription key (and invalidating the old) via the Proxmox shop. I did so recently. You don't need to boot into the old system.



That's inconvenient and unexpected that Forgot Password does not work. Maybe e-mail the sales office ( https://www.proxmox.com/en/about/contact ), as they handle issues with subscriptions? Or use the contact form: https://shop.proxmox.com/submitticket.php?step=2&deptid=1

Thank you - it seems that "my.proxmox.com" has a different login from "shop.proxmox.com" which also has a different login than "forum.proxmox.com" - I had to register at "my" to get access (which brought its own challenges with our email link verification software, but... oh well). Subscriptions are handled in "shop" anyway, which I found when I was trying to send a support ticket to sales, as you suggested.

I think I'm all sorted out on the access, password and subscription side. Now, hopefully support can help me get this system back bootable, or at least migrated. 20+ years of using and administering Linux (not full time, though) and I still encounter new challenges regularly.

 

[Chris]

Thank you - it seems that "my.proxmox.com" has a different login from "shop.proxmox.com" which also has a different login than "forum.proxmox.com" - I had to register at "my" to get access (which brought its own challenges with our email link verification software, but... oh well). Subscriptions are handled in "shop" anyway, which I found when I was trying to send a support ticket to sales, as you suggested.

I think I'm all sorted out on the access, password and subscription side. Now, hopefully support can help me get this system back bootable, or at least migrated. 20+ years of using and administering Linux (not full time, though) and I still encounter new challenges regularly.

Hi,
the easiest way to restore a failed node would be by restoring the guests from backup. https://pve.proxmox.com/pve-docs/pve-admin-guide.html#_backup_and_restore
I assume this is a single node? If that is the case, you might be able to recover the configs by following the procedure described here https://pve.proxmox.com/pve-docs/pve-admin-guide.html#_recovery
Your VM/CT data have to be either moved to the new storage afterwards or you have to configure the storage with the existing data and adapt the guest configs accordingly.

 

[mobrien118]

I ended up just re-installing Proxmox on the 2nd hard drive (swapped as new boot media), creating "new" VMS and then using DD to bitwise copy the LVM partitions from the existing VMs over to the new ones. Then I wiped the old disk, formatted it for LVM and extended the existing VG to include it (yes, I know it is not fault-tolerant and is single POF).

I'm glad I did because the PCI passthrough support is MUCH MUCH MUCH easier in 8.2 than it was in 8.1.