[SOLVED] Server stuck at boot after ProxMox VE Update

[csandhmech]

Earlier this year, I purchased a refurbished Dell R720xd server so I could run a few VMs. I put ProxMox VE on it as the OS and it's been up and running without flaw ever since. I have four VMs (2 Linux, 1 Windows 10, and TrueNas) on it. Earlier this week, I finally decided to check for updates via the web GUI. It downloaded and installed the updates, then needed a reboot. After I rebooted the server is when I ran into the hanging issue. My R720xd gets to the firmware initializing screen, then blips a message up too fast to read, then the screen goes blank and nothing else happens. I can still get to the iDRAC controller just fine but it isn't loading proxmox.

Info on the R720xd:
2 - Intel Xeon CPUs E5-2697 v2
8 - 8 DDR-3 32GB PC3L-12800L RAM (It is 7 now as I discovered one of the sticks went bad (guessing it was limping along until the reboot))
1 - Dell Broadcom 5720 Quad-Port Gig Network Daughter card
7 - 8TB SAS drives
2 pools, 1 for the ProxMox OS and the VMs and their storage, and a ~35TB pool for TrueNas
If any other information on the server would be helpful, let me know.

Things I've tried so far:
-Rebooting
-Powering down completely
-Power down, pull the ethernet cables, pull the power cords, press and hold the power button for 20 seconds, plug power back in, let it sit for a few minutes. power back up
-I can still get into the BIOS, Lifecycle, and iDRAC on boot up, so I ran a full hardware test (that is where I discovered a ram stick went bad), no other issues were detected
-I read somewhere about someone else having a similar issue and they reseated all the drives, so I tried that but it didn't help
-I recorded the message that pops up too fast to read on the screen and played it back. It said that it was disconnecting the UEFI drivers. I read somewhere that meant the system was trying to boot from a USB stick. That is how I originally put ProxMox on the server, so I thought that could be it. I checked in the BIOS settings and it is set to boot from the hard drive.
-I read somewhere that there was a kernel missing in ProxMox 8 and you had to go through changing some files or something. The problem with that is I have no idea how to get to anything to do that. With ProxMox being the OS and it's not loading, I can't get to anything like that.
-When the server is "stuck" I can still see all four ports on the ethernet card are connected to my network. I can see their MAC addresses but not the VMs themselves. (Passthrough issue?)

It could very well be something else and it wasn't the update that broke it, but the reboot instead. I don't know. Is it possible to update ProxMox from a USB drive in the server without erasing everything I have on the drives? What else can I check for on the ProxMox end to verify if it was that or another server issue? Keep in mind, the hardware test claims everything is ok. I had rebooted the server with the previous version of ProxMox quite a few times without any issues. It always just came back up, spun the VMs up, and away I went.

Help would be greatly appreciated.

 

[LnxBil]

Have you tried booting the old kernel or is that what you meant by "rebooted the server with the previous version of PVE"?

 

[csandhmech]

Have you tried booting the old kernel or is that what you meant by "rebooted the server with the previous version of PVE"?

I have not tried booting it with the old kernel. I meant with the older version of ProxMox that was on it before the update. With the first version of ProxMox I had on the server, the Windows VM would occasionally hang and the only fix I had for it was to reboot the server. It always came back up. Is it possible to go back to that version in my situation without deleting everything?

 

[LnxBil]

Is it possible to go back to that version in my situation without deleting everything?

Booting the older kernel is simple: Just choose it in the boot menu.

Installing will always format the disk you're installing to, so it will not work.

 

[csandhmech]

Booting the older kernel is simple: Just choose it in the boot menu.

Installing will always format the disk you're installing to, so it will not work.

I really appreciate your assistance but please forgive my ignorance. I'm not seeing a way to do that in the boot menu of my R720xd. Booting from C drive seems to be the only option. I did just remember, somewhere around here I have a drive dock. I could yank the drive out of the server where ProxMox lives, put it in the dock, then connect that to my PC and possibly do something that way.

 

[LnxBil]

Normally you boot from the disk and the first thing is a boot menu where you can choose your kernel to boot. You could try there "the other" kernel, that is not per default selected and booted.

 

[csandhmech]

Normally you boot from the disk and the first thing is a boot menu where you can choose your kernel to boot. You could try there "the other" kernel, that is not per default selected and booted.

There is no boot menu other than the servers bios boot menu. Unfortunately, ProxMox isn't loading at all. It seems to be stuck at the point where ProxMox would normally load up. The server would display "Initializing firmware something or other" then a bar would fill up on the bottom of the screen, then ProxMox would load up and I could then browse into ProxMox. I'll see if I can't find that drive dock and see if I try to get to ProxMox that way.

 

[BaridH]

Its is a huge bug of Proxmox
Everyone trying to update proxmox endup with unbootable system today because of grub issue after update.

what is the update of
update-grub
proxmox-boot-tool status

 

[csandhmech]

Its is a huge bug of Proxmox
Everyone trying to update proxmox endup with unbootable system today because of grub issue after update.

what is the update of
update-grub
proxmox-boot-tool status

I kind of gathered that I'm not the only one with this issue from reading other forums. I hope there is a fix soon. I'm kind of thinking that maybe I could copy the data off of my drives of my server, blow proxmox out, put on a Linux operating system, and use the regular proxmox instead of the OS version. Then at least I could get in to the file system to change things, rather then an expensive paperweight.

If you're asking me what the update of update-grub proxmox-boot-tool status is, I don't even know what that is. If it's something I should know and could use, please let me know and tell me how to use it. Obviously a command prompt but without proxmox running, I can't imagine that would do much. Hopefully I'm wrong!

 

[LnxBil]

Its is a huge bug of Proxmox
Everyone trying to update proxmox endup with unbootable system today because of grub issue after update.

Really? I haven't seen that. Any pointers?

I hope there is a fix soon.

Even if there is, you're already in a bricked state. Unless the underlying problem is known, you cannot fix it. Proxmox VE (please write it that way, that's the product name and how it is spelled) has a built-in rescue system from the iso to boot up and fix things manually.

 

[BaridH]

I writed the detailed thread with logs but it was not approved and deleted.

 

[csandhmech]

Really? I haven't seen that. Any pointers?


Even if there is, you're already in a bricked state. Unless the underlying problem is known, you cannot fix it. Proxmox VE (please write it that way, that's the product name and how it is spelled) has a built-in rescue system from the iso to boot up and fix things manually.

So to use the built in rescue tool of ProxMox, I am guessing (since I don't have the .iso anymore) redownload the Installer ISO (v8.0), throw it on a thumb drive, plug it into the server, set up the server to boot from the USB from the BIOS settings, and I should be able to see the repair tool on the thumb drive that way? That would be a pretty slick repair tool system if I have all that correct. I just downloaded the ISO file now and will get it on a thumb drive.

I did find my drive dock but naturally, that doesn't work with the drives I have in my server. I've got SAS drives in the server. Didn't even think of that! So fixing it that way is out. Hopefully the repair tool works. Thanks for the suggestion.

 

[csandhmech]

Finally fixed it!

"Booting the older kernel is simple: Just choose it in the boot menu."
LnxBil, I'm thinking you were referencing if I pressed F11 on server bootup, not F2 (obviously different for whichever system/server brand you're using). I completely forgot about F11! I must have used that to install it initially. I'm getting better at forgetting and worse at everything else the older I get. In the F11 menu, was the menu you mentioned. I am currently up and running on version 7.4-16. It works, so I think I'll leave it at that. I don't want to go through all that. From what I read on other forums, people were saying not to use the web update way of updating. That seemed to be where they ran into issues. Maybe some day, I'll check into updating it a different way.

So you did fix it, I was just too dumb to realize what you were talking about! I hope this helps somebody else. Thank you very much.

 

[LnxBil]

LnxBil, I'm thinking you were referencing if I pressed F11 on server bootup, not F2 (obviously different for whichever system/server brand you're using). I completely forgot about F11! I must have used that to install it initially.

No, I did not meant the boot menu of the server (I would have said so), I meant the boot menu that is common in BIOS and EFI boots with PVE. There is still (unless deactivated manually) a menu with entries (including a rescue mode in older installs) - at least in all PVE installation I've ever seen.

In a just installed UEFI VM and updated it. It looks like this:


That's the menu I meant.

Thank you very much.

No problem. Glad it worked out!