[HELP] Issue after an upgrade from v8.1.10 on a HP Gen8 micro server

[Fryl]

Hi,

It was a good morning and then i have decided to update/upgrade my proxmox ... bad idea it seems...

I was currently on Proxmox v8.1.10 on HP Proliant Gen8 microserver (16GB, Intel processor 1265v2)
Just click to Refresh and next Upgrade button from menu Updates on my pve1 node

Everything was working well (before and even on upgrading process)
Advice to reboot
=> ok I reboot

Now, Proxmox is not booting anymore...
I get an error message that i don't understand

To help please find in attachment 2 screen captures:
- HPE ILO screen capture
- old webpage (connection error since upgrade as proxmox in not booting) that was keep into my browser

Could you please help me to solve this issue ?
Thanks a lot for your helps and your answers.

 

[Fryl]

As Gen8 micro server is not able to boot directly on first disk (small SSD), i have an usb key plugged with Grub to chain the boot to SSD partition (where proxmox setup is done)
It seems that this usb key grub is still working when i see this screen capture on ILO

Where can i search to solve this proxmox booting?

 

[carles89]

Are you able to boot with an older kernel?

If so, you may be affected with one of the breaking changes in PVE 8.2. You can check them here: https://pve.proxmox.com/wiki/Roadmap#Proxmox_VE_8.2

 

[Fryl]

Are you able to boot with an older kernel?

If so, you may be affected with one of the breaking changes in PVE 8.2. You can check them here: https://pve.proxmox.com/wiki/Roadmap#Proxmox_VE_8.2

Hi @carles89
Thank you for your message
How can I do that please ?

 

[esi_y]

You are not booting the updated kernel / initramfs, obviously. You need to select another boot option, i.e. you need to select Advanced options on that "blue" (GRUB) screen and select older item there. You should really be watching the iLO output all the time and capture that for troubleshooting.

EDIT: If you manage to boot with previous kernel, just make it permanent for future boots [1].

You can then try again (unpin) when newer kernels come out, but the current one was testing quality at best.

[1] https://pve.proxmox.com/wiki/Host_Bootloader#sysboot_kernel_pin

 

[esi_y]

i have an usb key plugged with Grub to chain the boot to SSD partition (where proxmox setup is done)

The other option is that you are in fact booting something old for the updated system - how did you make this setup, as stock PVE install does not let you specify custom boot/ESP location?

 

[carles89]

How can I do that please ?

You should be able to select an older kernel going to Advanced Options on the Grub blue screen,as esi_y said:

You need to select another boot option, i.e. you need to select Advanced options on that "blue" (GRUB) screen and select older item there.

 

[Fryl]

Thanks a lot @esi_y and @carles89
You help me and it solves temporary situation
Many Many Many thanks to you

As you wrote, i have selected in grub menu the advanced option -> and previous kernel -> proxmox has booted (v8.2.7 vs 8.1.10 before) -> now i have access again to my node !!!

Really thank you, i was so worried about solutions

 

[Fryl]

The other option is that you are in fact booting something old for the updated system - how did you make this setup, as stock PVE install does not let you specify custom boot/ESP location?

Sorry i am not sure to well understand your question

When i have to setup proxmox (and before install drives inc ssd inside front bay for boot) i have to mandatory create a grub boot usb key (seems to be the only solution to boot on front bay on Gen8 micro server)
For sure if you have another way to boot on front bay ssd on gen8 microserver, i will be very interesting and happy to apply it (avoiding usb key plugged, too sensible non trusting hardware...)

 

[Fryl]

Sorry to disturb you again
Even if proxmox has booted right, it seems that i get new errors with critical warnings on ILO view about proxmox instance.... don't know why

 

[Fryl]

sdd seems to be the usb key ... my bad :-(

 

[esi_y]

Sorry i am not sure to well understand your question

I understand the setup, I know there is 4 HDDs + ODD + internal USB + SD card. I remember ODD boot was NOT possible in AHCI mode (HW RAID off).

But the internal USB was possible and so was SD card (probably the most popular option).

When i have to setup proxmox (and before install drives inc ssd inside front bay for boot) i have to mandatory create a grub boot usb key (seems to be the only solution to boot on front bay on Gen8 micro server)

How did you do this?

For sure if you have another way to boot on front bay ssd on gen8 microserver, i will be very interesting and happy to apply it (avoiding usb key plugged, too sensible non trusting hardware...)

I think booting of a BAY is just fine, it's the ODD that has the issue. Nevertheless lots of people used the SD card (for boot partition).

sdd seems to be the usb key ... my bad :-(

Get a good small ~ 2G industrial SD card, it's never really written to (unless you use it). It will last forever.

Also, to keep my sanity, I would install on top of Debian.

 

[Fryl]

I understand the setup, I know there is 4 HDDs + ODD + internal USB + SD card. I remember ODD boot was NOT possible in AHCI mode (HW RAID off).

no SD card, just an usb key plugged on mainboard with grub

But the internal USB was possible and so was SD card (probably the most popular option).

Sorry, i wasn't aware about this SD card alternative
I just tried to use a "good quality" usb key with 16GB -> Sandisk UltraFit was one choice (it was in 2021 or 22 if i remember well ... (fish memory inside )

How did you do this?

i have created usb key content with a tutorial (seems not anymore available) => grub install and standard proxmox setup

I think booting of a BAY is just fine, it's the ODD that has the issue. Nevertheless lots of people used the SD card (for boot partition).



Get a good small ~ 2G industrial SD card, it's never really written to (unless you use it). It will last forever.

Could you please give me a kind of reference i could search for ?

Also, to keep my sanity, I would install on top of Debian.

I am not so experimented to being able to do that ... :-(


I found a post on error i get
seems to be a known issue but i have to understand solution and how to apply it...
https://forum.proxmox.com/threads/updated-to-8-2-dma-error.145907/

edit: seems not the same errors

 

[esi_y]

no SD card, just an usb key plugged on mainboard with grub

Sorry, i wasn't aware about this SD card alternative

It's right next to it. In fact, let me link quickstart guide:
https://www.hpe.com/psnow/doc/c04123182

And user one:
https://www.hpe.com/psnow/doc/c03787037

I just tried to use a "good quality" usb key with 16GB -> Sandisk UltraFit was one choice

I am not saying it went bad, if it's just EFI partition, it is not really shredded. The main reason I would use industrial SD (it's just a bit more expensive, mostly has better specs for environment, i.e. temperature) is that it should fare better in comparison, also I do not need it to be any big size, so prices are low.

i have created usb key content with a tutorial (seems not anymore available) => grub install and standard proxmox setup

It is good to keep offline copies of whatever you followed to know how it might interact with e.g. future changes that Proxmox put into their own tooling.

Could you please give me a kind of reference i could search for ?

I could try to blindly search ... and I guess I found your issue:

"There is not a Linux driver for the embedded SATA software RAID controller, don't enable the RAID function If you really need RAID you can install a P212 Smart Array controller."

I cannot find it myself quickly in the guides, but you essentially would need to have AHCI mode for your drives:
https://www.reddit.com/r/homelab/comments/yu60s1/hp_microserver_gen8_b120i_ahci_mode/

Also ... the quickstart points out that:

NOTE: HDD bays 1 and 2 support 6.0Gb/s SATA. HDD bays 3 and 4 support 3.0Gb/s SATA.

I believe you will be better off with HDDs in the 3/4 bay and keep 1/2 for SSDs.

I am not so experimented to being able to do that ... :-(

You have virtualisation solution, why not e.g. install Debian into a VM, then install Proxmox on it and see? There really isn't anything special about installing stock Debian, you have more flexibility when choosing partitioning (e.g. you can choose the USB/SD card for /boot). And then install of PVE on top is supported and well documented, it's ~5 lines:

https://pve.proxmox.com/wiki/Install_Proxmox_VE_on_Debian_12_Bookworm

I found a post on error i get
seems to be a known issue but i have to understand solution and how to apply it...
https://forum.proxmox.com/threads/updated-to-8-2-dma-error.145907/

edit: seems not the same errors

Did you have any power outage or such that the EFI stored there and its filesystem might have gotten corrupted?

 

[Fryl]

Hi @esi_y

Thanks a lot for all these elements, advices and your experienced / best practices sharing
I'm focusing on backup everything as the server behavior is a little strange ... I hope that i will be in situation next to use all your very interesting elements & all links to improve this issue context

I think that usb key has perhaps suffered when i have to reboot and next shutdown server => i will search for a SD card (right with your advices) and convert this boot on SD vs current usb key
=> you're right : i should have to save this tutorial in pdf or other to renew/make change now in accordance with it => shame on me

I will read too your elements about debian to understand and identify if i will be able to do that


Many Many Many Many Thanks to you !!!!

 

[esi_y]

No worries, take your time!

Also, I quickly checked for currently sold industrial microSDs, e.g.:
https://www.kingston.com/en/memory-cards/industrial-grade-microsd-uhs-i-u3?partnum=sdcit2/8gbsp

It does not have to be Kingston, but it is nice if has "TLC in pSLC mode", whatever brand you choose. They seem to start at 8GB nowadays, which is more than enough for ESP partition and inexpensive.

BTW It is also possible to use either SSD on that ODD SATA (for CD ROM, just there was special adapter for power cable needed) or even NVMe SSD, but you would need to get PCIe adapter card (the correct one for NVMe, has to come with low-profile bracket, also inexpensive), e.g.:
https://akasa.co.uk/search.php?seed=AK-PCCM2P-01

But I suppose you do not really need it for now as your 4 bays are not fully used yet.