Proxmox 4.2 Grub Rescue after install

[Valhera]

Hi Guys,

I recently installed version 4.2 on a DL360G7 but after multiple installation attempts after reboot the system always ends up in Grub Rescue mode.

I know that this is related to Grub expecting to boot from HD0 and i installed on a different hard disk to hda and I know from reviewing the forum that the Proxmox installer only installs the boot loader on hda.

I have tried installing on the internal SD card, various USB ports and a hard disk but due to factors such as bios, control configuration and drive spin up time I have not been able to lock down hda.

I have not had this issue with vmware or virtuozzo so its confusing why the installer doesn't install the boot loader and configure grub appropriately to boot from the drive its installed on.

Any help you can provide would be appreciated.

 

[RobFantini]

Hi Guys,

I recently installed version 4.2 on a DL360G7 but after multiple installation attempts after reboot the system always ends up in Grub Rescue mode.

I know that this is related to Grub expecting to boot from HD0 and i installed on a different hard disk to hda and I know from reviewing the forum that the Proxmox installer only installs the boot loader on hda.

I have tried installing on the internal SD card, various USB ports and a hard disk but due to factors such as bios, control configuration and drive spin up time I have not been able to lock down hda.

I have not had this issue with vmware or virtuozzo so its confusing why the installer doesn't install the boot loader and configure grub appropriately to boot from the drive its installed on.

Any help you can provide would be appreciated.

do you have multiple disks ?

check bios to make sure the install disk gets booted from.

sometimes in the past I'd leave only the install to disk powered up for the install, then added rest after making sure the ssd booted.

 

[Valhera]

Yes that correct 8 disks in total.

Not possible to set the boot disk in bios as we are running a LSI controller, I have tried setting this controller to the boot device with lots of iterations but no luck.

I will try what you have recommended and let you know how I go, sounds like it will work.

Thank you very much for your advice.

 

[RobFantini]

for operating system we usually use 2 on board sata zfs raid-1 . if you do so then set then 1ST and 2ND boot disk priority to the 2 boot drives. good luck.

 

[Valhera]

Hi Guys,

I removed all drives and Proxmox installed as recommended but on reboot the system went straight into Grub Rescue mode.

I am sure this is happening as the SD card that I am installing Proxmox onto comes up as sdh as opposed to sda that Grub is expecting.

I think my only way around this issue is to modify the grub configuration file.

Such a simple thing, Proxmox should be able to be installed on an SD card or a USB or for that matter a drive of our choosing with out the pain of caring about the drive names.

 

[fabian]

what does "ls" in the grub rescue shell output? this should show at least the hard disk that /boot is residing on, with "ls (hdx,y)/" (replace x and y with numbers from the previous "ls" output) you should be able to list the content to check what it is

 

[RobFantini]

Hi Guys,

I removed all drives and Proxmox installed as recommended but on reboot the system went straight into Grub Rescue mode.

I am sure this is happening as the SD card that I am installing Proxmox onto comes up as sdh as opposed to sda that Grub is expecting.

I think my only way around this issue is to modify the grub configuration file.

Such a simple thing, Proxmox should be able to be installed on an SD card or a USB or for that matter a drive of our choosing with out the pain of caring about the drive names.

try remove all but one drive, then install to the drive? in the past this has been a bios/hba issue for me. sometimes I have not been able to keep the lsi card and bios to assign the same same hard drive to 1ST boot priority.

also are us using IT mode with the card?

 

[Valhera]

Hi Fabian,

I have attached a picture for the ls command

Please let me know what you think.

 

[Valhera]

try remove all but one drive, then install to the drive? in the past this has been a bios/hba issue for me. sometimes I have not been able to keep the lsi card and bios to assign the same same hard drive to 1ST boot priority.

also are us using IT mode with the card?

Hi Rob, just to let you know that I removed all drives accept one and did the install, the installer picked the drive up as sda during install. Yes I am running the controller in JBOD mode which I believe is the same and IT mode. Also just to let you know the LSI controller is not causing my issue at this point in time as I am booting from an SD card which is internal to the HP Server and no hard disks are currently connected to the LSI card, from what I have been told by a colleague the SD card is part of the USB bus which is why the OS see's it as hdh

 

[RobFantini]

is that a raid card or hba? in unsure send the model. I know it is not the cause of the issue. for zfs a hba is better then a raid card. I have a bin of lsi raid cards to sell sometime..

regarding seeing the ssd as usb, I have old ssd's that are seen as usb by some motherboards. I never tracked down why, instead used a diff ssd .

good luck!

 

[fabian]

Hi Rob, just to let you know that I removed all drives accept one and did the install, the installer picked the drive up as sda during install. Yes I am running the controller in JBOD mode which I believe is the same and IT mode. Also just to let you know the LSI controller is not causing my issue at this point in time as I am booting from an SD card which is internal to the HP Server and no hard disks are currently connected to the LSI card, from what I have been told by a colleague the SD card is part of the USB bus which is why the OS see's it as hdh

the installer sees disks differently then boot in early grub, because in the installer we already have a fully booted linux kernel. if you only have the SD card connected, install, and then reboot (still with only the sd card connected) and you still end up in the rescue shell, please collect the output of "set" and "ls" in the grub rescue shell.

sometimes booting from such special devices requires some BIOS tweaking, sometimes it is simply not possible at all (e.g., many laptops cannot boot from their internal card readers).

as a side note, installing proxmox's root fs on an sd card will probably not work as well as you think - at least not without a lot of manual tweaking. the write performance is usually horrible, and stuff like the statistics collected by pvestatd and log files tend to be already too much.

 

[Valhera]

the installer sees disks differently then boot in early grub, because in the installer we already have a fully booted linux kernel. if you only have the SD card connected, install, and then reboot (still with only the sd card connected) and you still end up in the rescue shell, please collect the output of "set" and "ls" in the grub rescue shell.

sometimes booting from such special devices requires some BIOS tweaking, sometimes it is simply not possible at all (e.g., many laptops cannot boot from their internal card readers).

as a side note, installing proxmox's root fs on an sd card will probably not work as well as you think - at least not without a lot of manual tweaking. the write performance is usually horrible, and stuff like the statistics collected by pvestatd and log files tend to be already too much.

Hi Fabian,

Thanks for the feedback, I assumed that this was the issue. Other virtualisation products such as vmware and virtuozzo have absolutely no issue dealing with installing on an SD card or USB stick as I have installed both products on the exact same server as we have been doing testing of each product, is this something that will be fixed in the future. I find it very strange that there are hardly any posts about this issue, I would have to assume that the majority of your install based is not on enterprise hardware? I also provide a "set" and "ls" outputs in my posts, I attached screen shots.

 

[fabian]

Proxmox does not have a problem with installing on sd cards or usb sticks - it's your mainboard/BIOS that has a problem booting from them. This works on most reasonable systems - but unfortunately not all, and even "enterprise" server hardware is no guarantee for sane boot options.

If you do the steps I described in my last post, you can at least tell for sure whether this is the cause of the problem (i.e., whether grub sees the card or not).

 

[Valhera]

Hi Fabian,

Whats clear is that when Proxmox installation CD it identifying the drives differently as opposed to when the full OS boots, this is clearly an issue with the installation process, is this something that Proxmox are going to fix.

I have provided in previous replies in this post attachments with screen shots of "ls" and s"set" outputs from Grub Rescue.

I have been working on this issue now on and off for 4 weeks and its taking far to long to resolve, I have tried many different combinations of bios setting to no avail, playing with bios settings is also just a stop gap, Proxmox need to fix this issue.

As I mentioned before VMWare and Virtuoso have no issue installing on either the internal SD card or USB drive on this server so I'm sorry its not the mother board, in fact I am 100% sure its not the motherboard as I have 5 other servers in the same rack of the exact same model and hardware configuration and every single server has the same issue with Proxmox but absolutely no issue with VMWare or Virtuozzo.

 

[Valhera]

I have attached the outputs again, please note that they are from a fresh installation

 

[fabian]

Hi Fabian,

Whats clear is that when Proxmox installation CD it identifying the drives differently as opposed to when the full OS boots, this is clearly an issue with the installation process, is this something that Proxmox are going to fix.

I already explained to you that if there is a difference, it is between installation environment and the bootloader/BIOS, not between installation environment and installed Proxmox (as the latter are both Linux kernel based environments).

I have provided in previous replies in this post attachments with screen shots of "ls" and s"set" outputs from Grub Rescue.

it was not clear whether those were with all the disks attached or just the SD card (which was what I specifically asked for).

I have been working on this issue now on and off for 4 weeks and its taking far to long to resolve, I have tried many different combinations of bios setting to no avail, playing with bios settings is also just a stop gap, Proxmox need to fix this issue.

Proxmox does not need to do anything, especially not fix problems that are most likely not caused by Proxmox. A polite tone goes a long way when communicating, btw.

As I mentioned before VMWare and Virtuoso have no issue installing on either the internal SD card or USB drive on this server so I'm sorry its not the mother board, in fact I am 100% sure its not the motherboard as I have 5 other servers in the same rack of the exact same model and hardware configuration and every single server has the same issue with Proxmox but absolutely no issue with VMWare or Virtuozzo.

I did not say that it is a hardware issue as in "this specific mainboard is broken", I said it might be a BIOS/mainboard issue as in "many motherboards simply do not support booting from such devices or need some options changed or a BIOS update to support it". Do you know what VMWare and Virtuoso use as a bootloader? Do you know whether they really install the bootloader on the SD card? Maybe the issue is something else, and your mainboard does support booting from it even with Grub, but so far you did not give any information that could help resolving it.

For example, you consistently typed the LVM volume part wrong in your attempts in the grub rescue shell. Please try:
"ls (lvm/pve-root)"

which should output the information about your root LV, including that it is an ext* filesystem. You should also be able to list the files on it, including /boot:

"ls (lvm/pve-root)/boot"

If any of this fails, feel free to post the error messages here - but please try to read and understand everything I posted before.

 

[Valhera]

Out of curiosity is the premium support similar to this or do your team members actually get in remote support sessions to troubleshoot?

Because this is not progressing, I think its also important that one of your team take a look at the system because you might find that you installer actually does have an issue and this could help quite a few people.

There was no tone in my post, just frustration as i have been trying for some time to get these boxes online.

Thanks fin advance for your reply

 

[fabian]

As noted on http://www.proxmox.com/en/proxmox-ve/pricing , the two highest support tiers (premium and standard) include remote troubleshooting via SSH, and all tiers except community include direct support via our suppor ticketing system.

Did you try the commands I posted?

 

[Valhera]

No not at this stage as I am going through the process of following the instructions to manually install Proxmox on Debian, if I don't have any luck with this method I will reinstall with the CD and come back to you with the output.

 

[Valhera]

Hi Fabian,

I resolve the my issue by installing Debian Jessie and when going through the installation and instead of installing Grub I installed Lilo and its now working perfectly, happily booting off the USB key.

I would recommend that you allow the selection of Lilo as a boot loader during the installation of your CD.

Thanks for your help

Regards

Russell