Proper way to install Proxmox in UEFI mode.

[M_D]

I have a Thinkstation P500 which I am using as a PVE server. A week or so ago I installed PVE on this server, from an ISO loaded onto a USB, and all worked well. I have been testing a few VM's etc.

Then, a day or two ago, it dawned on me for some reason (when I was inside the firmware config setting up a SAS controller) that the system was booting Legacy First (E.G using BIOS, not UEFI). I discovered that indeed I had installed Proxmox in BIOS mode, without realizing.

I did some research, and it seems this is really not what you want, due to the potential for issues with, it seems, espeicially ZFS. My goal is to run my two primary SSD's in a RAID1 ZFS (mirror) config, for the Proxmox OS itself, and some VM OS disks. To be fair, as far as I know, this was working fine on BIOS mode, but I had not tried any of the advanced ZFS features. Since I had done very little PVE config, I decided it was a good time to backups the VM's I had onto an external disk, and reinstall PVE, this time in UEFI mode, to prevent any future issues.

However, I have had no end of issues. I created the normal ISO installer on a USB using Rufus, again. When I did it the 2nd time, Rufus did something odd (due to previous cached settings from the last bootable USB I made, which was MS-DOS), and insisted on using BIOS (UEFI + CSM) mode. I tried a 3rd time, after deleting the rufus.ini, and then it, as expected, showed UEFI in the Rufus program.

That is not too relevant though. Pretty sure I have a valid UEFI-bootable USB installer. I attach it to the server, and configure the BIOS as thus:

- Boot Mode: UEFI Only.
- CSM: Enabled.

I tried disabling CSM, but it said the video card was in Legacy mode, and this must be changed to UEFI mode before CSM could be disabled. I tried changing the video OpRom to UEFI, but as I expected, I lost all video output and had to reset the CMOS. So I am stuck with CSM on, but I would think it would be OK with boot mode set to UEFI only.

However, when booting from the USB, I get a 1962 No OS Found error. Not helpful. When using BIOS mode, PVE installer runs fine, first time. I am really needing some advice. Mr GPT seems to think it might be a good idea to install in BIOS mode, and somehow convert to UEFI mode afterwards. But I would like some input on the "right" way to do it, as this must be a common requirement!

Thanks.

 

[LnxBil]

I did some research, and it seems this is really not what you want, due to the potential for issues with, it seems, espeicially ZFS.

I don't know what you mean. ZFS runs fine in any setup. Why change something if it works for you? Don't believe in stuff writte online. Most people have no clue what they're talking about.

But I would like some input on the "right" way to do it, as this must be a common requirement!

There is practically no difference in installing PVE on BIOS or UEFI from the installer perspective. Booting is the same, install dialog is the same and in the end, the only difference is layout on the disk.
UEFI itself is just the specification and as there are multiple BIOS vendors available, the same is true for the UEFI implementation. There are ones that work flawlessly and there are others. I don't know about the Thinkstation P500, so I cannot be of help there.

Does the Thinkstation have a proper management interface in which you can just mount an iso image to be bootet from? With this step, you can test if the problem lies in the usb boot disk creation or not.

 

[M_D]

I don't know what you mean. ZFS runs fine in any setup. Why change something if it works for you? Don't believe in stuff writte online. Most people have no clue what they're talking about.

Fair enough. Maybe I am unnecessarily concerned. Thanks for pointing out that BIOS mode is not in itself a problem.

Here is an example of something I read which made me think installing in BIOS mode might turn out to be a foot-gun in future - from: https://forum.proxmox.com/threads/best-setup-for-zfs.60605/

Install ProxmoxVE v6 or higher in UEFI mode. This will make it use systemd-boot as bootloader and store the kernel image on the EFI service partition (ESP). If you install it in BIOS mode GRUB will be used as bootloader and the boot volume will be on the ZPool. This will limit you in terms of features you can use in the zpool (for example encryption) because GRUB only supports a very limited subset of ZFS features.

UEFI itself is just the specification and as there are multiple BIOS vendors available, the same is true for the UEFI implementation. There are ones that work flawlessly and there are others. I don't know about the Thinkstation P500, so I cannot be of help there.

OK. I think it may be an issue with the firmware not recognising the USB installer in UEFI mode?

Does the Thinkstation have a proper management interface in which you can just mount an iso image to be bootet from? With this step, you can test if the problem lies in the usb boot disk creation or not.

Unsure. Haven't tried that yet.


Would appreciate any other input: Either pointers on why my system won't boot UEFI, or any experiences running BIOS mode - issues? No issues?

 

[M_D]

Of course I also need to be 100% sure that with whatever boot mode I end up using, the required boot files are present on both disks in the ZFS mirror, so if one fails, not only is the data preserved, but the system will still boot, whilst I replace the failed disk...

 

[IsThisThingOn]

IMHO CSM creates more problems than it solves.
AFAIK CSM only helps with booting in UEFI but not being able to Secure boot.
Since that is not the case for Proxmox, this does not apply.

So I would set it to UEFI only and CSM disabled to get a UEFI and secure boot enabled Proxmox install.

Your GPU problem is strange to me. I would not even know, what you could do with OpROM on a GPU. Either way, I would disable all OpROM (also for NIC) and of course disable all PXE boot stuff. That way your System might boot even faster, since it is not needlessly initiating stuff that it is not using.

 

[M_D]

Your GPU problem is strange to me. I would not even know, what you could do with OpROM on a GPU. Either way, I would disable all OpROM (also for NIC) and of course disable all PXE boot stuff. That way your System might boot even faster, since it is not needlessly initiating stuff that it is not using.

OK. I will try this again when I have physical access to the box again. The last I tried, I put Video OpROM to UEFI (it was legacy), and lost all video output - had to reset it back to gain access even to the firmware configuration.

I am not too familiar with OpROM's, I can't remember if there is a "disable" option available...

 

[IsThisThingOn]

On my supermicro I can disable all OpROMs, but I don't have a GPU, so unfortunately I also don't know.

Maybe try to disable fast boot, this is known to cause issues with Linux, especially the MSI implementation of it.

 

[M_D]

OK - I will experiment with disabling fast boot, as well as putting OpROMs to disabled, all though I suspect in doing so I will loose access to my firmware configuration, because my server doesn't have an integrated GPU. (Does have a serial port, but I don't think it can output BIOS screens to the serial port...

 

[IsThisThingOn]

AFAIK OpROM has nothing to do with your GPU working after the BIOS initialized.
AFAIK OpROM is used in NIC to configure vNICs or in storage controllers to configure RAID, spinup delay, firmware updates and stuff.
You don't need OpROM for simple GPU video output.

 

[celemine1gig]

You are using a dedicated GPU, right? If yes, which one exactly? If it has no UEFI spec conform GOP-support, then you simply get no Video output in native UEFI or CSM-mode with Video output set to "UEFI".

 

[M_D]

AFAIK OpROM has nothing to do with your GPU working after the BIOS initialized.
AFAIK OpROM is used in NIC to configure vNICs or in storage controllers to configure RAID, spinup delay, firmware updates and stuff.
You don't need OpROM for simple GPU video output.

OK, that's interesting. This is the first time I have even heard of OpROM to be fair.

You are using a dedicated GPU, right? If yes, which one exactly? If it has no UEFI spec conform GOP-support, then you simply get no Video output in native UEFI or CSM-mode with Video output set to "Legacy".

Yes, dedicated GPU, no integrated CPU in this server. Its a NVIDIA Quadro, will have to check on the exact model. Thing is, the video output is set to legacy, and when trying to boot in UEFI mode, I still get the BIOS video output. When I set the Video OpROM to 'UEFI', I get no video output at all.

I haven't tried with the OpROM disabled entirely yet. I may also be able to try some other combinations, like disable the OpROM, and see if I can also disable CSM before rebooting - or whether I will just get the message about video being in legacy mode when trying to disable the CSM...

Anyway I wouldn't have thought any of this would be causing the system not to recognise Proxmox as a UEFI bootable disk...

 

[celemine1gig]

Your missing GPU display output is completely untelated to the systems' ability to perform a full UEFI boot.
And if you set your system to boot Legacy-first, then why should it boot Proxmox in UEFI mode?
The Proxmox installer can perform a Legacy boot. So it simply does and installs that way.
You can even boot via UEFI, while the Video output mode is set to Legacy (CSM).

You should definitely try to understand some basics, before performing the next changes.

 

[IsThisThingOn]

You are using a dedicated GPU, right? If yes, which one exactly? If it has no UEFI spec conform GOP-support, then you simply get no Video output in native UEFI or CSM-mode with Video output set to "Legacy".

I am using a 3060 on my gaming rig that does not even have the OpROM option in BIOS.
I am not using any GPU on my Supermicro servers but IPMI ASPEED.

You should definitely try to understand some basics, before performing the next changes.

Yeah, lets try one step after another.

- Go into BIOS
- Enable UEFI only
- Disable CSM
- Disable OpROM
- Disable PXE
- Set the boot order to boot from USB first
- Or even better, while you are in the BIOS, try to boot from the USB stick. If the USB stick shows up twice (once as legacy, once as UEFI) make sure to select the UEFI one
- Go trough the install
- Congrats you know have a clean UEFI Proxmox installation and you system should boot reasonably fast.


My guess is that you just repurposed an old Thinkstation P500 as your server. This is totally fine, and I like not creating e-waste.
Still, for stuff like that, a server board with IPMI is so much more convenient.

 

[celemine1gig]

...

- Go into BIOS
- Enable UEFI only
- Disable CSM

And if he does that, he'll once again see nothing from his GPU until a supported OS took over.

 

[IsThisThingOn]

If that is really is the case, yes Then we can take it from there.

 

[M_D]

Your missing GPU display output is completely untelated to the systems' ability to perform a full UEFI boot.
And if you set your system to boot Legacy-first, then why should it boot Proxmox in UEFI mode?
The Proxmox installer can perform a Legacy boot. So it simply does and installs that way.
You can even boot via UEFI, while the Video output mode is set to Legacy (CSM).

You should definitely try to understand some basics, before performing the next changes.

I think I do understand this reasonably OK.
I never said I expected it to boot UEFI when in Legacy First mode . Just when I first test-installed PVE on the server, I forgot to check the boot mode.
I know PVE can boot in Legacy mode, but as I explained earlier, I would rather UEFI if possibly to avoid any Grub-ZFS issues later on.

Yeah, lets try one step after another.

I will try out these steps soon. Although I do suspect, as @celemine1gig said, I will lose video output via GPU until some kind of OS loads - If PVE installs then this is all great, until the next time I need to change a setting in firmware - I suppose I could then 1. Reset CMOS, 2. Go into firmware and configure whatever I need to, and 3. lastly, put the CSM back to off just before saving the changes. But that seems quite a procedure...

Still, for stuff like that, a server board with IPMI is so much more convenient.

Of course. I have the Thinkstation though, so thought I would use it. I believe one can install a BMC card in the Thinkstation, although I had a quick look and they don't seem very readily available. I think it may also have some Intel vPro features, but I have not tested how capable these are.

I will update once I have tried the steps above...

 

[IsThisThingOn]

until the next time I need to change a setting in firmware - I suppose I could then 1. Reset CMOS, 2. Go into firmware and configure whatever I need to, and 3. lastly, put the CSM back to off just before saving the changes. But that seems quite a procedure...

I don't quiet get how you think these things are exclusive to each other?
That should not be the case at all.
You should always be able enter BIOS and you should always be able to get video when your system loads grub, and when your system loaded a OS from grub.

That Thinkstation is EoL and will not get any future BIOS updates, so you should also never loose the current BIOS settings. But yes, if you do, you first have to re-enter your configs. When the BIOS has the default to boot legacy, it won't be able to boot a UEFI Proxmox OS.

 

[M_D]

You should always be able enter BIOS and you should always be able to get video when your system loads grub, and when your system loaded a OS from grub.

Well yes, but: I may be missing something, and this is somewhat theoretical until I do some more testing, but if we are assuming what @celemine1gig stated here:

And if he does that, he'll once again see nothing from his GPU until a supported OS took over.

E.G to disable CSM I have to disable the Legacy Video OpRom, which means I lose video until OS loads etc etc.

I will have no video output from the box until an OS loads - which is well and good, but if I need to get into the BIOS for some reason (config change, new hardware, OS dies), the only way of doing so would be to reset the CMOS - because I have no video from the BIOS itself (without OS loaded), and I can't realistically expect to change the settings back to regain video output from the BIOS when I can't see what I am doing...

 

[M_D]

Just did the following:

- Booted the server to BIOS config.
- Checked Secure Boot - it's "Status = User Mode", and "Secure Boot = Disabled"
- Set "Boot Mode = UEFI Only"
- Set "Quick Boot = Disabled" (It was enabled). I think this mainly just controls the time the logo screen displays for, based on the Help Message? Or not?
- Tried to disable CSM. Got the video policy is in Legacy mode error.
- So, I went into CSM Configuration, and under "Option ROM execution order", set video to "Do Not Launch". Other options were Legacy (What it was on), and UEFI.
- Entered the Primary Boot Sequence menu.
- Made sure "USB KEY: USB KEY 1: SMI USB DISK 1100" was first in the boot order, followed by USB FDD, and the two SATA boot drives a little further down.
- Excluded "Mezz1-PCIe-P1", the NIC, the USB HDD USB Card Reader, and the SATA DVD RAM drive from the boot sequence entirely.

- Hit F10 to "Save configuration and reset".

I then waited, and at POST, it did a beep sequence: Long-Long-Short-Short-Short + plus another small beep at the end, and that was it - no video output, no Proxmox installer. Pretty much the issue I had before.

 

[M_D]

Then, I powered down the server, powered up again and hit F1 (BIOS Key) repeatedly whilst it started up. Same result. Time to reset the CMOS and try again!