[TUTORIAL] bootable NVME install on old hardware made easy with pcie adapter and clover

jieiku

Member
Jan 1, 2020
24
15
8
41
My 1TB nvme is installed into an nvme pcie adapter (these can be bought for like $10)

First you simply install proxmox to the NVME, this is straight forward just like installing to a hard drive, the problem comes in to play when you try booting from the NVME, older bios do not support that, even some that are uefi do not support booting from nvme drives.

Clover makes booting an nvme drive via USB easy: https://github.com/CloverHackyColor/CloverBootloader/releases

I formatted my thumb drive with a FAT16 122M partition
I downloaded the CloverV2-5125.zip and extracted to the thumb drive

next you need to enable the NvmExpressDxe.efi driver, do so by copying the file from the disabled drivers "off" folder to the uefi drivers folder:

Code:
cp /usbdrive/EFI/CLOVER/drivers/off/UEFI/Other/NvmExpressDxe.efi  /usbdrive/EFI/CLOVER/drivers/UEFI/NvmExpressDxe.efi

Now at this point you can boot from that usb drive, and once clover loads you can press "F3" this button displays hidden items, after pressing it you should see a menu item in the middle of the screen to boot the internal NVME drive. Make sure it is selected and press enter, this passes it to the proxmox grub bootloader and proxmox loads, easy peasy.

Now, the next step is to configure clover so that you do not have to press "F3" or select an entry, so it will automatically select and boot proxmox.

This involves editing the "efi/clover/config.plist" file on your usb drive https://wiki.archlinux.org/index.php/Clover

Here is my config.plist, you will have to adjust the Volume key to match the PARTUUID of your EFI partition on your nvme drive:

XML:
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
  <key>Boot</key>
  <dict>
    <key>Timeout</key>
    <integer>0</integer>
    <key>DefaultVolume</key>
    <string>LastBootedVolume</string>
  </dict>
  <key>GUI</key>
  <dict>
    <key>Custom</key>
    <dict>
      <key>Entries</key>
      <array>
        <dict>
          <key>Path</key>
          <string>\EFI\proxmox\grubx64.efi</string>
          <key>Title</key>
          <string>Proxmox</string>
          <key>Type</key>
          <string>Linux</string>
          <key>Volume</key>
          <string>33BA6C23-4772-294D-9053-72A49FCAEF39</string>
          <key>VolumeType</key>
          <string>Internal</string>
        </dict>
      </array>
    </dict>
  </dict>
</dict>
</plist>

The volume key is easy, first boot into proxmox. (make sure the volume key in config.plist is all CAPS/Capitalized)

(you can boot into proxmox using the clover usb and pressing F3, or you can boot using the proxmox installer and choose the debug install, at the first prompt type exit, then another prompt will load with full Linux tools available.)

look for which partition on your nvme drive is the EFI partition
Code:
fdisk -l

next with blkid you can get the PARTUUID for that EFI partition.
Code:
blkid

The sample config.plist above has the correct path for the current incarnation of proxmox.
If this path should however change this is how you find it, mount the EFI partition and find the EFI boot file path:

Code:
mkdir /drive
mount /dev/nvme0n1p2 /drive
find /drive -iname "*.efi"

with your new efi/clover/config.plist file in place on your clover usb drive with a timeout of 0 your machine should boot and go straight into proxmox, there will not even be any indication that you are using clover :)

Also if your machine has any board mounted usb ports or an internal usb header, then you can install the usb drive inside your case so that there is no chance of it being disturbed.
 
Last edited:
  • Like
Reactions: lamplight

Mike.S

Member
Dec 24, 2018
8
1
6
61
I installed Proxmox on (2) M.2 NVME/PCIe adapter on older server that does not support PCI bifurcation. Installation was ZFS RAID1.

I created the clover bootable USB with BDUtility. It formats the USB stick in Windows, downloads Clover and install it on the stick.
http://cvad-mac.narod.ru/index/bootdiskutility_exe/0-5

I could not get the Clover USB to boot in BIOS mode. Switched server to boot in UEFI mode.

When manually booting Proxmox from Clover by pressing F3, I noticed that the grub splash screen was no longer present. Finally, I stumbled onto this Proxmox document and determined that I'm booing EFI with Systemd-boot, not grub.
https://pve.proxmox.com/wiki/Host_Bootloader

I adjusted the config.plist file accordingly. I added a 5 second delay in case one of the NVME fails you can still manually boot from the other.


XML:
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
  <key>Boot</key>
  <dict>
    <key>Timeout</key>
    <integer>5</integer>
    <key>DefaultVolume</key>
    <string>LastBootedVolume</string>
  </dict>
  <key>GUI</key>
  <dict>
    <key>Custom</key>
    <dict>
      <key>Entries</key>
      <array>
        <dict>
          <key>Path</key>
          <string>\EFI\systemd\systemd-bootx64.efi</string>
          <key>Title</key>
          <string>Arch Linux</string>
          <key>Type</key>
          <string>Linux</string>
          <key>Volume</key>
          <string>11A58929-BBCB-47C6-970C-14568FA8D608</string>
          <key>VolumeType</key>
          <string>Internal</string>
        </dict>
        <dict>
          <key>Path</key>
          <string>\EFI\systemd\systemd-bootx64.efi</string>
          <key>Title</key>
          <string>Arch Linux</string>
          <key>Type</key>
          <string>Linux</string>
          <key>Volume</key>
          <string>FF7234B8-F2FC-44BD-BCD4-47236FA36D2D</string>
          <key>VolumeType</key>
          <string>Internal</string>
        </dict>
      </array>
    </dict>
  </dict>
</dict>
</plist>
 
  • Like
Reactions: jieiku

jieiku

Member
Jan 1, 2020
24
15
8
41
Thanks @Mike.S when I wrote the tutorial I was using LVM for the proxmox install. I recently changed to using ZFS and your post was very helpful!

Code:
<string>\EFI\systemd\systemd-bootx64.efi</string>
 

Proxlmoxl

New Member
Aug 29, 2021
2
0
1
47
Hi, I have similar problem with my HP gen8 Microserver. Only booting from SD or USB, no EFI. I installed clover on 2 different SD Cards, but it hangs with Attempting Boot From USB DriceKey (C:) F3 doesn't reakt.

First I installed proxmox via usb with latest proxmox iso 7 at nvme/pcie SSD without problems. because drive isn't visibel in bootorder I put nvme in my pc, boot it and did a grub-install to sd card. rollback to microserver this grub tells me, it didn't find lvmid. I tried DUET but hangs at same point like clover did but with BError!

Can someone help me?
proxlmoxl
 

FernandoT

New Member
Oct 10, 2021
9
0
1
55
Hi Everybody,

In my search for a solution to my problem I stumbled upon this article.
I have a HPE Microserver GEN10 Plus running Proxmox 6.4 flawlessly from a USB ssd and with the PCI SATA controller and drives passed through to a XPENLOGY vm.
This works great, I share the storage as NFS to the host and store data and VM's etc. on NFS.

Now I want to setup my HP Microserver GEN8 (with 16Gb and a Xeon cpu) as a backup server so I can test upgrades etc.
But cant get the passthrough to work, even with the grubb fixes for IMOU and Relaxed RMRR etc.

Once I start the VM Proxmox hangs and I have to cold boot the server.
Than it suddenly I though, well offcourse, you are passing the PCIE SATA controler as a whole to the VM and the OS drive is in the ODD port which is on the same controller, that wont work. :(
For the rest Proxmox works fine as long as I do not pass the sata controller to the VM.

So I started looking for a fix and found this. Installed Proxmox on a SSD in a USB 3.0 external case and prepared the SD card with the clover boot as per you instructions.
The boot takes very long and looks as if it hangs with the booting from c:\ message and a weird 6 bellow it, but if I after a while press F3 and enter a couple of times it generates an error (I had to film it as it goes by really quick) and then it boots proxmox.
The odd thing is that I can login to Proxmox and see that the root partition is 30Gb and not the full 120Gb I assigned, also the networking does not work.

The error is as follows:
Unexpected SPCR Access With. Defaulting to byte suze.
Firmware Bug: the Bios has corrupted hw-PMU resources etc.

Reading all volumes:
Volume group "pve" no found
Cannot process Volume group PVE
Then some of the messages of recovering journal and caching etc. and it boots.
When I login I can see the files and dir structure but nothing seems to be running.
It looks like it cant mount the complete volume group and only loads root.


Anybody a clue what might be going on?
Just need a nudge in the right direction as I am quite stuck at the moment.
 

FernandoT

New Member
Oct 10, 2021
9
0
1
55
Hi Everybody,

In my search for a solution to my problem I stumbled upon this article.
I have a HPE Microserver GEN10 Plus running Proxmox 6.4 flawlessly from a USB ssd and with the PCI SATA controller and drives passed through to a XPENLOGY vm.
This works great, I share the storage as NFS to the host and store data and VM's etc. on NFS.

Now I want to setup my HP Microserver GENven with the grubb fixes for IMOU and Relaxed RMRR etc.

Once I start the VM Proxmox hangs and I have to cold boot the server.
Than it suddenly I though, well offcourse, you are passing the PCIE SATA controler as a whole to the VM and the OS drive is in the ODD port which is on the same controller, that wont work. :(
For the rest Proxmox works fine as long as I do not pass the sata controller to the VM.

So I started looking for a fix and found this. Installed Proxmox on a SSD in a USB 3.0 external case and prepared the SD card with the clover boot as per you instructions.
The boot takes very long and looks as if it hangs with the booting from c:\ message and a weird 6 bellow it, but if I after a while press F3 and enter a couple of times it generates an error (I had to film it as it goes by really quick) and then it boots proxmox.
The odd thing is that I can login to Proxmox and see that the root partition is 30Gb and not the full 120Gb I assigned, also the networking does not work.

The error is as follows:
Unexpected SPCR Access With. Defaulting to byte suze.
Firmware Bug: the Bios has corrupted hw-PMU resources etc.

Reading all volumes:
Volume group "pve" no found
Cannot process Volume group PVE
Then some of the messages of recovering journal and caching etc. and it boots.
When I login I can see the files and dir structure but nothing seems to be running.
It looks like it cant mount the complete volume group and only loads root.


Anybody a clue what might be going on?
Just need a nudge in the right direction as I am quite stuck at the moment.
 

FernandoT

New Member
Oct 10, 2021
9
0
1
55
Ok Further digging and research point out the error messages I receive are just cosmetic. I think I have to search the root cause in the probable fact that not all volumes are being mounted and therefor no modules etc. are being loaded.
The thing is why is this the case as it loads a part of the os of the USB SSD when booting with Clover but then it does not mount the actual volume group.
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get your own in 60 seconds.

Buy now!