[TUTORIAL] bootable NVME install on old hardware made easy with pcie adapter and clover

jieiku

Active Member
Jan 1, 2020
31
23
28
43
My 1TB nvme is installed into an nvme pcie adapter (these can be bought for like $10)

First you simply install proxmox to the NVME, this is straight forward just like installing to a hard drive, the problem comes in to play when you try booting from the NVME, older bios do not support that, even some that are uefi do not support booting from nvme drives.

Clover makes booting an nvme drive via USB easy: https://github.com/CloverHackyColor/CloverBootloader/releases

I formatted my thumb drive with a FAT16 122M partition
I downloaded the CloverV2-5125.zip and extracted to the thumb drive

next you need to enable the NvmExpressDxe.efi driver, do so by copying the file from the disabled drivers "off" folder to the uefi drivers folder:

Code:
cp /usbdrive/EFI/CLOVER/drivers/off/UEFI/Other/NvmExpressDxe.efi  /usbdrive/EFI/CLOVER/drivers/UEFI/NvmExpressDxe.efi

Now at this point you can boot from that usb drive, and once clover loads you can press "F3" this button displays hidden items, after pressing it you should see a menu item in the middle of the screen to boot the internal NVME drive. Make sure it is selected and press enter, this passes it to the proxmox grub bootloader and proxmox loads, easy peasy.

Now, the next step is to configure clover so that you do not have to press "F3" or select an entry, so it will automatically select and boot proxmox.

This involves editing the "efi/clover/config.plist" file on your usb drive https://wiki.archlinux.org/index.php/Clover

Here is my config.plist, you will have to adjust the Volume key to match the PARTUUID of your EFI partition on your nvme drive:

XML:
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
  <key>Boot</key>
  <dict>
    <key>Timeout</key>
    <integer>0</integer>
    <key>DefaultVolume</key>
    <string>LastBootedVolume</string>
  </dict>
  <key>GUI</key>
  <dict>
    <key>Custom</key>
    <dict>
      <key>Entries</key>
      <array>
        <dict>
          <key>Path</key>
          <string>\EFI\proxmox\grubx64.efi</string>
          <key>Title</key>
          <string>Proxmox</string>
          <key>Type</key>
          <string>Linux</string>
          <key>Volume</key>
          <string>33BA6C23-4772-294D-9053-72A49FCAEF39</string>
          <key>VolumeType</key>
          <string>Internal</string>
        </dict>
      </array>
    </dict>
  </dict>
</dict>
</plist>

The volume key is easy, first boot into proxmox. (make sure the volume key in config.plist is all CAPS/Capitalized)

(you can boot into proxmox using the clover usb and pressing F3, or you can boot using the proxmox installer and choose the debug install, at the first prompt type exit, then another prompt will load with full Linux tools available.)

look for which partition on your nvme drive is the EFI partition
Code:
fdisk -l

next with blkid you can get the PARTUUID for that EFI partition.
Code:
blkid

The sample config.plist above has the correct path for the current incarnation of proxmox.
If this path should however change this is how you find it, mount the EFI partition and find the EFI boot file path:

Code:
mkdir /drive
mount /dev/nvme0n1p2 /drive
find /drive -iname "*.efi"

with your new efi/clover/config.plist file in place on your clover usb drive with a timeout of 0 your machine should boot and go straight into proxmox, there will not even be any indication that you are using clover :)

Also if your machine has any board mounted usb ports or an internal usb header, then you can install the usb drive inside your case so that there is no chance of it being disturbed.
 
Last edited:
I installed Proxmox on (2) M.2 NVME/PCIe adapter on older server that does not support PCI bifurcation. Installation was ZFS RAID1.

I created the clover bootable USB with BDUtility. It formats the USB stick in Windows, downloads Clover and install it on the stick.
http://cvad-mac.narod.ru/index/bootdiskutility_exe/0-5

I could not get the Clover USB to boot in BIOS mode. Switched server to boot in UEFI mode.

When manually booting Proxmox from Clover by pressing F3, I noticed that the grub splash screen was no longer present. Finally, I stumbled onto this Proxmox document and determined that I'm booing EFI with Systemd-boot, not grub.
https://pve.proxmox.com/wiki/Host_Bootloader

I adjusted the config.plist file accordingly. I added a 5 second delay in case one of the NVME fails you can still manually boot from the other.


XML:
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
  <key>Boot</key>
  <dict>
    <key>Timeout</key>
    <integer>5</integer>
    <key>DefaultVolume</key>
    <string>LastBootedVolume</string>
  </dict>
  <key>GUI</key>
  <dict>
    <key>Custom</key>
    <dict>
      <key>Entries</key>
      <array>
        <dict>
          <key>Path</key>
          <string>\EFI\systemd\systemd-bootx64.efi</string>
          <key>Title</key>
          <string>Arch Linux</string>
          <key>Type</key>
          <string>Linux</string>
          <key>Volume</key>
          <string>11A58929-BBCB-47C6-970C-14568FA8D608</string>
          <key>VolumeType</key>
          <string>Internal</string>
        </dict>
        <dict>
          <key>Path</key>
          <string>\EFI\systemd\systemd-bootx64.efi</string>
          <key>Title</key>
          <string>Arch Linux</string>
          <key>Type</key>
          <string>Linux</string>
          <key>Volume</key>
          <string>FF7234B8-F2FC-44BD-BCD4-47236FA36D2D</string>
          <key>VolumeType</key>
          <string>Internal</string>
        </dict>
      </array>
    </dict>
  </dict>
</dict>
</plist>
 
  • Like
Reactions: jieiku
Thanks @Mike.S when I wrote the tutorial I was using LVM for the proxmox install. I recently changed to using ZFS and your post was very helpful!

Code:
<string>\EFI\systemd\systemd-bootx64.efi</string>
 
Hi, I have similar problem with my HP gen8 Microserver. Only booting from SD or USB, no EFI. I installed clover on 2 different SD Cards, but it hangs with Attempting Boot From USB DriceKey (C:) F3 doesn't reakt.

First I installed proxmox via usb with latest proxmox iso 7 at nvme/pcie SSD without problems. because drive isn't visibel in bootorder I put nvme in my pc, boot it and did a grub-install to sd card. rollback to microserver this grub tells me, it didn't find lvmid. I tried DUET but hangs at same point like clover did but with BError!

Can someone help me?
proxlmoxl
 
Hi Everybody,

In my search for a solution to my problem I stumbled upon this article.
I have a HPE Microserver GEN10 Plus running Proxmox 6.4 flawlessly from a USB ssd and with the PCI SATA controller and drives passed through to a XPENLOGY vm.
This works great, I share the storage as NFS to the host and store data and VM's etc. on NFS.

Now I want to setup my HP Microserver GEN8 (with 16Gb and a Xeon cpu) as a backup server so I can test upgrades etc.
But cant get the passthrough to work, even with the grubb fixes for IMOU and Relaxed RMRR etc.

Once I start the VM Proxmox hangs and I have to cold boot the server.
Than it suddenly I though, well offcourse, you are passing the PCIE SATA controler as a whole to the VM and the OS drive is in the ODD port which is on the same controller, that wont work. :(
For the rest Proxmox works fine as long as I do not pass the sata controller to the VM.

So I started looking for a fix and found this. Installed Proxmox on a SSD in a USB 3.0 external case and prepared the SD card with the clover boot as per you instructions.
The boot takes very long and looks as if it hangs with the booting from c:\ message and a weird 6 bellow it, but if I after a while press F3 and enter a couple of times it generates an error (I had to film it as it goes by really quick) and then it boots proxmox.
The odd thing is that I can login to Proxmox and see that the root partition is 30Gb and not the full 120Gb I assigned, also the networking does not work.

The error is as follows:
Unexpected SPCR Access With. Defaulting to byte suze.
Firmware Bug: the Bios has corrupted hw-PMU resources etc.

Reading all volumes:
Volume group "pve" no found
Cannot process Volume group PVE
Then some of the messages of recovering journal and caching etc. and it boots.
When I login I can see the files and dir structure but nothing seems to be running.
It looks like it cant mount the complete volume group and only loads root.


Anybody a clue what might be going on?
Just need a nudge in the right direction as I am quite stuck at the moment.
 
Hi Everybody,

In my search for a solution to my problem I stumbled upon this article.
I have a HPE Microserver GEN10 Plus running Proxmox 6.4 flawlessly from a USB ssd and with the PCI SATA controller and drives passed through to a XPENLOGY vm.
This works great, I share the storage as NFS to the host and store data and VM's etc. on NFS.

Now I want to setup my HP Microserver GENven with the grubb fixes for IMOU and Relaxed RMRR etc.

Once I start the VM Proxmox hangs and I have to cold boot the server.
Than it suddenly I though, well offcourse, you are passing the PCIE SATA controler as a whole to the VM and the OS drive is in the ODD port which is on the same controller, that wont work. :(
For the rest Proxmox works fine as long as I do not pass the sata controller to the VM.

So I started looking for a fix and found this. Installed Proxmox on a SSD in a USB 3.0 external case and prepared the SD card with the clover boot as per you instructions.
The boot takes very long and looks as if it hangs with the booting from c:\ message and a weird 6 bellow it, but if I after a while press F3 and enter a couple of times it generates an error (I had to film it as it goes by really quick) and then it boots proxmox.
The odd thing is that I can login to Proxmox and see that the root partition is 30Gb and not the full 120Gb I assigned, also the networking does not work.

The error is as follows:
Unexpected SPCR Access With. Defaulting to byte suze.
Firmware Bug: the Bios has corrupted hw-PMU resources etc.

Reading all volumes:
Volume group "pve" no found
Cannot process Volume group PVE
Then some of the messages of recovering journal and caching etc. and it boots.
When I login I can see the files and dir structure but nothing seems to be running.
It looks like it cant mount the complete volume group and only loads root.


Anybody a clue what might be going on?
Just need a nudge in the right direction as I am quite stuck at the moment.
 
Ok Further digging and research point out the error messages I receive are just cosmetic. I think I have to search the root cause in the probable fact that not all volumes are being mounted and therefor no modules etc. are being loaded.
The thing is why is this the case as it loads a part of the os of the USB SSD when booting with Clover but then it does not mount the actual volume group.
 
So for some reason I followed every step up to the editing the config file, but it refuses to automatically boot into proxmox. this is my confg file at the moment

XML:
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
  <key>Boot</key>
  <dict>
    <key>Timeout</key>
    <integer>0</integer>
    <key>DefaultVolume</key>
    <string>LastBootedVolume</string>
  </dict>
  <key>GUI</key>
  <dict>
    <key>Custom</key>
    <dict>
      <key>Entries</key>
      <array>
        <dict>
          <key>Path</key>
          <string>\EFI\proxmox\grubx64.efi</string>
          <key>Title</key>
          <string>Proxmox</string>
          <key>Type</key>
          <string>Linux</string>
          <key>Volume</key>
          <string>05a68ebf-3d98-4635-bff4-c63cb50743ab</string>
          <key>VolumeType</key>
          <string>Internal</string>
        </dict>
      </array>
    </dict>
  </dict>
</dict>
</plist>
 
Hello everyone!

I have a HP Proliant DL160 G6 that i'm having trouble booting from NVME using this method.

When i boot with the USB stick it just shows a black screen with a blinking cursor but no prompt.

Can anyone point me on the right direction or if there's another approach i could use to boot this thing?
 
So for some reason I followed every step up to the editing the config file, but it refuses to automatically boot into proxmox. this is my confg file at the moment

XML:
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
  <key>Boot</key>
  <dict>
    <key>Timeout</key>
    <integer>0</integer>
    <key>DefaultVolume</key>
    <string>LastBootedVolume</string>
  </dict>
  <key>GUI</key>
  <dict>
    <key>Custom</key>
    <dict>
      <key>Entries</key>
      <array>
        <dict>
          <key>Path</key>
          <string>\EFI\proxmox\grubx64.efi</string>
          <key>Title</key>
          <string>Proxmox</string>
          <key>Type</key>
          <string>Linux</string>
          <key>Volume</key>
          <string>05a68ebf-3d98-4635-bff4-c63cb50743ab</string>
          <key>VolumeType</key>
          <string>Internal</string>
        </dict>
      </array>
    </dict>
  </dict>
</dict>
</plist>
Hi,

I think you have to change the PARTUUID to uppercase for it to be taken into account.
For my part it worked.
 
I mounted clover to usb, Copied NvmExpressDxe.efi to /usbdrive/EFI/CLOVER/drivers/UEFI/. When I boot and f10 I get the clover menu. I choose to boot from new nvme ssd w win10. Good so far. But then I just get black screen and it stays there. I don't think it's a problem accessing the new nvme OS drive, because If I boot regular, I can access the nvme drive. Also, when I'm at the clover menu and I choose to boot from the old sata SDD with the OS, I still get the black screen hangup. In other words, if I boot regular, the old SSD OS works fine, but If I go through the clover menu I can't boot from anything. How to fix the clover on the USB? I just downloaded clover today, so it is the latest version.
 
I installed Proxmox on (2) M.2 NVME/PCIe adapter on older server that does not support PCI bifurcation. Installation was ZFS RAID1.

I created the clover bootable USB with BDUtility. It formats the USB stick in Windows, downloads Clover and install it on the stick.
http://cvad-mac.narod.ru/index/bootdiskutility_exe/0-5

I could not get the Clover USB to boot in BIOS mode. Switched server to boot in UEFI mode.

When manually booting Proxmox from Clover by pressing F3, I noticed that the grub splash screen was no longer present. Finally, I stumbled onto this Proxmox document and determined that I'm booing EFI with Systemd-boot, not grub.
https://pve.proxmox.com/wiki/Host_Bootloader

I adjusted the config.plist file accordingly. I added a 5 second delay in case one of the NVME fails you can still manually boot from the other.


XML:
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
  <key>Boot</key>
  <dict>
    <key>Timeout</key>
    <integer>5</integer>
    <key>DefaultVolume</key>
    <string>LastBootedVolume</string>
  </dict>
  <key>GUI</key>
  <dict>
    <key>Custom</key>
    <dict>
      <key>Entries</key>
      <array>
        <dict>
          <key>Path</key>
          <string>\EFI\systemd\systemd-bootx64.efi</string>
          <key>Title</key>
          <string>Arch Linux</string>
          <key>Type</key>
          <string>Linux</string>
          <key>Volume</key>
          <string>11A58929-BBCB-47C6-970C-14568FA8D608</string>
          <key>VolumeType</key>
          <string>Internal</string>
        </dict>
        <dict>
          <key>Path</key>
          <string>\EFI\systemd\systemd-bootx64.efi</string>
          <key>Title</key>
          <string>Arch Linux</string>
          <key>Type</key>
          <string>Linux</string>
          <key>Volume</key>
          <string>FF7234B8-F2FC-44BD-BCD4-47236FA36D2D</string>
          <key>VolumeType</key>
          <string>Internal</string>
        </dict>
      </array>
    </dict>
  </dict>
</dict>
</plist>

I know this is an old thread but I just wanted to say thank you for posting your version of the config file and also to Jieiku for the original post.
Been looking around for someone who did the configuration the same way I was looking to.

I installed Clover onto an R740XD using the IDSDM and was able to boot into my pair of NVME drives in a single PCI-E card.
I didn't install Proxmox but TrueNAS Scale but the config file works just the same.
 
  • Like
Reactions: uijin_hong
Hello, I can succesfully boot latest PVE from both of mine NVMe disks in ZFS1 RAID from Clover, boot option is automatically selected and boot is initiated after timeout. Unfortunatelly after that, the boot stuck on this screen and I have to hit Enter to continue boot. Is this problem of PVE or Clover? Thank you[ATTACH type="full"]73075[/ATTACH]
 

Attachments

  • P_20240815_104328.jpg
    P_20240815_104328.jpg
    126.2 KB · Views: 6
Hello, I can succesfully boot latest PVE from both of mine NVMe disks in ZFS1 RAID from Clover, boot option is automatically selected and boot is initiated after timeout. Unfortunatelly after that, the boot stuck on this screen and I have to hit Enter to continue boot. Is this problem of PVE or Clover? Thank you[ATTACH type="full"]73075[/ATTACH]

I have the same setup you do for dual NVME m.2 in ZFS1.
My timeout is set to 0 below but it goes to a screen saying "boot in 3 s" but nothing happens and I have to press enter.
I have the code I used from another user above but with my device ID, maybe someone can help me understand what I can change to automate this boot.

XML:
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
  <key>Boot</key>
  <dict>
    <key>Timeout</key>
    <integer>0</integer>
    <key>DefaultVolume</key>
    <string>LastBootedVolume</string>
  </dict>
  <key>GUI</key>
  <dict>
    <key>Custom</key>
    <dict>
      <key>Entries</key>
      <array>
        <dict>
          <key>Path</key>
          <string>\EFI\systemd\systemd-bootx64.efi</string>
          <key>Title</key>
          <string>Proxmox</string>
          <key>Type</key>
          <string>Linux</string>
          <key>Volume</key>
          <string>87912DA3-49CD-4D7A-8478-5D62E8D72C78</string>
          <key>VolumeType</key>
          <string>Internal</string>
        </dict>
      </array>
    </dict>
  </dict>
</dict>
</plist>
 
Last edited:
I finally found that this timeout is not related to Clover but to Grub. I had to set timeout to 0 in /boot/grub/grub.cfg and my server is now able to boot without pressing any keys :) Hope it helps.
 
Thanks for this guide; I can confirm this works on my Dell Wyse 5070 with an m.2 NVME drive plugged into an AE Key adapter board plugged into the WiFi slot. I previously had tried supergrub2, rescatux, and refind ISOs booted off of Ventoy and none seemed to detect Proxmox. However, Proxmox happily installed to the NVME drive. The only odd thing was that I kept getting a black screen after hitting F3 and choosing Proxmox in Clover v2 5160. I did some reading and there was evidently some merger of different codebases in v5122; I rolled back and downloaded v5121, changed the config and everything works great. That could just be a drivers issue but I stick with v5121 and didn't investigate further.
After smoke testing this with a USB drive, I was able to use a Ubuntu live ISO to boot up, mount the EMMC storage, and copy the USB contents over to EMMC. In gparted I configured the "boot" flag and now the built-in EMMC storage bootstraps the boot process and I don't need to have a USB drive.
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!