[TUTORIAL] bootable NVME install on old hardware made easy with pcie adapter and clover

J

jieiku

Active Member
Jan 1, 2020
31
20
28
43
My 1TB nvme is installed into an nvme pcie adapter (these can be bought for like $10)

First you simply install proxmox to the NVME, this is straight forward just like installing to a hard drive, the problem comes in to play when you try booting from the NVME, older bios do not support that, even some that are uefi do not support booting from nvme drives.

Clover makes booting an nvme drive via USB easy: https://github.com/CloverHackyColor/CloverBootloader/releases

I formatted my thumb drive with a FAT16 122M partition
I downloaded the CloverV2-5125.zip and extracted to the thumb drive

next you need to enable the NvmExpressDxe.efi driver, do so by copying the file from the disabled drivers "off" folder to the uefi drivers folder:

Code: 
cp /usbdrive/EFI/CLOVER/drivers/off/UEFI/Other/NvmExpressDxe.efi  /usbdrive/EFI/CLOVER/drivers/UEFI/NvmExpressDxe.efi

Now at this point you can boot from that usb drive, and once clover loads you can press "F3" this button displays hidden items, after pressing it you should see a menu item in the middle of the screen to boot the internal NVME drive. Make sure it is selected and press enter, this passes it to the proxmox grub bootloader and proxmox loads, easy peasy.

Now, the next step is to configure clover so that you do not have to press "F3" or select an entry, so it will automatically select and boot proxmox.

This involves editing the "efi/clover/config.plist" file on your usb drive https://wiki.archlinux.org/index.php/Clover

Here is my config.plist, you will have to adjust the Volume key to match the PARTUUID of your EFI partition on your nvme drive:

XML: 
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
  <key>Boot</key>
  <dict>
    <key>Timeout</key>
    <integer>0</integer>
    <key>DefaultVolume</key>
    <string>LastBootedVolume</string>
  </dict>
  <key>GUI</key>
  <dict>
    <key>Custom</key>
    <dict>
      <key>Entries</key>
      <array>
        <dict>
          <key>Path</key>
          <string>\EFI\proxmox\grubx64.efi</string>
          <key>Title</key>
          <string>Proxmox</string>
          <key>Type</key>
          <string>Linux</string>
          <key>Volume</key>
          <string>33BA6C23-4772-294D-9053-72A49FCAEF39</string>
          <key>VolumeType</key>
          <string>Internal</string>
        </dict>
      </array>
    </dict>
  </dict>
</dict>
</plist>

The volume key is easy, first boot into proxmox. (make sure the volume key in config.plist is all CAPS/Capitalized)

(you can boot into proxmox using the clover usb and pressing F3, or you can boot using the proxmox installer and choose the debug install, at the first prompt type exit, then another prompt will load with full Linux tools available.)

look for which partition on your nvme drive is the EFI partition
Code: 
fdisk -l

next with blkid you can get the PARTUUID for that EFI partition.
Code: 
blkid

The sample config.plist above has the correct path for the current incarnation of proxmox.
If this path should however change this is how you find it, mount the EFI partition and find the EFI boot file path:

Code: 
mkdir /drive
mount /dev/nvme0n1p2 /drive
find /drive -iname "*.efi"

with your new efi/clover/config.plist file in place on your clover usb drive with a timeout of 0 your machine should boot and go straight into proxmox, there will not even be any indication that you are using clover :)

Also if your machine has any board mounted usb ports or an internal usb header, then you can install the usb drive inside your case so that there is no chance of it being disturbed.
 
Last edited:
  • Like
Reactions: GarrettB, rabbifede, chOzcA75vE0poCY0F6XC and 1 other person
I installed Proxmox on (2) M.2 NVME/PCIe adapter on older server that does not support PCI bifurcation. Installation was ZFS RAID1.

I created the clover bootable USB with BDUtility. It formats the USB stick in Windows, downloads Clover and install it on the stick.
http://cvad-mac.narod.ru/index/bootdiskutility_exe/0-5

I could not get the Clover USB to boot in BIOS mode. Switched server to boot in UEFI mode.

When manually booting Proxmox from Clover by pressing F3, I noticed that the grub splash screen was no longer present. Finally, I stumbled onto this Proxmox document and determined that I'm booing EFI with Systemd-boot, not grub.
https://pve.proxmox.com/wiki/Host_Bootloader

I adjusted the config.plist file accordingly. I added a 5 second delay in case one of the NVME fails you can still manually boot from the other.


XML: 
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
  <key>Boot</key>
  <dict>
    <key>Timeout</key>
    <integer>5</integer>
    <key>DefaultVolume</key>
    <string>LastBootedVolume</string>
  </dict>
  <key>GUI</key>
  <dict>
    <key>Custom</key>
    <dict>
      <key>Entries</key>
      <array>
        <dict>
          <key>Path</key>
          <string>\EFI\systemd\systemd-bootx64.efi</string>
          <key>Title</key>
          <string>Arch Linux</string>
          <key>Type</key>
          <string>Linux</string>
          <key>Volume</key>
          <string>11A58929-BBCB-47C6-970C-14568FA8D608</string>
          <key>VolumeType</key>
          <string>Internal</string>
        </dict>
        <dict>
          <key>Path</key>
          <string>\EFI\systemd\systemd-bootx64.efi</string>
          <key>Title</key>
          <string>Arch Linux</string>
          <key>Type</key>
          <string>Linux</string>
          <key>Volume</key>
          <string>FF7234B8-F2FC-44BD-BCD4-47236FA36D2D</string>
          <key>VolumeType</key>
          <string>Internal</string>
        </dict>
      </array>
    </dict>
  </dict>
</dict>
</plist>
 
  • Like
Reactions: jieiku
Thanks @Mike.S when I wrote the tutorial I was using LVM for the proxmox install. I recently changed to using ZFS and your post was very helpful!

Code: 
<string>\EFI\systemd\systemd-bootx64.efi</string>
 
Hi, I have similar problem with my HP gen8 Microserver. Only booting from SD or USB, no EFI. I installed clover on 2 different SD Cards, but it hangs with Attempting Boot From USB DriceKey (C:) F3 doesn't reakt.

First I installed proxmox via usb with latest proxmox iso 7 at nvme/pcie SSD without problems. because drive isn't visibel in bootorder I put nvme in my pc, boot it and did a grub-install to sd card. rollback to microserver this grub tells me, it didn't find lvmid. I tried DUET but hangs at same point like clover did but with BError!

Can someone help me?
proxlmoxl
 
Hi Everybody,

In my search for a solution to my problem I stumbled upon this article.
I have a HPE Microserver GEN10 Plus running Proxmox 6.4 flawlessly from a USB ssd and with the PCI SATA controller and drives passed through to a XPENLOGY vm.
This works great, I share the storage as NFS to the host and store data and VM's etc. on NFS.

Now I want to setup my HP Microserver GEN8 (with 16Gb and a Xeon cpu) as a backup server so I can test upgrades etc.
But cant get the passthrough to work, even with the grubb fixes for IMOU and Relaxed RMRR etc.

Once I start the VM Proxmox hangs and I have to cold boot the server.
Than it suddenly I though, well offcourse, you are passing the PCIE SATA controler as a whole to the VM and the OS drive is in the ODD port which is on the same controller, that wont work. :(
For the rest Proxmox works fine as long as I do not pass the sata controller to the VM.

So I started looking for a fix and found this. Installed Proxmox on a SSD in a USB 3.0 external case and prepared the SD card with the clover boot as per you instructions.
The boot takes very long and looks as if it hangs with the booting from c:\ message and a weird 6 bellow it, but if I after a while press F3 and enter a couple of times it generates an error (I had to film it as it goes by really quick) and then it boots proxmox.
The odd thing is that I can login to Proxmox and see that the root partition is 30Gb and not the full 120Gb I assigned, also the networking does not work.

The error is as follows:
Unexpected SPCR Access With. Defaulting to byte suze.
Firmware Bug: the Bios has corrupted hw-PMU resources etc.

Reading all volumes:
Volume group "pve" no found
Cannot process Volume group PVE
Then some of the messages of recovering journal and caching etc. and it boots.
When I login I can see the files and dir structure but nothing seems to be running.
It looks like it cant mount the complete volume group and only loads root.


Anybody a clue what might be going on?
Just need a nudge in the right direction as I am quite stuck at the moment.
 
FernandoT said:
Hi Everybody,

In my search for a solution to my problem I stumbled upon this article.
I have a HPE Microserver GEN10 Plus running Proxmox 6.4 flawlessly from a USB ssd and with the PCI SATA controller and drives passed through to a XPENLOGY vm.
This works great, I share the storage as NFS to the host and store data and VM's etc. on NFS.

Now I want to setup my HP Microserver GENven with the grubb fixes for IMOU and Relaxed RMRR etc.

Once I start the VM Proxmox hangs and I have to cold boot the server.
Than it suddenly I though, well offcourse, you are passing the PCIE SATA controler as a whole to the VM and the OS drive is in the ODD port which is on the same controller, that wont work. :(
For the rest Proxmox works fine as long as I do not pass the sata controller to the VM.

So I started looking for a fix and found this. Installed Proxmox on a SSD in a USB 3.0 external case and prepared the SD card with the clover boot as per you instructions.
The boot takes very long and looks as if it hangs with the booting from c:\ message and a weird 6 bellow it, but if I after a while press F3 and enter a couple of times it generates an error (I had to film it as it goes by really quick) and then it boots proxmox.
The odd thing is that I can login to Proxmox and see that the root partition is 30Gb and not the full 120Gb I assigned, also the networking does not work.

The error is as follows:
Unexpected SPCR Access With. Defaulting to byte suze.
Firmware Bug: the Bios has corrupted hw-PMU resources etc.

Reading all volumes:
Volume group "pve" no found
Cannot process Volume group PVE
Then some of the messages of recovering journal and caching etc. and it boots.
When I login I can see the files and dir structure but nothing seems to be running.
It looks like it cant mount the complete volume group and only loads root.


Anybody a clue what might be going on?
Just need a nudge in the right direction as I am quite stuck at the moment.
Click to expand...
 
Ok Further digging and research point out the error messages I receive are just cosmetic. I think I have to search the root cause in the probable fact that not all volumes are being mounted and therefor no modules etc. are being loaded.
The thing is why is this the case as it loads a part of the os of the USB SSD when booting with Clover but then it does not mount the actual volume group.
 
So for some reason I followed every step up to the editing the config file, but it refuses to automatically boot into proxmox. this is my confg file at the moment

XML: 
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
  <key>Boot</key>
  <dict>
    <key>Timeout</key>
    <integer>0</integer>
    <key>DefaultVolume</key>
    <string>LastBootedVolume</string>
  </dict>
  <key>GUI</key>
  <dict>
    <key>Custom</key>
    <dict>
      <key>Entries</key>
      <array>
        <dict>
          <key>Path</key>
          <string>\EFI\proxmox\grubx64.efi</string>
          <key>Title</key>
          <string>Proxmox</string>
          <key>Type</key>
          <string>Linux</string>
          <key>Volume</key>
          <string>05a68ebf-3d98-4635-bff4-c63cb50743ab</string>
          <key>VolumeType</key>
          <string>Internal</string>
        </dict>
      </array>
    </dict>
  </dict>
</dict>
</plist>
 
Hello everyone!

I have a HP Proliant DL160 G6 that i'm having trouble booting from NVME using this method.

When i boot with the USB stick it just shows a black screen with a blinking cursor but no prompt.

Can anyone point me on the right direction or if there's another approach i could use to boot this thing?
 
dantelogan said:
So for some reason I followed every step up to the editing the config file, but it refuses to automatically boot into proxmox. this is my confg file at the moment

XML: 
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
  <key>Boot</key>
  <dict>
    <key>Timeout</key>
    <integer>0</integer>
    <key>DefaultVolume</key>
    <string>LastBootedVolume</string>
  </dict>
  <key>GUI</key>
  <dict>
    <key>Custom</key>
    <dict>
      <key>Entries</key>
      <array>
        <dict>
          <key>Path</key>
          <string>\EFI\proxmox\grubx64.efi</string>
          <key>Title</key>
          <string>Proxmox</string>
          <key>Type</key>
          <string>Linux</string>
          <key>Volume</key>
          <string>05a68ebf-3d98-4635-bff4-c63cb50743ab</string>
          <key>VolumeType</key>
          <string>Internal</string>
        </dict>
      </array>
    </dict>
  </dict>
</dict>
</plist>
Click to expand...
Hi,

I think you have to change the PARTUUID to uppercase for it to be taken into account.
For my part it worked.
 
I mounted clover to usb, Copied NvmExpressDxe.efi to /usbdrive/EFI/CLOVER/drivers/UEFI/. When I boot and f10 I get the clover menu. I choose to boot from new nvme ssd w win10. Good so far. But then I just get black screen and it stays there. I don't think it's a problem accessing the new nvme OS drive, because If I boot regular, I can access the nvme drive. Also, when I'm at the clover menu and I choose to boot from the old sata SDD with the OS, I still get the black screen hangup. In other words, if I boot regular, the old SSD OS works fine, but If I go through the clover menu I can't boot from anything. How to fix the clover on the USB? I just downloaded clover today, so it is the latest version.
 
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom
Back