How to install Proxmox on PCIe NVMe disk when bios can't boot from it? SD card?

S

SanderM

Member
Oct 21, 2016
40
1
6
40
I have a couple Dell R710 servers with an PCI-e NVMe SSD and an SD card inside.
The BIOS doesn't support booting from the PCI-e SSD card so the only bootable thing is an SD card.

I can install proxmox on the SD card, works without problems.
I can also install proxmox on the NVMe SSD. It is detected during installation. But the BIOS can't boot from it so after installation I can't boot into proxmox.

Now I was wondering if there's any way I could install proxmox on the SSD but use the SD card for booting up proxmox, which is on the SSD...

Maybe there's a way to have the bootloader installed on the SD card which then continues booting from the SSD after the bootloader is loaded from the SD card?

Thanks for any tips!!!
 
yeah, but you'll need to set this up manually (for example, using a debian jessie installation which you upgrade to PVE).

grub is comprised of several parts ("stages"), of which only the first on needs to be installed on the "boot device" as seen by the bios. so you can either only install the stage 1 onto your SD card, and let it load stage 2 from your SSD, or you can put both on your SD card and just put /boot with the initrd and kernel onto the SSD.

I'd recommend looking for "grub on usb stick" instructions for debian jessie - those should be fairly easy to adapt (basically, when using the debian jessie installer, telling it to install grub to the SD card instead of the SSD should be enough)
 
I'm not so very technical so if you could give me a link containing the right commands I'll try to understand them and read into the docs if needed.
I'll report back if I have issues and hopefully I can get this to work :)
 
One more question: what if I want to upgrade proxmox. Will I have to redo everything or isn't it required to update the grub stage1 on the SD?
 
I'm thinking.. If I use the Proxmox installation CD I can install everything on the SSD.
So proxmox, grub included, will be installed on the SSD completely. I just can't boot from it becuase of BIOS limitations.

Can't I copy the grub stage1 from the SSD to the SD in order to boot it? Or can't I install grub on the SD easily with a recovery ISO image or something... So I don't have to install Debian Jessie and upgrade it to proxmox manually...
 
SanderM said:
I'm thinking.. If I use the Proxmox installation CD I can install everything on the SSD.
So proxmox, grub included, will be installed on the SSD completely. I just can't boot from it becuase of BIOS limitations.

Can't I copy the grub stage1 from the SSD to the SD in order to boot it? Or can't I install grub on the SD easily with a recovery ISO image or something... So I don't have to install Debian Jessie and upgrade it to proxmox manually...
Click to expand...

yes, you can try running "grub-install" onto your SD card when booting from a rescue environment (easiest way - select "Rescue Boot" when booting form the PVE 4.3 installer image, it will boot your installed PVE from disk). depending on your platform (legacy or UEFI), you will have to format the disk accordingly (and maybe set the --target option for grub-install).
 
Sorry for the late reply but something unexpected happend and I wasn't able to reply faster.

Today I want to try to get this to work. The bios is legacy, not UEFI, so I should first install proxmox on the SSD and then boot into reque mode.
In resque mode what exactly should I do?
1. Use fdisk to partion the SD card as a single partition
2. Create an ext4 filesystem on it?
3. Run grub-install /dev/sd? (SD card device).

Is that all?
 
I think just creating an msdos partition table and running grub-install /dev/sd? should be enough - but you basically have unlimited tries (as long as you don't accidentally overwrite your SSD), so you can experiment with GPT/MBR and different setups ;)
 
I tried the following in my rescue environment:


vgscan
vgchange -ay

mkdir /PVE
mount /dev/pve/root /PVE
mount -t proc proc /PVE/proc
mount -t sysfs sys /PVE/sys
mount -o bind /dev /PVE/dev
chroot /PVE

Inside the chroot i did:

fdisk /dev/sdb -> create 1 single partition on the SD card, and exit fdisk.
mkfs.ext4 /dev/sdb1
grub-install /dev/sdb

(it's /dev/sdb because /dev/sda is the sata cdrom)


I didn't get any errors when doing this but when I reboot I get:

error: disk 'lvmid/1POEYZ-RTV2-bRsT-923U-2UES-sMEO-s3-InOv/c0SAfa-YNMZ-dlQY-v6eW-BjFm-EVVN-Lkg60g' not found.
Entering rescue mode...
grub rescue>
 
could you post the output of "ls" and "set" in the grub rescue prompt?

if everything else fails, you could put your whole "/boot" onto the SD card, but that means more wear (kernel and initrd are in /boot, and those get written on every kernel update at least)
 
Here's the output of "ls":

grub rescue> ls
(hd0)
grub rescue>

Here's the output of "set":
grub rescue> set
cmdpath=(hd0)
prefix=(lvmid/1POEYZ-RTV2.....)/boot/grub
root=lvmid/1POEYZ-RTV2......
grub rescue>

I'm not sure what my options are ? Can you help me out?

I don't think it'll be a problem for the kernel to be on the SD card, is it?
It's going to be updated every hour or so :)

But I'm not sure how I can do that if it's a requirement.
 
basically
  • boot into rescue mode using iso
  • format sdcard with msdos partition table
  • create 512MB ext2 partition, set bootable flag
  • mv old "/boot" to "/boot-old"
  • mount partition on sdcard to "/boot"
  • cp files from "/boot-old" to "/boot"
  • setup /etc/fstab entry for "/boot"
  • update-initramfs -u
  • update-grub2
  • "grub-install /dev/sd?" (where sd? is your sdcard)
if anything goes wrong, you can revert to the old state by removing the /etc/fstab line, unmounting /boot and moving /boot-old to /boot
 
SanderM said:
Thanks fabian!!

It's working now :)
Click to expand...

great. I'd recommend taking regular backups of your sdcard and checking its health (if possible)
 
But it looks like the install allocated 59Gb to /, 38Gb to /run and since it's a 256Gb SSD I wonder where the other space is now.
There's 95Gb in /dev/shm and /sys/fs/cgroup it sais in "df".

Maybe I should reïnstall and use the advanced options in the first screen where it asks for the disk?
I know I can select swap size etc...

If needed, what options should I set for a server with 192Gb ram and 256Gb SSD?
 
fabian said:
great. I'd recommend taking regular backups of your sdcard and checking its health (if possible)
Click to expand...

Do you really think it's a bad idea to have the kernel on SD? I don't think it's updated A LOT. Shouldn't the SD card be able to survive this for a long period?
 
SanderM said:
But it looks like the install allocated 59Gb to /, 38Gb to /run and since it's a 256Gb SSD I wonder where the other space is now.
There's 95Gb in /dev/shm and /sys/fs/cgroup it sais in "df".

Maybe I should reïnstall and use the advanced options in the first screen where it asks for the disk?
I know I can select swap size etc...

If needed, what options should I set for a server with 192Gb ram and 256Gb SSD?
Click to expand...

the default options in the installer will allocate most of the space to an LVM thin pool, which is not mounted and therefore not shown by tools like "df -h". check the GUI or "pvesm status" to see used and free space, and also check the documentation at http://pve.proxmox.com/pve-docs/pve-admin-guide.html#_logical_volume_manager_lvm and http://pve.proxmox.com/pve-docs/pve-admin-guide.html#_lvm_thin_backend

/run , /dev/shm and /sys/fs/cgroup are all tmpfs (memory-backed), so does don't actually take any disk space.

Do you really think it's a bad idea to have the kernel on SD? I don't think it's updated A LOT. Shouldn't the SD card be able to survive this for a long period?
Click to expand...

I'd be weary of any single point of failure. better safe than sorry, having a backup never hurts (and an image of the sdcard should be pretty small).
 
Ok...

The card never gets mounted, so never get written...It just kicks of to the other grub on the PVE disk...

My SD-Card:
Code: 
.
├── boot
  └── grub
  ├── fonts
  │ └── unicode.pf2
  ├── grub.cfg
  ├── grubenv
  ├── i386-pc
  │ ├── acpi.mod
  │ ├── ... lots of grub modules ...
  │ ├── zfsinfo.mod
  │ └── zfs.mod
  └── locale

My grub.cfg

Code: 
set default='0'
set timeout='0'

menuentry 'HDD1' {
  set root=hd1
  chainloader +1
}

Would be set root=hd0 in your case (your ls command says so)
 
Sounds like a better solution. I want to try this.
So I'll just install proxmox on the SSD, then create a partion on the SD card, format it, put my /boot on the SD card, then modify grub.cfg (adding the chainloader) and do grub-install?
 
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom
Back