Proxmox installation via PXE: solution

[Krokodox]

Hi there,

I would like to contribute to Proxmox with a simple(?) how-to that makes it possible to install PVE 2.0 via PXE.



PREREQUSITES

The following needs to be up and running for any kind of PXE + iPXE / gPXE installation, the setup of these services are specific for your environment, make sure that they all work as expected:

1. DHCP Server (see http://ipxe.org/howto/chainloading and http://ipxe.org/howto/dhcpd for how-to)
2. TFTP Server, I assume that it uses the "/var/www/boot/" directory as it's root
3. iPXE (http://ipxe.org/download)
4. HTTP Server, I assume that it uses the "/var/www/" directory as it's root
5. Files from the Syslinux project (http://www.syslinux.org/wiki/index.p...slinux_Project), i.e. "menu.c32" in the "/var/www/boot/" directory as well as a "deafult" menu file in the "/var/www/boot/pxelinux.cfg/" directory

NEW INITRD with embedded ISO

In order to have the installer find the .ISO image it needs to be embedded in the initrd.img that will be downloaded via PXE. Also, the init script in this new initrd.img needs to be modified so that it finds and mounts the embedded .ISO image instead of searching for a physical CD.

An overview of the required steps:

1. Extract initrd.img and linux26 from the .ISO image
2. Extract initrd.img to a directory
3. Copy the .ISO image to the initrd directory
4. Modify init in the initrd directory
5. Create a new initrd.img from the directory
6. Copy the new initrd.img to the PXE server

Detailed step-by-step guide:

1. Log in to your Linux workstation, copy the Proxmox .ISO image to your working directory and rename it to "proxmox.iso" for simplicity. Extract the files "\boot\isolinux\initrd.img" and "\boot\isolinux\linux26" from the .ISO image.
   
2. Rename "initrd.img" to "initrd.org.img" and unzip it:
   
   

   # mv initrd.img initrd.org.img
   # gzip -d -S ".img" ./initrd.org.img

   Create a directory for the files in the initrd, extract initrd to this directory:
   
   

   # mkdir initrd.tmp
   # cd initrd.tmp
   # cpio -i -d < ../initrd.org

3. Copy the .ISO image to this directory:
   
   

   # cp ../proxmox.iso .

4. Modify the startup script so that it will mount the embedded .ISO image instead of searching for a physical CD.
   
   

   # nano init

   At the end of the init script it searches for a CD/DVD reader containing a physical removeable CD disc. Modify the script so that it first checks whether there is an .ISO image ("proxmox.iso") in the root of the filesystem, and if so, mounts it. The colored + bolded text is what needs to be added between the "done" and "else" statements that are already present in the script:
   
   

   [...]
       done
   
   elif [ -f /proxmox.iso ]; then
       echo "found proxmox cdrom ISO image"
       echo "mounting /proxmox.iso image"
       mount -t iso9660 -o loop /proxmox.iso /mnt
       echo "mounting ISO done"
   
   else
   
       echo "searching for cdrom"    
   [...]

   Save the file and exit the editor, no more changes are needed.
   
5. Create a new initrd.img:
   
   

   # find . | cpio -H newc -o > ../initrd

   Zip this new initrd:
   
   

   # cd ..
   # gzip -9 -S ".img" initrd

6. You should now have an much enlarged "initrd.img" file containing all original files from the original "initrd.org.img" as well as the .ISO image and the modified init script. Copy it to your HTTP server together with the "linux26" file:
   
   

   # mkdir /var/www/boot/proxmox
   # cp linux26 /var/www/boot/proxmox/
   # cp initrd.img /var/www/boot/proxmox/

   Make sure that these files can be served via HTTP, i.e. test to download from a browser!

PXE MENU

I assume that your TFTP server's root is "/var/www/boot/" and that it contains the "menu.c32" file from the Syslinux project. You need to have your DHCP Server setup so that it serves iPXE or gPXE to your PXE enabled host and have iPXE / gPXE read the "pxelinux.cfg/default" file from the TFTP server's root.

Modify "/var/www/boot/pxelinux.cfg/default" so that it looks something like this:

UI http://<HTTP Server name or IP>/boot/menu.c32
PROMPT 0
DEFAULT Proxmox


MENU TITLE ..:: My PXE Menu ::..
MENU CLEAR


LABEL Proxmox
    MENU LABEL Proxmox
    LINUX http://<HTTP Server name or IP>/boot/proxmox/linux26
    APPEND vga=791 video=vesafb:ywrap,mtrr ramdisk_size=16777216 
    INITRD http://<HTTP Server name or IP>/boot/proxmox/initrd.img splash=verbose

Now everything should be setup for a successful installation of Proxmox VE via PXE to any PXE enabled host. These changes in the init file of the initrd.img image could perhaps be implement into the distribution of Proxmox, it would certainly make it easier to setup a PXE install environment



Either way, thank you for a very, very good product. keep up the good work!

[dietmar]

These changes in the init file of the initrd.img image could perhaps be implement into the distribution of Proxmox, it would certainly make it easier to setup a PXE install environment

OK, will include such test in the next release.

[Kjeld Flarup]

I tried your instructions, but my kernel panic'ed with this message

VFS: Cannot open root device "<NULL>" or unknown-block(8,1)
Please append a correct "root=" boot option; here are the available partions:
Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(8,1)

Is there something in your setup which I also need to know.
I'm trying to run this on a HP Proliant Blade with 64 bit Xeon

[Krokodox]

That's odd, I did not encounter anything like that. My setup is with DELL PowerEdge R200 Servers / Quadcore / 4 GB RAM / QLA2xxx FC HBA / no local drives.

What you could try is to boot the original initrd.img extracted from the original .ISO and see if you get the same error message.

If the original initrd.img downloads / boots OK and halts with a message stating that it can not find a CD then your PXE / iPXE / TFTP / DHCP, etc. setup is OK. If not then the problem is somewhere in your TFTP / HTTP / DHCP / iPXE / Syslinux setup. Check for more recent versions, try PXE booting something else and see that it works, etc.

If you get the original initrd.img booting from PXE then the problem is with the way your new initrd.img was created. Try to build one without editing the init file but with the .ISO image embedded. If this boots then you can manually mount the image as per the instructions above, i.e.:

mount -t iso9660 -o loop /proxmox.iso /mnt

and start the setup:

chroot /mnt sbin/unconfigured.sh

Happy hunting!

[Maxnet]

We have also experienced issues with iPXE and large image files on some systems, while it does work fine on others.
E.g. Vmware vSphere 5 also comes with a very large ramdisk.
When you try to pxe boot that on a problematic system, iPXE/syslinux either fails with an "out of resources" error during downloading the file (even though the system has 8 GB memory), or it crashes shortly after booting.
Not sure if it is an actual iPXE bug, or if it's an external issue like a bug in the BIOS or the network card's firmware, but fact is that large images simply do not work properly on some systems.

A more reliable way to install Proxmox 2 over the network, is to install Debian Squeeze first, and then install the Proxmox packages.

[HarusKG]

Hello. I got same issue as Kjeld Flarup. I used VE ver. 2.1. I put ISO file into initrd.img by instruction and did not edit the script, tried initrd.img with iso and without it (original image) and got same error. Tried to boot by PXE VirtualBox guest and my workstation, all the same. Need some help here =)


Best regards, Konstantin.

[Pilat]

Krokodox, thank you for the excellent instruction. I made ​​a small modification to it. Here are my results:

This is instruction for modification of Debian Squeeze standard PXE network install (http://www.debian.org/releases/stabl...h04s05.html.en).

menu.c32 from first post replaced by a standard vesamenu.c32, initrd.img and linux26 are loaded from a TFTP server.

initrd.img and linux26 prepared as written in the first post.

10.10.10.10 - dhcp, tftp, dns server address. They may be different, but I have everything on one server.

10.10.1.1 - default router

Set up a DHCP server, configure the address pool

root@dhcp:/srv# aptitude install dhcp3-server
root@dhcp:/srv# nano /etc/dhcp/dhcpd.conf:
...
root@dhcp:/srv# cat /etc/dhcp/dhcpd.conf:

option domain-name "local";
option domain-name-servers 10.10.10.10;
option subnet-mask 255.255.0.0;

subnet 10.10.0.0 netmask 255.255.0.0 {
    pool {
            range 10.10.1.2 10.10.1.200;
            next-server 10.10.10.10;
            filename "pxelinux.0";
            option routers 10.10.1.1;
            option domain-name-servers 10.10.10.10;
    }
}

Set up a TFTP server.

Read http://www.debian.org/releases/stabl...ru#where-files,
download and unpack netboot.tar.gz

root@dhcp:/srv# aptitude install tftpd-hpa
root@dhcp:/srv# cd /srv/tftp
root@dhcp:/srv/tftp# wget http://http.us.debian.org/debian/dists/squeeze/main/installer-amd64/current/images/netboot/netboot.tar.gz
root@dhcp:/srv/tftp# tar zxf netboot.tar.gz

root@dhcp:/srv/tftp# ls -la
total 9180
drwxr-xr-x 3 root root    4096 May  8 21:19 .
drwxr-xr-x 3 root root    4096 Jun  2 22:32 ..
drwxr-xr-x 3 root root    4096 May  8 21:19 debian-installer
-rw-r--r-- 1 root root 9366606 Jun  2 22:35 netboot.tar.gz
lrwxrwxrwx 1 root root      33 Jun  2 22:37 pxelinux.0 -> debian-installer/amd64/pxelinux.0
lrwxrwxrwx 1 root root      35 Jun  2 22:37 pxelinux.cfg -> debian-installer/amd64/pxelinux.cfg
-rw-r--r-- 1 root root      69 May  8 21:19 version.info

Put on the TFTP server previously created files:

cp initrd.img /srv/tftp/
cp linux26 /srv/tftp/

Edit /srv/tftp/debian-installer/amd64/pxelinux.cfg/default.

This is a standard file from the distribution Debian, need to comment out the line "include debian-installer/amd64/boot-screens/menu.cfg"
and add a boot Proxmox, as described in the first post. The only difference is that the files are taken not from the HTTP, and from a TFTP server.

root@dhcp:/srv/tftp# cat /srv/tftp/debian-installer/amd64/pxelinux.cfg/default

# D-I config version 2.0
#include debian-installer/amd64/boot-screens/menu.cfg
default debian-installer/amd64/boot-screens/vesamenu.c32

PROMPT 0
DEFAULT Proxmox

MENU TITLE ..:: My PXE Menu ::..
MENU CLEAR

LABEL Proxmox
    MENU LABEL Proxmox
    LINUX /linux26
    APPEND vga=791 video=vesafb:ywrap,mtrr ramdisk_size=16777216
    INITRD /initrd.img splash=verbose

Reatart DHCP server...

root@dhcp:/srv/tftp# /etc/init.d/isc-dhcp-server restart

Stopping ISC DHCP server: dhcpd failed!
Starting ISC DHCP server: dhcpd.

Now you can load the server to install Proxmox - it should boot from PXE

[Kjeld Flarup]

I tried your instructions, but my kernel panic'ed with this message

VFS: Cannot open root device "<NULL>" or unknown-block(8,1)
Please append a correct "root=" boot option; here are the available partions:
Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(8,1)

Is there something in your setup which I also need to know.
I'm trying to run this on a HP Proliant Blade with 64 bit Xeon

got past this problem with a cfg from the ubuntu installer
LABEL Proxmox
MENU LABEL Proxmox
LINUX /proxmox/linux26
append vga=normal initrd=/proxmox/initrd.img -- quiet

Now it aborts after

Detecting network settings... done
\nInstallation aborted - unable to continue

Not really much information to go on there

[6opo9a]

Why do not use the original iso with the memdisk?

[Kjeld Flarup]

It is a lot easier to deploy in data centers with PXE

[mrballcb]

got past this problem with a cfg from the ubuntu installer
LABEL Proxmox
MENU LABEL Proxmox
LINUX /proxmox/linux26
append vga=normal initrd=/proxmox/initrd.img -- quiet

Now it aborts after
>> Detecting network settings... done
>> \nInstallation aborted - unable to continue
Not really much information to go on there

I experienced that same error this morning when I was setting up PXE boot for Proxmox 2.1. That happens because the first thing the installer tries to do is start up X, and since you've told it to go into a lower res mode than what the installer wants, it just aborts. (I agree it would be nice if it would explain that it couldn't start up X.) You need to use the correct vga and video parameters for X to be able to start. It would also help for you to include splash=verbose because it's a good indicator of things working properly when you see actual output from the kernel and the splash screen.

I too built the new initrd.img as described in the original post with the iso included. I named it "initrd.iso.img" to make it clear to other humans who might be looking at it that it serves a different purpose from the default proxmox initrd.img (as if you couldn't tell from the much larger size). This is what my pxelinux.cfg/default looks like:

label proxmox
   kernel proxmox/linux26
   append initrd=proxmox/initrd.iso.img ramdisk_size=524288 vga=791 video=vesafb:ywrap,mtrr splash=verbose
#   append initrd=proxmox/initrd.iso.img ramdisk_size=524288 proxdebug

I'll also note that some others have been using some large numbers for ramdisk_size. That value is the number of KB, so in my example above, 524288 KB => 512 MB, which is plenty large enough for the initrd with the iso to be expanded:

[root@dump02 build]# du -sh initrd.tmp/
410M    initrd.tmp/

[mrballcb]

I wrote a little script to be able to do this quickly in the future. Sample output:

[root@dump02 /]# build_proxmox_iso_initrd.sh /download/Proxmox/
Using proxmox-2.1.iso
Mounting iso image
Unmounted iso, extracting contents of initrd...
Added iso, creating and compressing the new initrd...
Cleaning up temp files...
Done!  Look in /download/Proxmox/pxeboot for pxeboot files.

And here's the script:

#!/bin/bash

BASEDIR=${1:-./}
pushd $BASEDIR >/dev/null

[ -L "proxmox.iso" ] && rm proxmox.iso &>/dev/null

for ISO in *.iso; do
    if [ "$ISO" = "*.iso" ]; then continue; fi
    if [ "$ISO" = "proxmox.iso" ]; then continue; fi
    echo "Using $ISO"
    ln -s "$ISO" proxmox.iso
done
if [ ! -f "proxmox.iso" ]; then
    echo "Couldn't find a proxmox iso, aborting."
    echo "Add /path/to/iso_dir to the commandline."
    exit 2
fi
[ -d pxeboot ] || mkdir  pxeboot
pushd pxeboot >/dev/null
[ -d mnt ] || mkdir  mnt
echo "Mounting iso image"
if ! mount -t iso9660 -o ro,loop ../proxmox.iso mnt/ ; then
    echo "Failed to mount iso image, aborting."
    exit 3
fi
cp  mnt/boot/isolinux/linux26 .
rm -f  initrd.orig initrd.orig.img initrd.img
cp  mnt/boot/isolinux/initrd.img initrd.orig.img
if ! umount  mnt ; then
    echo "Couldn't unmount iso image, aborting."
    exit 4
fi
echo "Unmounted iso, extracting contents of initrd..."
gzip -d -S ".img" ./initrd.orig.img
rm -rf initrd.tmp
mkdir  initrd.tmp
pushd initrd.tmp >/dev/null
echo "Added iso, creating and compressing the new initrd..."
cpio -i -d < ../initrd.orig 2>/dev/null
cp ../../proxmox.iso proxmox.iso
(find . | cpio -H newc -o > ../initrd.iso) 2>/dev/null
popd >/dev/null
rm -f initrd.iso.img
gzip -9 -S ".img" initrd.iso

# Now clean up temp stuff
echo "Cleaning up temp files..."
rmdir  mnt
rm -rf initrd.tmp
rm  ./initrd.orig

echo "Done!  Look in $PWD for pxeboot files."
popd >/dev/null
popd >/dev/null

[6opo9a]

It is a lot easier to deploy in data centers with PXE

I know. I meant using pxe to load pxelinux to load memdisk with original iso-installer. Or do you want to make diskless nodes?

[mrballcb]

That's odd, I did not encounter anything like that. My setup is with DELL PowerEdge R200 Servers / Quadcore / 4 GB RAM / QLA2xxx FC HBA / no local drives.

I wanted to add that I am using Dell PowerEdge R210's, but with 8 GB of RAM and local disks. The Dell machines just seem to work with little to no issues, I just make sure that the virtualization features of the CPU's are enabled in the BIOS.

[delicatepc]

And here's the script:

#!/bin/bash

BASEDIR=${1:-./}
pushd $BASEDIR >/dev/null

[ -L "proxmox.iso" ] && rm proxmox.iso &>/dev/null

for ISO in *.iso; do
    if [ "$ISO" = "*.iso" ]; then continue; fi
    if [ "$ISO" = "proxmox.iso" ]; then continue; fi
    echo "Using $ISO"
    ln -s "$ISO" proxmox.iso
done
if [ ! -f "proxmox.iso" ]; then
    echo "Couldn't find a proxmox iso, aborting."
    echo "Add /path/to/iso_dir to the commandline."
    exit 2
fi
[ -d pxeboot ] || mkdir  pxeboot
pushd pxeboot >/dev/null
[ -d mnt ] || mkdir  mnt
echo "Mounting iso image"
if ! mount -t iso9660 -o ro,loop ../proxmox.iso mnt/ ; then
    echo "Failed to mount iso image, aborting."
    exit 3
fi
cp  mnt/boot/isolinux/linux26 .
rm -f  initrd.orig initrd.orig.img initrd.img
cp  mnt/boot/isolinux/initrd.img initrd.orig.img
if ! umount  mnt ; then
    echo "Couldn't unmount iso image, aborting."
    exit 4
fi
echo "Unmounted iso, extracting contents of initrd..."
gzip -d -S ".img" ./initrd.orig.img
rm -rf initrd.tmp
mkdir  initrd.tmp
pushd initrd.tmp >/dev/null
echo "Added iso, creating and compressing the new initrd..."
cpio -i -d < ../initrd.orig 2>/dev/null
cp ../../proxmox.iso proxmox.iso
(find . | cpio -H newc -o > ../initrd.iso) 2>/dev/null
popd >/dev/null
rm -f initrd.iso.img
gzip -9 -S ".img" initrd.iso

# Now clean up temp stuff
echo "Cleaning up temp files..."
rmdir  mnt
rm -rf initrd.tmp
rm  ./initrd.orig

echo "Done!  Look in $PWD for pxeboot files."
popd >/dev/null
popd >/dev/null

I used a Linux Mint Live boot CD (which I booted over PXE) to create the resulting files of the script. This method works. Tested on a virtualbox VM.

MANY THANKS TO mrballcb for this.

My entries in my "menu cfg":

LABEL -    
MENU LABEL Linux Mint 12 Live CD
    KERNEL linux/mint12/casper/vmlinuz
    APPEND root=/dev/nfs boot=casper netboot=nfs nfsroot=10.9.9.10:/export/tftpboot/linux/mint12 quiet splash --
    INITRD /linux/mint12/casper/initrd.lz
    
LABEL -    
MENU LABEL ProxMox (latest)
    KERNEL linux/proxmox/linux26
    APPEND ramdisk_size=524288 vga=791 video=vesafb:ywrap,mtrr splash=verbose
    INITRD linux/proxmox/initrd.iso.img

Would it be possible to have a section on ProxMox downloads called PXE which would be basically zips containing the two needed files (linux26 and initrd.iso.img)? Im sure the community will be happy to generate and upload them as needed. It took me 10 minutes to do this.... and redoing it will take 5.

-dpc

[mrballcb]

I just wanted to mention that the script creates a PXE bootable initrd for 2.2 as well, since I just ran it, deployed the images on our PXE servers, and tested the PXE boot. I did, however, make one small change to the script to handle multiple versions of proxmox iso that are found:

if [ -L "proxmox.iso" ]; then
rm -f "proxmox.iso";
fi
ln -s "$ISO" proxmox.iso

Alternatively, you could have just changed the "ln -s" to "ln -sf", but that seems like the sloppy way to fix it.

...Todd

[6opo9a]

Thank you! Very useful topic!

[mrballcb]

I know. I meant using pxe to load pxelinux to load memdisk with original iso-installer. Or do you want to make diskless nodes?

Everything written below is wrong. Ignore it. ...Todd

I just wanted to follow up for anybody who stumbles across this thread. You do not *NEED* to run the script I posted earlier to extract the kernel/initrd and rebuild it with the iso in it. User 6opo9a's question above was such a quick reply that I read right past it. What he is implying, and I am confirming, is that it's easier to just use memdisk to boot the iso directly.

PXE boot has a utility named memdisk which provides a virtual floppy/hd/iso mount capability. To use it, load memdisk as the kernel, and then specify the proxmox iso as the initrd (must be located in the /tftpboot file structure, you can't reference a file outside there since tftpd _normally_ chroots) in the append line. You need to also add "iso" to the end of the line so that it knows which type of file it's mounting. For reference, see the specs at http://www.syslinux.org/wiki/index.php/MEMDISK.

This is what my pxelinux.cfg/default configuration looks like:

label proxmox
   kernel memdisk
   append initrd=proxmox/2.2/proxmox.iso iso

label proxmox22
   kernel proxmox/2.2/linux26
   append initrd=proxmox/2.2/initrd.iso.img ramdisk_size=524288 vga=791 video=vesafb:ywrap,mtrr splash=verbose
# Debug mode
#   append initrd=proxmox/2.2/initrd.iso.img ramdisk_size=524288 proxdebug

label proxmox21
   kernel proxmox/2.1/linux26
   append initrd=proxmox/2.1/initrd.iso.img ramdisk_size=524288 vga=791 video=vesafb:ywrap,mtrr splash=verbose
# Debug mode
#   append initrd=proxmox/2.1/initrd.iso.img ramdisk_size=524288 proxdebug

[6opo9a]

Sorry, but using the only memdisk doesn't work, because after the kernel is loaded it "forgets" about a fake cdrom. More details are here.

[mrballcb]

Sorry, but using the only memdisk doesn't work, because after the kernel is loaded it "forgets" about a fake cdrom. More details are here.

Ugh, I hate it when I'm that wrong. I had just booted it to the cd boot screen and assumed all was well. Thanks for the correction. So you DO have to still run the script to generate the special initrd.

[6opo9a]

Thats ok, thx for your nice script! =)