Deploying multiple proxmox host easily?

[strandvaskeren]

Hi.

I teach at a tech college. One of our courses is in building cluster networks with proxmox, ceph and all that jazz. At the start of the course each student is assigned a physical rack server in our server room, preinstalled with proxmox. Each student typically starts out by installing 3 to 5 nested proxmox vm's and try their luck with clustering those, migrating vm's between the nested nodes, try out ceph and so on.

Once they have the hang of it, the students then group up and try the same stuff directly on the physical servers, reinstall the physical servers to suit their needs, configure the storage switches and stuff like that.

Basically they get to play around with a bunch of rack servers and get hands on experience with clustering, it's a lot of fun.

The course is 4 weeks and at the end of the course we are left with some happy students and a couple of racks filled with servers with all sorts of configurations.

This means that every 4 weeks us teachers are left with the task of doing a fresh proxmox install on 25 physical servers and even though installing proxmox from scratch isn't that big of a task, it gets really old doing it 25 times every 4 weeks.

We already use a fog server to roll out images to the classroom pc's after each course and was hoping to be able to roll out an image with a fresh proxmox install on the rack servers the same way, but so far it has defeated us. Currently we are trying our luck with rolling out a debian server install and then upgrading that to proxmox, but it still needs some work before it's done.

Our dhcp server is set up with mac address bindings so each rack server gets the same ip each time and the dhcp also offers a hostname to the servers.

My question is, how would you guys go about installing proxmox on 25 physical servers every 4 weeks?

 

[gurubert]

I would also assume that automating a Debian installation and then adding the Proxmox repos and packages would be the easiest way.

Have you looked at https://fai-project.org/ ?

 

[cheiss]

Hi,

since the Proxmox VE 8.2, the installation ISO features an Automated Installation!
I think this should cover your needs, although you will still need a tool like e.g. Ansible to automate any custom changes after, if you need them.

 

[strandvaskeren]

Sorry for the delay in replying, didn't find the time to play around much until now.

I'm trying my luck with the Automated Installation but I find the documentation linked a bit lacking regarding which options are available. Are there a more complete documentation available?

Currently, following the examples, I've managed to create a bootable usb from a proxmox iso and an answer.toml, so far so good. Booting from the usb auto installs proxmox.

However, is there an option to have it grab the fqdn from the dhcp server? I already have the mac address and a hostname set up on my dhcp server, so if for example I install a debian server the hostname pops up as a suggestion from the dhcp server, would be cool if there were a similar option in the answer.toml, for example fqdn = "from-dhcp" or something like that.

Also is there an option to have the server shut down after the automatic install rather than rebooting? The servers I'm installing on are headless and set to boot from usb as the first option and the regular boot drive as a second, so I can pop in a usb and restart the server when I want a fresh install, but once it's done installing it reboots and start another install. I would love an option to do shutdown rather than a reboot after the install is completed - that way I can pop a usb install stick in each server, power on the servers and wait a couple of minutes for them to install and shut down. I can then remove the usb sticks and power on the servers again.

 

[cheiss]

but I find the documentation linked a bit lacking regarding which options are available. Are there a more complete documentation available?

Could you elaborate? The linked documentation already lists every possible option of the answer file in the answer file format section.

However, is there an option to have it grab the fqdn from the dhcp server?

No, but it's already filed and will be possible in the future: https://bugzilla.proxmox.com/show_bug.cgi?id=5811
Currently, this can be done by supplying a customized answer file for each host if done via HTTP, see the Answer fetched via HTTP section & examples.

Also is there an option to have the server shut down after the automatic install rather than rebooting?

No, but also seems somewhat obscure. Feel free to file a bug request over at our Bugzilla tho, were we keep track of such things (and can discuss them more, if needed).

I can pop in a usb and restart the server when I want a fresh install, but once it's done installing it reboots and start another install

That sounds more like simply a wrong boot order. Are those server using legacy BIOS or UEFI? At least with UEFI, Proxmox VE should correctly itself as first in the boot order after installation.

 

[strandvaskeren]

First, thank you for the answer.

Could you elaborate? The linked documentation already lists every possible option of the answer file in the answer file format section.

I didn't know it was the complete documentation, I was hoping for a couple of more options.

No, but it's already filed and will be possible in the future: https://bugzilla.proxmox.com/show_bug.cgi?id=5811
Currently, this can be done by supplying a customized answer file for each host if done via HTTP, see the Answer fetched via HTTP section & examples.

Okay, but it seems a bit meh to have to go through a bunch of different answer files fetched via http when the dhcp server is already able to provide the necessary information.

No, but also seems somewhat obscure. Feel free to file a bug request over at our Bugzilla tho, were we keep track of such things (and can discuss them more, if needed).

That sounds more like simply a wrong boot order. Are those server using legacy BIOS or UEFI? At least with UEFI, Proxmox VE should correctly itself as first in the boot order after installation.

Might be obscure to you, not to me..

My use case is 25 rack servers and only a single monitor and keyboard. I would love the ability to reinstall all 25 servers without having to connect mouse and keyboard.

I can do the answer fetched via http or heck, even create and label 25 individual usb sticks, that's all fine.

Let's say I create 25 usb sticks, mount one in each of my 25 rack servers and power them all on. A couple of minutes later the automatic installation is done on all 25 servers. Proxmox changes the boot order and reboots. I remove the usb sticks and Bob's your uncle, for now..

Next time I need to reinstall the 25 servers I have to connect the single monitor and keyboard to each and every single server to change the boot order back to booting from the usb. Rebooting a rack server and changing the boot order is at least a 7-8 minute task - times 25, so I would really love an option to tell proxmox to leave my boot order alone and just boot from the usb first (if present) and then skip to option two and boot from the regular boot drive if no usb is attached.

So I would love an option to tell proxmox not to mess with my boot order and I would love an option to have the installer do a shutdown when done rather than a reboot - that way I would know, without attaching a monitor, when the installation was finished and thus when to remove the usb sticks.

Please ask you automated install development team to try doing repeated installs on headless systems a couple of times and it should be obvious what I'm talking about. If you have monitors and keyboards connected to the systems you do automated installs on it's a breeze, but let me know how repeat installs on those systems without a monitor works out for you.

Edit: Rereading my post I realise that I come across a bit demanding, that wasn't my intention.

 

[cheiss]

I didn't know it was the complete documentation, I was hoping for a couple of more options.

Well, it was only introduced with the latest (aka. 8.2) release in April this here.
So there is definitely room for improvement, not arguing against that!

Okay, but it seems a bit meh to have to go through a bunch of different answer files fetched via http when the dhcp server is already able to provide the necessary information.

Yes, it's more a workaround for your case. Just wanted to note the possibility of it.

Next time I need to reinstall the 25 servers I have to connect the single monitor and keyboard to each and every single server to change the boot order back to booting from the usb.

Normally, especially for racked systems, you have some sort of KVM/IPMI system to access/provide these things remotely anyway, maybe even have an API for setting the boot device - e.g. Redfish has these things in their API. That's also what we mostly tested here.

But I see where you are coming from and why it might be useful. So feel free to create a report over at https://bugzilla.proxmox.com/ (describing your usecase), where it will be properly tracked - as here in the forum things can get easily lost.

so I would really love an option to tell proxmox to leave my boot order alone

Every OS installer does the same, at least that I'm know of. It's also only possible for UEFI - but you also didn't answer what mode you're using.

Please ask you automated install development team

Well, it's mostly @aaron (who wrote the auto-installer in the first place) and me doing the development on this, so you've definitely reached the right people!

 

[aaron]

My question is, how would you guys go about installing proxmox on 25 physical servers every 4 weeks?

Check what the iLO/iDrac/IPMI of the servers offers in automation. As @cheiss mentioned, they usually have APIs and some might also allow access via SSH. But that depends heavily on the actual server hardware and the type of out of band mgmt that is integrated.

Set up some infrastructure around it:

• Have a small machine that provides the answer files (we have a basic example in python in the docs, and at the end is a link to a community project that offers a web UI)
• Prepare answer files for each server. I assume they will always have the same basic configuration?
• network share to place the prepared ISO
• DHCP Server

Depending on your choice, prepare the ISO to either have the URL to fetch the answer file from "hard coded" in the ISO, or to fetch it via DNS or DHCP options. This highly depends on the infrastructure around it.

Then to do a fresh installation:

• on the IPMI/iLO/iDrac/… you can hopefully automate the following steps,
  • make sure they are powered off
  • attach the ISO on the network share as virtual CDROM
  • set the virtual CDROM as next boot option
  • power on
• The prepared ISO should start the automated install and fetch the answer file for that particular server
• Once the installation is done, it should reboot into the fresh Proxmox VE installation

Here it depends on how you want to proceed. Since you know the password for the root user, you can use tools like ansible to automate the next steps, like installing and configuring any other settings within Proxmox VE.

To shut them down once done, you can either use ssh (ansible or other automatiion tools help) or you could send a regular power down command via the IPMI/iLO/iDrac/… as well.

That the installer sets the new installation as the new default boot option is expected and won't change. In any other scenario, that is what is expected and wanted.

If you want to adapt the boot order, you can use efibootmgr on a running Proxmox VE instance. I would make sure that the important boot entries either have all the same ID, or same name, so it can be scripted reliably.
Or alternatively use the out of band mgmt interface, the boot order should be changeable there as well in most implementations.

 

[strandvaskeren]

Then to do a fresh installation:

• on the IPMI/iLO/iDrac/… you can hopefully automate the following steps,
  • make sure they are powered off
  • attach the ISO on the network share as virtual CDROM
  • set the virtual CDROM as next boot option
  • power on
• The prepared ISO should start the automated install and fetch the answer file for that particular server
• Once the installation is done, it should reboot into the fresh Proxmox VE installation

Going into the ilo of 25 servers and setting the boot order each time I need a fresh install is exactly the thing I'd like to avoid. Connecting to the ilo, then wait 7-8 minutes for the server to boot just so I can get to the bios and changing the next boot option is fine if it's a single server or two, but gets really tedious when it's 25 servers. Or I could use efibootmgr on the existing proxmox installs, but I rarely know what password my students have used. Either way I'm looking at hours of work messing with boot orders and stuff.

However I think I've managed to solve it by jumping through a bunch of hoops, at least it works on my home setup, I'll try it on the servers at work during this upcoming week.

At first I configured the boot order in the bios as:
1) usb
2) network boot
3) proxmox

I then used a usb stick to do an automatic install of proxmox. The install worked fine, but once the install was done it did a reboot and restarted installing from the usb - and with no monitor attached it was hard to know when to pull the usb stick.

Now my bios boot order looks like:

1) network boot
2) proxmox
3) usb

The trouble with that boot order is, that each time I'd like the server to boot from the usb, I need to manually set the next boot option on each of the 25 servers, which sucks doing as we are back to hours of work.

So as a workaround I installed debian server on a server and made a crontab job:
@reboot efibootmgr --bootnext 0001
@reboot systemctl poweroff

I grabbed an image with my fog server, which I can now deploy when needed.

So now I can power off all 25 servers using the power buttons - 2 minute job.

Next I use the fog server to roll out the debian image on all the server, the fog server does wake-on-lan and rolls out the images in a couple of minutes. Once the images have rolled out, the servers reboots into the debian image that sets the bootnext to point at the usb and then shuts the server down - letting me know the job is done - takes roughly 20 minutes to complete and I don't need to be present.

I then put a usb stick with proxmox automatic install in each server and press the power button to fire them up, the usb installer does it's job and the system reboots into proxmox. At that point I can either remove the usb sticks or leave them in, doesn't matter. Inserting usb's and pressing 25 power buttons is a 2 minute task, the installation takes maybe 15 minutes, but I don't need to be present.

However it would be SO much easier if the automatic proxmox install offered an option to do a shutdown rather than a reboot after the installation finished, that way I could leave usb boot as the primary option in bios, put a usb stick in a server, turn on the power, wait 10-15 minutes for the installation to complete and shut the server down - letting me know the installation was complete - I could then remove the usb and power up the server again.. That would be 2 minutes of shutting down the servers and adding usb sticks, 10-15 minutes of drinking a cup of coffee, then another 2 minutes of removing the usb sticks and turning on the power again.

The regular installer already has the option to remove the checkmark in "Automatically reboot after successful installation", why not offer a similar checkbox for "Automatically shutdown after successful installation" and make it an optional parameter for the answer.toml file?

 

[aaron]

Going into the ilo of 25 servers and setting the boot order each time I need a fresh install is exactly the thing I'd like to avoid. Connecting to the ilo, then wait 7-8 minutes for the server to boot just so I can get to the bios and changing the next boot option is fine if it's a single server or two, but gets really tedious when it's 25 servers. Or I could use efibootmgr on the existing proxmox installs, but I rarely know what password my students have used. Either way I'm looking at hours of work messing with boot orders and stuff.

iLO should also support the Redfish API. This way, you can interact with the servers in an automated way. Ansible has a community plugin for Redfish as well: https://docs.ansible.com/ansible/latest/collections/community/general/redfish_command_module.html

I am not sure how well it works, but with this, it should be possible to fully automate the whole process without manually going into the iLO of each individual server.

By using the iLO via the API, you should be able to map the ISO as virtual CDROM from a network share and power control them. I would definitely look into that as controlling the servers via the iLO (via the API), a script and/or ansible can do all the stuff without ever physically walking over to the servers.

However it would be SO much easier if the automatic proxmox install offered an option to do a shutdown rather than a reboot after the installation finished

That could potentially be implemented. Can you please open a feature request in our bugtracker so we don't forget about it?

 

[strandvaskeren]

A follow-up.

While the fog server roll-out idea worked great in my home environment, it didn't work well with the servers at work, controlling the boot order was a pain..

I've ended up ssh'ing into the ilo's of the servers and doing:

power off
onetimeboot usb
power on

..and that did the trick.

Some of the servers had a virtual ilo interface enabled which messed with the dhcp process, but after disabling those on each server, everything went well.

I haven't gotten around to doing an ansible script yet, but right now I can pop a usb stick into each server, then ssh into the ilo's and issuing those 3 commands, making it about 5 minutes of ssh'ing followed by about 20 minutes of waiting for the usb sticks to work their magic - which is fine.

So thank you all for your input, you've been great.

 

[hvisage]

A follow-up.

While the fog server roll-out idea worked great in my home environment, it didn't work well with the servers at work, controlling the boot order was a pain..

I've ended up ssh'ing into the ilo's of the servers and doing:

power off
onetimeboot usb
power on

..and that did the trick.

Some of the servers had a virtual ilo interface enabled which messed with the dhcp process, but after disabling those on each server, everything went well.

I am busy with a process to add the needed bootstrap (in my case FRR & zerotier) on the SuperMicro X10DRT-PS (Yes, it's the old I use in test/home going to the DC but using the current H12 AMD versions in prod too) -> the biggest pain with this device is the lack of a SHARED port between the host and BMC/ILO/iDRAC . IF that were there, then to have the ILO/iDrac/BMC in the SAME management network as the host, it eases things like PXEboot!!! and a OOB 1G RJ45 network vs the production/data plane running 10/25 SPF+/SFP28. then (as do OVH/etc.) you can always boot form the network/PXE an then just have a DB entry that makes the PXE boot config pushed to the booting server, defaults to the installation media, or the local disk(s) or even MEMtest86 etc. etc.

Now I ALSO don't have that, and the process I'm focusing on, is to use the CIFS/SMB mounted PVE-automated-installation.iso and then just do the equivalent of onetimeboot CD. Each device DO have a floppy image that gets uploaded (still need to check whether that is in RAM or flash/eeprom on the BCM...) that then contains details/configs/information specific to this host - Note, in my case the data/production plane is a mesh configuration, so I don't have a L2 with DHCP+PXEboot capable in my setup.