Automated Installer with 8.2 - how to actually serve answer from HTTP?

[stv.Bundesadmin]

Hi there,
dear Proxmox Team,

TLDR: it is totally confusing that the installer SENDS JSON data instead of simply fetching a regular answer file! No idea how i can use that... i'm not devops...

IMHO, the documentation for this new feature is lacking critical information.
Like a lot of Proxmox documentation, it's more of "here is what's available", but not really giving easy to grasp answers to "Use-Case xyz".

I am at a loss on how to serve the answer .txt-file - because i simply can't provide one from HTTP, right?

I put my validated auto.txt on that webserver and prepare the ISO with
proxmox-auto-install-assistant prepare-iso --fetch-from http --url http://10.10.10.10/pxtest/auto.txt proxmox-ve_8.2-1.iso

But that won't work, because only after digging through webserver logs i find out that it's doing a POST instead of a GET
Well, yes, it's mentioned in the docs - in a single line, without any further explanation.
The HTTP(S) POST request sends JSON data that can help to identify the physical machine and use that information to generate a custom answer file.
That is totally unexpected!

I 100% expected it to fetch a file.
Now I have no idea how to handle receiving a JSON Post and then somehow dynamically sending back a valid answer.
I'm a regular sysadmin, never had to deal with writing my own webserver and delivering code back to requests like that...


I guess i understand why - we need a dynamic system to deliver different data per machine (like matching MAC to a static IPs) without replacing the text file every few seconds or generating seperate ISOs for each host ;-)
But there is only a single search-hit for "POST" in https://pve.proxmox.com/wiki/Automated_Installation - that is just not enough to use it.


This needs thorough examples!
What software should we use as webserver/JSON-recipient? How do we code dynamic answers? ...
I'd actually expect that to be put in a single .txt file - with the filter-matching, "if x, then y", so you could add all youe servers in there and have it statically served.

 

[Max Carrara]

Hi there!

Thank you for voicing your concern - that is definitely something we want to document.

I'm in the process of writing such an HTTP server and providing it as an example. I have in fact written one for my own testing purposes, but it's not really "pretty" enough to be used by others yet. I'll let you know once the example server has been added to our docs.

In your case, if you just want to serve a static file like you described, you can just respond to the POST request with the file's contents. So, you don't need to read the JSON data of the request at all, if that's what you want. You can find an example below.

Do you have a couple other wishes and suggestions that I should add? At the moment I've got the following in mind:

• A section that describes in more detail how fetching the answer file via HTTP works
• Multiple example servers
  • A very simple server that just serves a static file
  • A more advanced server that sends an answer file depending on the host's MAC address (should be easily configurable!)
  • ...

Below is an example for a simple HTTP server that does what you want until the documentation is updated.

---

Simple Example Server​

Edit: This example can now be found in our wiki.

 

[stv.Bundesadmin]

Thank you so much @Max Carrara! That's exactly what i'd have wished for from the beginning, very glad you picked up on this so quickly!
I'll try to get your simple example running while waiting for the proper solution.


> Do you have a couple other wishes and suggestions that I should add?
Well, instead of having a complicated JSON POST/complicated Webserver solution, i think a way more accessible solution would be to have a single static file with "groups". (Those who want to automate, could automatically generate this file instead of dynamic webserver configs?)
=> Way less overhead in getting it started and maintaining it!


Dummy code for a "multipurpose static answer.toml" - the installer.iso would need to be able to deal with it:

## Really global settings
[global]
keyboard = "de"
country = "de"
root_password = "dummy123456"
root_ssh_keys = [
    "ssh-ed25519 AAA...]

## per server matching
if (filter.xyz = "MAC:MAC:MAC:MAC")
    
    [global]
    fqdn = "prod01.domain.internal"
    root_password = "specific123456"
    
    [network]
    static-ip-goes-here
    
    [disk-setup]
    filesystem = "ext4"
    filter_match = "all"
    filter.ID_SERIAL_SHORT = "drive-scsi0"
    filter.ID_BUS = "scsi"
fi

if (filter.xyz = "MAC:MAC:MAC:MAC")
    
    [global]
    fqdn = "dummy-test02.domain.internal"
    root_password = "test123"
    
    [network]
    source = "from-dhcp"
    
    [disk-setup]
    filesystem = "ext4"
    filter_match = "all"
    filter.ID_SERIAL_SHORT = "drive-scsi0"
    filter.ID_BUS = "scsi"
fi

 

[Max Carrara]

Thank you so much @Max Carrara! That's exactly what i'd have wished for from the beginning, very glad you picked up on this so quickly!
I'll try to get your simple example running while waiting for the proper solution.

You're welcome, I hope it all works out!

Well, instead of having a complicated JSON POST/complicated Webserver solution, i think a way more accessible solution would be to have a single static file with "groups". (Those who want to automate, could automatically generate this file instead of dynamic webserver configs?)
=> Way less overhead in getting it started and maintaining it!

Ah, I see what you mean now. The TOML file format doesn't support such syntax explicitly, but I'm sure that we could come up with something in that regard; I can see how that would be useful.

Thank you for your feedback! We'll see what we can do - this is definitely something we're going to have a look at.

 

[NatanBackwards]

I developed an application that might be useful to you. It serves up multiple answer variations based on system information. I just made it so I am sure it has some bugs/improvements that can be made. Let me know what you think. I am still working out how to incorporate some more advanced matching mechanisms for the dynamic answers. It can be easily installed to Docker and there is also an Ansible script to install to an LXC.

https://github.com/natankeddem/autopve

 

[Max Carrara]

A short update for the curious: We've now added a couple examples in our wiki, including one that allows you to set up hosts depending on their MAC address. The example I had posted above has been moved there as well.

@NatanBackwards' fantastic application is listed there too. Great work!

 

[NatanBackwards]

A short update for the curious: We've now added a couple examples in our wiki, including one that allows you to set up hosts depending on their MAC address. The example I had posted above has been moved there as well.

@NatanBackwards' fantastic application is listed there too. Great work!

Wow, thank you so much for the kind words and linking to my application. I should have an improved matching system pushed live this weekend. I am also working on getting some test coverage setup.

 

[filou59]

Hello NatanBackwards

Your work is very cool.
It took me longer to discover how Ansible worked, but since I've been wanting to get started for a long time, it was the perfect opportunity.

If I may, a small improvement that would make life easier would be to be able to copy an existing profile, so as not to start from 0.

Otherwise in the meantime we can edit the storage-general.json file located in /root/data

systemctl stop autopve.service
nano /root/autopve/data/storage-general.json
systemctl start autopve.service