Where to find basic understanding of Clones, Templates and -snapshot

[imikejackson]

I am wanting to setup some VMs for CI builds. At this point I have a "generic" VM for both Ubuntu and Windows setup and ready to go. We use Azure and an Azure Agent to look for PRs and Merges on our GitHub repositories. Each VM needs to register its own unique Azure Agent. This is easy enough by running a few scripts inside of the VM to set things like host name, agent name and get the agent fired up and looking for a build. At the end of the build I need the VM to shutdown (not reboot), any changes to the VM disk are discarded, and then the VM should actually restart itself. My plan on this node was to create a VM (100) and then convert that to a template (101). Now create several clones from the templates (say these are VMs 102 and 103) where those VMs get the needed customizations. At this point I should hand edit the vm conf file for 102 and 103 to add the "args: -snapshot". Inside the VM, the Azure agent script uses the "--once" flag to start which allows the agent to do a single build and then the agent exits at which point you can do "shutdown -h now". The VM turns itself off and all those changes go away.

Great. How do I get the VM to turn itself back on? Is that done with, oddly, a Shutdown hook?

Another question. How do I actually "Update" the VM 101 with new OS updates and such? Seems like I have to convert it back to a full clone, do the update, then make a template back out of it? But I feel like VMs 102 and 103 will not lose their connection to the original VM disk?

Do I just skip the template part and manage this some other way?

I also would like to do this same thing, but on the 5 other Proxmox servers that I have going?

If a kind soul could point me in the right direction that would be fantastic. Yes, YouTube is also my friend for some of this but not finding anything specific to my needs.

 

[bbgeek17]

Sounds like a custom private cloud, and frankly overcomplicated.

I would:
- create a cloud image. Either base it on one of the official images or build your own
- create an appropriate Custom CloudInit that takes care of all customization, including shutdown if necessary
- when a VM is needed it is cloned from the custom template you created and Custom cloud-init is applied during boot
- after the task is complete the VM is destroyed
- when updates are needed, either update the original qcow and redo the template, or use another method to inject the changes.

We drive our CI/CD on Openstack with 100s of VMs created and destroyed every hour. Multiple Proxmox Clusters are some of these VMs. I realize that this is Proxmox forum, but things work too well to invest into transitioning to Proxmox for this particular purpose.

Best

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[imikejackson]

@bbgeek17
Thank you for the pointers. I will look up the information about "cloud-init" images and creating a custom image that has what we need.

I am curious about the statement "when a VM is needed ... " part. How do you manage that? Something would have to be registered with Azure to be able to understand that a VM is needed. I feel like that is something that would need to be handled on the proxmox server side rather than the VM side of things?

Typically, the Azure Agent gets fired up on startup of the VM and registers itself.

 

[bbgeek17]

I feel like that is something that would need to be handled on the proxmox server side rather than the VM side of things?

Proxmox is a Hypervisor, not a CI/CD controller. You need some external driver that talks to PVE via API/Ansible/Puppet/Terraform/whatever. Its not Proxmox' job to know when you need a VM and whether it should be customized, up or down.
I dont know what you are doing with Azure, but sounds like you need a dedicated controller (either a PVE VM or Azure VM) that orchestrates things.

Good luck

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[imikejackson]

We run an "Azure Agent" on the VM that gets the signal from Azure itself that a build needs to happen.

I've settled on getting the VM completely setup and working. Within the VM, the script that is run on startup that runs the Azure Agent, the last command is to shutdown the VM. The on the ProxMox side, add the VM to the HA group to always make sure it is restarted. Also I then use the "-snaphot" added to the configuration file to put the VM back into the "prebuild" state.

There are a lot of solutions to this and all the different server side technologies are not in my wheel house. I think between the HA and snapshots I can sort of manage the VMs. We will only have like 12 VMs to manage so tolerable but not great.