[SOLVED] Cloud-Init remove snippet and vm dependency

[Hai]

I've currently have a NFS share mounted to my cluster that stores cloud-init ISOs and Scripts(Snippets).
Everything works cool, I can create a template on my main cluster. I'am attaching the vendor script to my template aswell to install qemu-ga on first boot.

$qm set 9003 --cicustom "vendor=cloud-init-iso-manager:snippets/qemuagent-onboot.yaml"

#the content of qemuagent-boot.yaml

#cloud-config
runcmd:
    - apt update
    - apt install -y qemu-guest-agent
    - systemctl start qemu-guest-agent
    - reboot
ssh_pwauth: True

The issue is when i for example have my NFS share down, i cannot start the VMs that were previously cloned from my template.
The documentation says:

The custom config files have to be on a storage that supports snippets and have to be available on all nodes the VM is going to be migrated to. Otherwise the VM won’t be able to start. For example:

I actually only need this script to run on the first boot. Is there really no workaround to remove the dependency from the snippet and VM? Maybe there is a possibility to include the script inside the template?

I do want to clone VMs and have them completely independent.

 

[jsterr]

You can inject qemu-guest-agent with libguestfs-tools for example, so it is inside your image before deploying it. Example:

virt-customize -a /mnt/pve/cephfs/template/iso/jammy-server-cloudimg-amd64.img --install qemu-guest-agent

 

[bbgeek17]

I actually only need this script to run on the first boot. Is there really no workaround to remove the dependency from the snippet and VM? Maybe there is a possibility to include the script inside the template?

What you need to realize is that the snippets are packaged as ISO and attached as cdrom each time a VM is powered on. There are sections of Cloud Init that will be executed on each boot, not just first one. Proxmox does not have a way to know that your particular snippet is no longer needed. The logic in VM startup will try to find the specified snippet every time your start a VM and if its missing, the start will fail, as you found out.
Your options are:
- after initial boot disable cloud-init or remove the portions that are no longer needed, so you dont depend on external NAS
- wrap the start into a script that copies the snippet to local storage and then starts VM
- inject the snippet and not just the agent, as @jsterr mentioned. I think the cloud-init config needs to be in /etc. Consult the Cloud Init documentation.
- dont shutdown your NFS share
- build an Openstack like metadata service that provides CloudInit over the network
- manually build CloudInit ISO and attach it to VM, so PVE is not trying to rebuild it.

And perhaps a few others. But as long as "qm config ID" refers to a snippet file, the VM wont cold start if its missing.

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[Hai]

I've taken both of your advices and injected cloud-config script into the image, which installs the guest agent, and allows pwauth login.

Inside My NFS server I've made a copy of the original image so I could have custom image with the script injected:

admin@cloud-init-iso-manager:/mnt/cloudiso/template/iso$ cp jammy-server-cloudimg-amd64.img jammy-server-cloudimg-amd64-with-script-guestagent-onboot.img

Then using the libguestfs-tools injected the Script into ISO:

$sudo virt-customize -a /mnt/cloudiso/template/iso/jammy-server-cloudimg-amd64-with-script-guestagent-onboot.img --run-command 'mkdir -p /etc/cloud && echo "#cloud-config
runcmd:
    - apt update
    - apt install -y qemu-guest-agent
    - systemctl start qemu-guest-agent
    - reboot
ssh_pwauth: True" > /etc/cloud/cloud.cfg.d/qemuagent-onboot.cfg'

And now the VM became independent of my NFS share, as it lost dependency to snippet which in this case is not needed at all. Thank you jsterr and bbgeek17 for your suggestions!