Question about the --cicustom (custom cloud-init) feature

[dj423]

I have been working on my process of spinning up new templates/instances from cloud images for various distros. All is going good so far. Where I am struggling is with the --cicustom switch when applying to a VM instance, it appears I am not able to seed the user-data and the network config for the same target VM. I'm sure it's just user error...

I need to pass it the user-data as well as the network config, I have tried it two ways, with both on the same line like:

qm set <vmid>  --cicustom "user=local:snippets/user-data.yaml,network=local:snippets/v1-network.yaml"

and as separate lines:


# SET cicustom settings on vm

qm set <vmid>  --cicustom "user=local:snippets/user-data.yaml"

qm set <vmid> --cicustom "network=local:snippets/v1-network.yaml"  # for some reason breaks above user-data settings...

If I include the network config, for some reason the user-data settings are not applied. So I have it working with only the user-data - and setting the basic IP with --ipconfig0 ip=192.168.0.43/24,gw=192.168.0.1 as a band-aid for now.

Ultimately, I would like the seed the network config from the cloud-init yaml file as this streamlines the automated provisioning process a bit.
Followed this as a guide: https://pve.proxmox.com/wiki/Cloud-Init_Support
"They can all be specified together or mixed and matched however needed."

Trying to figure out what I am doing wrong while applying both the user and network settings. Individually they work, but not both when applied to the same instance. Thought I would post here and see if anyone has had luck getting both user and network settings seeded to the VM instance. Probably just a syntax goof on my part.

Hope everyone is having a wonderful holiday season!

 

[dj423]

Ok as I suspected, im an idiot! Turned out I had a typo in my network config yaml and cloud-init choked on it. Once I used the correct interface names, it now works as advertised.
Success!

 

[fba]

# SET cicustom settings on vm

qm set <vmid>  --cicustom "user=local:snippets/user-data.yaml"

qm set <vmid> --cicustom "network=local:snippets/v1-network.yaml"  # for some reason breaks above user-data settings...

As far as I know, all parameters are set in a single command. If you leave some out, they get automatically created ones assigned. That's why the second command nullifies the first.

 

[dj423]

As far as I know, all parameters are set in a single command. If you leave some out, they get automatically created ones assigned. That's why the second command nullifies the first.

Been my experience so far as well. Experimenting with getting the default assigned "hostname" and "fqdn" values to pass through to the instance. Pretty sure, long as I leave those out of my custom user-data it should pass the vm name as hostname and cloud-init set that.

 

[fba]

Pretty sure, long as I leave those out of my custom user-data it should pass the vm name as hostname and cloud-init set that.

My understanding is that you have to leave out the entire user-data for the automatic generation of values, not just single values only. Have not test setup available atm, so it is just a guess from reading the documentation.

 

[dj423]

My understanding is that you have to leave out the entire user-data for the automatic generation of values, not just single values only. Have not test setup available atm, so it is just a guess from reading the documentation.

I think you are correct.

For example, when I look at the config on the VM with qm:

root@pve1:~/scripts# qm cloudinit dump 301 user
#cloud-config
hostname: alma-test-vm
manage_etc_hosts: true
fqdn: alma-test-vm
chpasswd:
  expire: False
users:
  - default
package_upgrade: true

But when I mount the cloud-init cd from within the instance, I do not see hostname: or the fqdn: values present in the user-data file. So they seem to get stripped out.

Is there a way to merge/combine the auto generated keys with my custom user-data.yaml file? Or a way to seed the hostname from pve to seed into the config file (user-data, meta-data, or even vendor-data) imported with the -cicustom function? Working on the automation flow of provisioning vm's and it would be handy to have them auto named, at least for initial spin up.

 

[fba]

When the new vm is created, you know about the mac address of the nic. Feed it to your DHCP server and let it provide the hostname together with all the other usual options.
Another possibility would be to write the vm id into the serial field in SMBIOS information. With dmidecode command (dmidecode --string system-serial-number) you can retrieve it in the vm and set hostname accordingly. It should be possible to use it runcmd option of cloud-init.

 

[dj423]

Ah ok, I think that might work.

Sort of like this example:

name=$(pvesh get /nodes/pve1/qemu/103/config | grep name | awk '{print $4}')

uuid=$(pvesh get /nodes/pve1/qemu/103/config | grep smbios1 | awk '{print $4}')

# set the uuid/serial values (base64 encode)

qm set 103 --smbios1 "$uuid,serial=$(echo $name | base64),base64=1"
update VM 103: -smbios1 uuid=b5168a36-fc1f-4172-88bd-65e6ca0aef67,serial=YWxtYS10ZXN0Mgo=,base64=1

Then seed the user-data with runcmd:
- hostnamectl set-hostname $(dmidecode --string system-serial-number)

[root@hostname ansible]# hostnamectl set-hostname $(dmidecode --string system-serial-number)
[root@hostname ansible]# hostname
alma-test2

Works in theory, will do some testing. Thanks for the info!

 

[dj423]

FWIW: Thought I would add for anyone else reading this, it appears some years back a patch here was put in to allow seeding a vendor-data file where if one were to so choose - can load all the custom / 100+ line jinja templated cloud-config files needed for cloud workloads. Been on the struggle bus for a hot minute on my pve test cluster, not being able to fully automate the provisioning plane elements like I do on other provider platforms.

I am testing this now in a lab pve to test migrating a boatload of instances that all rely on a cloud-init heavy multi-distro, multi provider uniform config (iterative locale, tz, NFS mounts, packages, bootstrap scripts, syslog settings, IPA configuration, etc) to spin up without a ton of headaches. Big thanks to the pve-dev team for that feature!! Was worried about how I was going to code all this hackery to make it work, but I think vendor-data will have to work - to not step on the 'integrated cloud-init' since for some reason it seeds the user-data file, and not the vendor-data file in the cloudinit iso. A few others have observed this as well, reference link: https://forum.proxmox.com/threads/combining-custom-cloud-init-with-auto-generated.59008/page-2

Hoping to eventually move this into a production phase, looks promising so far.

 

[dj423]

For anyone that serves up their configs from an http server - noting this here for reference - since I know I will forget all this..

How to seed http/s path to cloud-init configs with nocloud datasource (for centralized provisioning config management)
Ref: https://cloudinit.readthedocs.io/en...custom-webserver-kernel-commandline-or-smbios

We should be able to seed this in a smbios string as per the docs. Ref: https://pve.proxmox.com/wiki/Manual:_qm.conf

serial=<Base64 encoded string>
Set SMBIOS1 serial number.

ex:

# grab uuid

pvesh get /nodes/pve1/qemu/101/config | grep smbios1 | awk '{print $4}'

qm set 101 --smbios1 "uuid=5b0f7dcf-cfe3-4bf3-87a2-1cad29bd51f9,serial=$(printf '%s' 'ds=nocloud-net;s=http://10.10.0.1/path/to/configs/' | base64),base64=1"

Future: also test seeding via "seedFrom" in latest version of ci Ref: https://cloudinit.readthedocs.io/en...ources/nocloud.html#source-3-custom-webserver