Cloud-Init registering DNS with template name.

[snknick]

As per title.. have set up cloud-init for rapid VM creation, however the hostname change is occurring after the machine registers with the DNS server, resulting in the IP/MAC being associated with the template name instead of the VMs new name.

I realise this is the DNS being bad at updating, but is there a way to have the hostname updated prior to the network connection being established?

In the meantime I've "solved" it by adding the following to my template:

/etc/cloud/cloud.cfg.d/99_dnsfix.cfg

runcmd:
 - '/usr/sbin/dhclient'

But it's not exactly elegant. It works, so it's fine, but if I've missed a better solution I'd love to know.

 

[bbgeek17]

Make sure you are using "metadata" part of cloud-init, its normally set before everything else. Also take a look at "local" boot stage, it seems to specifically call out the problem you are trying to solve:

This stage must block network bring-up or any stale configuration that might have already been applied. Otherwise, that could have negative effects such as DHCP hooks or broadcast of an old hostname.

https://cloudinit.readthedocs.io/en/latest/topics/instancedata.html

I'd say keep tinkering with your cloud-config, it may depend on how you prepared the template and what OS you are using

---

Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[snknick]

Interesting as I've been using the Debian 10 and 11 official cloud images, I would have thought that hostname before network would be standard to avoid this exact issue.

Looks like I have a bit more learning to do as I'm not sure how exactly to change up the order of things, I've been using the default cloud-init files (minus the one I added as a workaround) and just editing the values proxmox supplies.

 

[bbgeek17]

The official cloud image is geared towards Openstack style metadata service. But it will fall back to local metadata file, which is what Proxmox provides. Are you using custom metadata file? You should be able to provide hostname there.
Keep in mind if you created a proxmox template from cloudimage and did not properly sanitize it, some of the cloud-init parts are only run "once" on first boot. So if you are not writing the originally provided image to disk on each VM create, but actually cloning from VM that was booted - not everything will be run.
Examine cloud-init log on the VM after boot, see if you can spot an issue.

---

Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[snknick]

Well for the official images I don't boot them at all (other than when I modified it to add my workaround). Pretty much the setup is:

wget https://cdimage.debian.org/cdimage/openstack/current-10/debian-10-openstack-amd64.qcow2
qm create 9000 --name debian10-cloud --memory 2048 --net0 virtio,bridge=vmbr0
qm importdisk 9000 debian-10-openstack-amd64.qcow2 TestZFS -format qcow2
qm set 9000 --scsihw virtio-scsi-pci --scsi0 TestZFS:vm-9000-disk-0
qm set 9000 --ide2 TestZFS:cloudinit --boot c --bootdisk scsi0 --serial0 socket --vga serial0

Then I convert to template and clone, changing the values in the cloud-init drive if needed. I also tried with the Debian "generic cloud" image just in case, but it seems most people use the OpenStack versions regardless.

Checking the logs of a fresh install of that exact process I do see something interesting (VM name is clouddnstest)

2022-03-21 01:25:41,754 - cc_set_hostname.py[DEBUG]: Setting the hostname to debian.example.com (clouddnstest)
2022-03-21 01:25:41,754 - util.py[DEBUG]: Reading from /etc/hostname (quiet=False)
2022-03-21 01:25:41,754 - util.py[DEBUG]: Read 19 bytes from /etc/hostname
2022-03-21 01:25:41,754 - util.py[DEBUG]: Writing to /etc/hostname - wb: [644] 13 bytes
2022-03-21 01:25:41,755 - __init__.py[DEBUG]: Non-persistently setting the system hostname to clouddnstest

That's line 425 of the logs, the network is being brought up and referenced significantly earlier.

Looking at the server console itself the very first reference to cloud-init is the following:

Starting Initial cloud-init job (pre-networking)...

You mentioned before to make sure I'm using the metadata part of cloud-init however I'm not entirely clear on how to do that. Based off what I'm reading that metadata should be provided by ProxMox and one of those values should be the VMName, which is used as the hostname. This happens but it's definitely not happening prior to the network making a connection.

 

[bbgeek17]

look into these two options:
--cicustom [meta=<volume>] [,network=<volume>] [,user=<volume>] [,vendor=<volume>]
cloud-init: Specify custom files to replace the automatically generated ones at start.

--citype <configdrive2 | nocloud | opennebula>
Specifies the cloud-init configuration format. The default depends on the configured operating system type (ostype. We use the nocloud format for Linux, and configdrive2 for windows.

Check the meta file generated now, either via mounting sr0 or "qm cloudinit dump". Is the format correct for your version of cloudinit? Things could differ and you may find building meta file manually an easier solution.

---

Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[snknick]

OK so.. I think I have it figured out. I have no idea why it works like this but I've tested it a dozen times now and I'm getting the same thing.

Under the Cloud-init options for ProxMox there is an IP Config setting. Naturally, I've been setting this to DHCP for my templates. When I do so however, the hostname is set after the network connection is made and causes the DNS issues

If I leave if on static (but with no values in the fields)? It works perfectly. Not only that, if I check /etc/networking/interfaces I can see that everything is set to DHCP.

I've tested this with my most recent template created with the citype of nocloud as per your suggestion as well as my old Debian 10 image which is the same downloaded image but with everything default. 100% of the time, changing the DHCP breaks things.

Testing with Debian 11 however it doesn't work either way but I made that image a few days ago and have only been testing with Debian 10 since then. Going to remake that now and see what happens.

I can't for the life of me think of a reason this would make the difference, but it does. At least for Debian 10... will report back with 11 shortly.

 

[snknick]

Not working for any of the Debian 11 images I've tried, however D11 does not have any OpenStack specific images which is what I've been using for Debian 10. I've tried generic and genericcloud for Debian 11 but neither work... leaving them on static just has them pick a randomly and already in use IP and DHCP has the same issue as before where the template name is registered with DNS not the new hostname.


I've checked the config dumps for every single template and VM, all of them have the correct values and I've tried the various cloud-init formats again with no luck.

Very confusing, I'd have thought deploying the premade debian cloud images through proxmox/cloud-init would be a very common task.. not sure what I'm missing.

 

[bbgeek17]

Have you checked your DNS? Its possible the initial DHCP request picks up the name from your DHCP/DNS server. The behavior could depend on the version of dhclient or cloud-init that comes with various levels of distros.

---

Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[snknick]

I have checked my DNS and I don't see anything obviously wrong, but I strongly suspect something is amiss (it's always DNS...).

I'll keep digging and update here in case I find something that might help someone else down the line.