[SOLVED] cicustom cloud-init stopped working

F

Finale3333

New Member
Oct 21, 2024
22
3
3
I've had a working proxmox box for a while now, I went to provision new VM's today and noticed the cicustom cloud init has stopped working.

I use IaC to provision the VM's and there has been no changes to the cloud-init scripts, the CICD pipelines, or provisioning scripts. Knowing this I dug a little deeper and started doing some manual commands with qm .

Here is the qm config:
Code: 
acpi: 1
agent: enabled=1,fstrim_cloned_disks=0,type=virtio
balloon: 0
bios: ovmf
boot: order=virtio0;net0
cicustom: user=local:snippets/ubuntu-noble-2404-init.yml
cores: 2
cpu: cputype=host
cpuunits: 1024
efidisk0: local-lvm:vm-205-disk-0,efitype=2m,pre-enrolled-keys=0,size=4M
ide2: local-lvm:vm-205-cloudinit,media=cdrom
ipconfig0: gw=192.168.10.1,ip=192.168.10.198/24
keyboard: en-us
machine: q35
memory: 4096
meta: creation-qemu=9.2.0,ctime=1743187441
name: iperf2
nameserver: 192.168.10.20 192.168.10.21
net0: virtio=BC:24:11:A8:80:B9,bridge=vmbr10g,firewall=0,mtu=1500,tag=1
numa: 0
onboot: 1
ostype: l26
protection: 0
scsihw: virtio-scsi-pci
serial0: socket
smbios1: uuid=7a0e4ac9-5fa6-4bad-aee6-ba9fa2fed6de
sockets: 1
tablet: 1
template: 0
virtio0: local-lvm:vm-205-disk-1,aio=io_uring,backup=1,cache=none,discard=on,iothread=1,replicate=1,size=12G
vmgenid: 5541d636-dc5a-4392-bbd9-cc666b2a161f

I hashed the snippet file, and it's the same one we've been using in IaC. As a sanity check I ran cloud-init schema schema --config-file /var/lib/vz/snippets/ubuntu-noble-2404-init.yml which returns a successful validation. I then ran qm cloudinit update 205 which says it regenerated the configuration, started the vm and terminaled into it to watch the kernel/systemd startup messages, and there is no recognition of the user data file.

I'm scratching my head on this one, I provisioned VMs yesterday without issue using the exact same base image and cloud init file.

Anyone have any other ideas or debugging ideas?

PVE
pve-manager/8.3.5/dac3aa88bac3f300 (running kernel: 6.8.12-9-pve)

Guest
Ubuntu 24.04.02
 
Last edited:
I do know about that but never ran into it myself, this is IaC 100% from end to end, and no changes to setup there, I have other VM's with ide cdrom (provisioned before today) that work fine.

Regardless, taking the suggestion, here's the new config:

Code: 
acpi: 1
agent: enabled=1,fstrim_cloned_disks=0,type=virtio
balloon: 0
bios: ovmf
boot: order=virtio0;net0
cicustom: user=local:snippets/ubuntu-noble-2404-init.yml
cores: 2
cpu: cputype=host
cpuunits: 1024
efidisk0: local-lvm:vm-205-disk-0,efitype=2m,pre-enrolled-keys=0,size=4M
ipconfig0: gw=192.168.10.1,ip=192.168.10.198/24
keyboard: en-us
machine: q35
memory: 4096
meta: creation-qemu=9.2.0,ctime=1743192878
name: iperf2
nameserver: 192.168.10.20 192.168.10.21
net0: virtio=BC:24:11:A8:72:74,bridge=vmbr10g,firewall=0,mtu=1500,tag=1
numa: 0
onboot: 1
ostype: l26
protection: 0
scsi0: local-lvm:vm-205-cloudinit,media=cdrom
scsihw: virtio-scsi-pci
serial0: socket
smbios1: uuid=02266509-b3ce-443f-921c-93465a75cb33
sockets: 1
tablet: 1
template: 0
virtio0: local-lvm:vm-205-disk-1,aio=io_uring,backup=1,cache=none,discard=on,iothread=1,replicate=1,size=12G
vmgenid: 747f2c9b-7484-43ea-a9ae-ba517940cb2c

I also reran set on cicustom & update just to be sure

Still a no go, full systemd output is attached
 

Attachments

Last edited:
I did the other try you mentioned, switching the bios to legacy. Regenerated the cloudinit, and still nothing.

Doing some more debugging I did the following:

Code: 
mkdir tmp && mount /dev/pve/vm-205-cloudinit tmp && cat tmp/user-data

Basically I mounted the cloud-init drive to pve's /root/tmp folder to see the user-data file and it is in the drive, so that for sure is working
 
Last edited:
I can't see what exactly you are trying, obviously. Have you fully power down'ed the VM between the tries? Are you using fresh image? Have you confirmed that the device is seen during boot process (i.e. qm start [vmid] && qm terminal [vmid] ).

It works fine here.
pveversion
pve-manager/8.3.5/dac3aa88bac3f300 (running kernel: 6.8.12-1-pve)

root@pve-1:~# ./vm_create.sh
variables used:
OSIMAGE == ubuntu-24.04-noble-server-cloudimg-amd64.img
STORAGE == bb-nvme
VMID == 9030
DEVICE == scsi0
NAME == vm9030
OSUSER == blockbridge
CONSOLE == vga
CLOUDINIT == local
BIOS == default
MEMORY == 4096
SOCKET == 1
CPU == host
RESIZE == yes
AGENT == yes
==============
qm create 9030 --memory 4096 --name vm9030 --socket 1 --onboot no --cpu cputype=host
qm importdisk 9030 /mnt/pve/bbnas/template/qcow/ubuntu-24.04-noble-server-cloudimg-amd64.img bb-nvme --format raw
qm set 9030 --scsihw virtio-scsi-single --scsi0 bb-nvme:vm-9030-disk-0,aio=native,iothread=1
update VM 9030: -scsi0 bb-nvme:vm-9030-disk-0,aio=native,iothread=1 -scsihw virtio-scsi-single
qm set 9030 -net0 virtio,bridge=vmbr0,firewall=1,mtu=1000
update VM 9030: -net0 virtio,bridge=vmbr0,firewall=1,mtu=1000
qm set 9030 --scsi1 bb-nvme:cloudinit
update VM 9030: -scsi1 bb-nvme:cloudinit
scsi1: successfully created disk 'bb-nvme:vm-9030-cloudinit,media=cdrom'
generating cloud-init ISO
qm disk resize 9030 scsi0 16G
qm set 9030 --boot c --bootdisk scsi0
update VM 9030: -boot c -bootdisk scsi0
qm set 9030 --serial0 socket --vga virtio
update VM 9030: -serial0 socket -vga virtio
qm set 9030 -ipconfig0 ip=dhcp
update VM 9030: -ipconfig0 ip=dhcp
qm set 9030 --cipassword blockbridge --ciuser blockbridge
update VM 9030: -cipassword <hidden> -ciuser blockbridge
qm set 9030 -sshkeys /root/.ssh/authorized_keys
qm set 9030 --agent 1
update VM 9030: -agent 1

Code: 
root@pve-1:~# qm start 9030 && qm terminal 9030
generating cloud-init ISO
starting serial terminal on interface serial0 (press Ctrl+O to exit)
[    0.000000] Linux version 6.8.0-31-generic (buildd@lcy02-amd64-080) (x86_64-linux-gnu-gcc-13 (Ubuntu 13.2.0-23ubuntu4) 13.2.0, GNU ld (GNU Binutils for Ubuntu) 2.42) #31-Ubuntu SMP PREEMPT_DYNAMIC Sat Apr 20 00:40:06 UTC 2024 (Ubuntu 6.8.0-31.31-generic 6.8.1)

...
[    0.867108] sr 1:0:0:1: [sr0] scsi3-mmc drive: 16x/50x cd/rw xa/form2 cdda tray
[    0.868322] cdrom: Uniform CD-ROM driver Revision: 3.20
[    0.873333] sr 1:0:0:1: Attached scsi generic sg1 type 5

...
Mar 28 21:15:12 ubuntu systemd[1]: Starting cloud-init.service - Initial cloud-init job (metadata service crawler)...
Mar 28 21:15:13 ubuntu cloud-init[695]: Cloud-init v. 24.1.3-0ubuntu3 running 'init' at Fri, 28 Mar 2025 21:15:13 +0000. Up 126.44 seconds.
Mar 28 21:15:13 ubuntu cloud-init[695]: ci-info: ++++++++++++++++++++++++++++++++Net device info++++++++++++++++++++++++++++++++
Mar 28 21:15:13 ubuntu cloud-init[695]: ci-info: +--------+------+----------------+---------------+--------+-------------------+
Mar 28 21:15:13 ubuntu cloud-init[695]: ci-info: | Device |  Up  |    Address     |      Mask     | Scope  |     Hw-Address    |
Mar 28 21:15:13 ubuntu cloud-init[695]: ci-info: +--------+------+----------------+---------------+--------+-------------------+
Mar 28 21:15:13 ubuntu cloud-init[695]: ci-info: |  eth0  | True | 172.16.202.125 | 255.255.255.0 | global | bc:24:14:78:a4:3d |
Mar 28 21:15:13 ubuntu cloud-init[695]: ci-info: |   lo   | True |   127.0.0.1    |   255.0.0.0   |  host  |         .         |
Mar 28 21:15:13 ubuntu cloud-init[695]: ci-info: |   lo   | True |    ::1/128     |       .       |  host  |         .         |
Mar 28 21:15:13 ubuntu cloud-init[695]: ci-info: +--------+------+----------------+---------------+--------+-------------------+
Mar 28 21:15:13 ubuntu cloud-init[695]: ci-info: +++++++++++++++++++++++++++++++Route IPv4 info+++++++++++++++++++++++++++++++
Mar 28 21:15:13 ubuntu cloud-init[695]: ci-info: +-------+--------------+--------------+-----------------+-----------+-------+
Mar 28 21:15:13 ubuntu cloud-init[695]: ci-info: | Route | Destination  |   Gateway    |     Genmask     | Interface | Flags |
Mar 28 21:15:13 ubuntu cloud-init[695]: ci-info: +-------+--------------+--------------+-----------------+-----------+-------+
Mar 28 21:15:13 ubuntu cloud-init[695]: ci-info: |   0   |   0.0.0.0    | 172.16.202.1 |     0.0.0.0     |    eth0   |   UG  |
Mar 28 21:15:13 ubuntu cloud-init[695]: ci-info: |   1   |  172.16.5.1  | 172.16.202.1 | 255.255.255.255 |    eth0   |  UGH  |
Mar 28 21:15:13 ubuntu cloud-init[695]: ci-info: |   2   | 172.16.202.0 |   0.0.0.0    |  255.255.255.0  |    eth0   |   U   |
Mar 28 21:15:13 ubuntu cloud-init[695]: ci-info: |   3   | 172.16.202.1 |   0.0.0.0    | 255.255.255.255 |    eth0   |   UH  |
Mar 28 21:15:13 ubuntu cloud-init[695]: ci-info: +-------+--------------+--------------+-----------------+-----------+-------+
Mar 28 21:15:13 ubuntu cloud-init[695]: ci-info: +++++++++++++++++++Route IPv6 info+++++++++++++++++++
Mar 28 21:15:13 ubuntu cloud-init[695]: ci-info: +-------+-------------+---------+-----------+-------+
Mar 28 21:15:13 ubuntu cloud-init[695]: ci-info: | Route | Destination | Gateway | Interface | Flags |
Mar 28 21:15:13 ubuntu cloud-init[695]: ci-info: +-------+-------------+---------+-----------+-------+
Mar 28 21:15:13 ubuntu cloud-init[695]: ci-info: +-------+-------------+---------+-----------+-------+
Mar 28 21:15:13 ubuntu kernel: EXT4-fs (sda1): resizing filesystem from 655099 to 3931899 blocks
Mar 28 21:15:13 ubuntu kernel: EXT4-fs (sda1): resized filesystem to 3931899
Mar 28 21:15:13 vm9030 systemd-resolved[517]: System hostname changed to 'vm9030'.


Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox
 
I am stopping the vm inbetween (qm stop [vmid]) and using a fresh image from ubuntu cloud image site, just did a hash check on it.

It looks like it's recognizing the CDDrive during kernel stage

Code: 
[    2.111945] ata3: SATA link down (SStatus 0 SControl 300)
[    2.112885] ata2.00: ATAPI: QEMU DVD-ROM, 2.5+, max UDMA/100
[    2.114337] ata2.00: applying bridge limits
[    2.115716] ata2.00: configured for UDMA/100
[    2.117302] scsi 1:0:0:0: CD-ROM            QEMU     QEMU DVD-ROM     2.5+ PQ: 0 ANSI: 5
[    2.132595] sr 1:0:0:0: [sr0] scsi3-mmc drive: 4x/4x cd/rw xa/form2 tray
[    2.134147] cdrom: Uniform CD-ROM driver Revision: 3.20
[    2.152821] sr 1:0:0:0: Attached scsi generic sg0 type 5
[    2.158253] input: QEMU QEMU USB Tablet as /devices/pci0000:00/0000:00:1d.7/usb3/3-1/3-1:1.0

My systemd output is like yours, it is configuring the network from the ipconfig0 part of the config, it's just not applying the user-data file. So it is doing cloud config, but not grabbing the user-data from the cloud config; and I did confirm the user-data is in the cloudinit drive (via the above post)

Code: 
[  OK  ] Reached target network.target - Network.
         Starting systemd-networkd-wait-onl…ait for Network to be Configured...
[  OK  ] Finished systemd-networkd-wait-onl… Wait for Network to be Configured.
         Starting cloud-init.service - Cloud-init: Network Stage...
[    6.468125] cloud-init[657]: Cloud-init v. 24.4-0ubuntu1~24.04.2 running 'init' at Fri, 28 Mar 2025 20:49:26 +0000. Up 6.44 seconds.
[    6.489744] cloud-init[657]: ci-info: +++++++++++++++++++++++++++++++++++++++Net device info+++++++++++++++++++++++++++++++++++++++
[    6.492399] cloud-init[657]: ci-info: +--------+------+------------------------------+---------------+--------+-------------------+
[    6.495023] cloud-init[657]: ci-info: | Device |  Up  |           Address            |      Mask     | Scope  |     Hw-Address    |
[    6.497693] cloud-init[657]: ci-info: +--------+------+------------------------------+---------------+--------+-------------------+
[    6.500258] cloud-init[657]: ci-info: |  eth0  | True |        192.168.10.198        | 255.255.255.0 | global | bc:24:11:a8:72:74 |
[    6.502778] cloud-init[657]: ci-info: |  eth0  | True | fe80::be24:11ff:fea8:7274/64 |       .       |  link  | bc:24:11:a8:72:74 |
[    6.505331] cloud-init[657]: ci-info: |   lo   | True |          127.0.0.1           |   255.0.0.0   |  host  |         .         |
[    6.507869] cloud-init[657]: ci-info: |   lo   | True |           ::1/128            |       .       |  host  |         .         |
[    6.510352] cloud-init[657]: ci-info: +--------+------+------------------------------+---------------+--------+-------------------+
[    6.512671] cloud-init[657]: ci-info: +++++++++++++++++++++++++++++Route IPv4 info+++++++++++++++++++++++++++++
[    6.514928] cloud-init[657]: ci-info: +-------+-------------+-------------+---------------+-----------+-------+
[    6.517072] cloud-init[657]: ci-info: | Route | Destination |   Gateway   |    Genmask    | Interface | Flags |
[    6.519114] cloud-init[657]: ci-info: +-------+-------------+-------------+---------------+-----------+-------+
[    6.521195] cloud-init[657]: ci-info: |   0   |   0.0.0.0   | 192.168.10.1|    0.0.0.0    |    eth0   |   UG  |
[    6.523247] cloud-init[657]: ci-info: |   1   | 192.168.0.0 |   0.0.0.0   | 255.255.252.0 |    eth0   |   U   |
[    6.525361] cloud-init[657]: ci-info: +-------+-------------+-------------+---------------+-----------+-------+
[    6.527355] cloud-init[657]: ci-info: +++++++++++++++++++Route IPv6 info+++++++++++++++++++
[    6.528992] cloud-init[657]: ci-info: +-------+-------------+---------+-----------+-------+
[    6.530678] cloud-init[657]: ci-info: | Route | Destination | Gateway | Interface | Flags |
[    6.532814] cloud-init[657]: ci-info: +-------+-------------+---------+-----------+-------+
[    6.534977] cloud-init[657]: ci-info: |   0   |  fe80::/64  |    ::   |    eth0   |   U   |
[    6.536984] cloud-init[657]: ci-info: |   2   |    local    |    ::   |    eth0   |   U   |
[    6.538921] cloud-init[657]: ci-info: |   3   |  multicast  |    ::   |    eth0   |   U   |
[    6.540863] cloud-init[657]: ci-info: +-------+-------------+---------+-----------+-------+
[  OK  ] Finished cloud-init.service - Cloud-init: Network Stage.
[  OK  ] Reached target cloud-config.target - Cloud-config availability.
[  OK  ] Reached target network-online.target - Network is Online.
[  OK  ] Reached target sysinit.target - System Initialization.
[  OK  ] Started apt-daily.timer - Daily apt download activities.
 
If it was my system, I would do the following steps:
a) Use built-in cloud-init to confirm functionality
b) Boot your system as you do now, if you are able to login - examine the cloudinit log
c) If you cant login, shutdown the VM after boot, mount the disk manually to hypervisor and examine the cloudinit log, as well as mount the generated ISO (lvm slice) and examine its context

Cheers

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox
 
Good ideas, I was about to do c) because the whole cloudinit log doesn't get output during bootup logs.

Thanks for your help! I didn't catch your bash script before, if the logs doesn't turn up anything obvious I'll use that to do a)
 
@bbgeek17

Your help on this was much appreciated, I was in the midst of trying to diagnose other bugs when this was brought up, hectic day.

After taking a look at the logs the user-data cloud init was getting initialized but was dying out right after ntp setup. After poking around the ntp proxy server setup, I discovered one of the onsite servers, was misbehaving. It was causing all sorts of errors in the cloud init, pure wackiness in the logs, timestamps all over the place, the server certificates were failing to generate.

I've added a note to log scrape from primary vm disk during first run in our provisioning automation so we can actively grab these logs. Never ran into a problem this contrived before during cloudimg installation so never really bothered.

....I've officially got one for the books
 
Last edited:
  • Like
Reactions: bbgeek17
You must log in or register to reply here.
Top Bottom