Proxmox ignores custom cloud-init from --cicustom paramter (tested with Ubuntu/CentOS)

tina · Oct 26, 2020

Hi,

I'm getting crazy to create a vm provisioned with cloud-init. Here is what i do:

Bash:

# create template
wget https://cloud.centos.org/centos/8/x86_64/images/CentOS-8-GenericCloud-8.2.2004-20200611.2.x86_64.qcow2
qm create 9000 --name centos-ci  --memory 2048 --net0 virtio,bridge=vmbr0
qm importdisk 9000 CentOS-8-GenericCloud-8.2.2004-20200611.2.x86_64.qcow2 local-lvm
qm set 9000 --scsihw virtio-scsi-pci --scsi0 local-lvm:vm-9000-disk-0
qm set 9000 --ide2 local-lvm:cloudinit
qm set 9000 --boot c --bootdisk scsi0
qm set 9000 --serial0 socket --vga serial0
qm template 9000

#create vm
qm clone 9000 300 --full true --name cli-test-cos --storage local-lvm
qm set 300 --cicustom user=local:snippets/user.yaml
qm start 300

The content of the /var/lib/zv/snippets/user.yaml
is

YAML:

chpasswd:
  expire: False
users:
  - k3s
    passwd: "$5$=4096$utiyPEPcWUUEiS7O33k2tmFP55zf2rK7KEYGuZn3zf6"
    sudo: "ALL=(ALL) NOPASSWD:ALL"
    ssh_authorized_keys:
      - "ssh-rsa AAAAB3NzaC1yc-....."

What i now expect is that the vm loads the user.yaml and setup the user/password. But nothing of this happens. I've also tried the same with ubuntu with this image.
In the "Cloud-Init" tab of the UI tells me, that no specific config is set:

while `qm config 300` looks like this:

Code:

boot: c
bootdisk: scsi0
cicustom: user=local:snippets/user.yaml
ide2: local-lvm:vm-300-cloudinit,media=cdrom,size=4M
memory: 2048
name: cli-test-cos
net0: virtio=F2:6D:2D:B3:87:C5,bridge=vmbr0
scsi0: local-lvm:vm-300-disk-0,size=10G
scsihw: virtio-scsi-pci
serial0: socket
smbios1: uuid=f7ab1f62-718c-4d70-9603-3c10858b7b6e
vga: serial0
vmgenid: a6520f50-d8ef-4ac4-8e68-ae80d48a0d6f

What addition steps i have to to to get the vm provisioned as expected?

Best regards
Tina

mira · Oct 27, 2020

Try running cloud-init clean and then reboot.

tina · Oct 27, 2020

mira said:
Try running cloud-init clean and then reboot.

Since I'm not able to login to the vm I'm not sure how to do so. Of couse i could additionally set the -ciuser and -sshkeys opion but i thought the cicustom with the user.yaml should do the same...

mira · Oct 27, 2020

Using a custom user snippet overrides the complete user config set in the GUI or config. This means you can't combine those 2 together. If you set ciuser, cipassword or sshkeys in the GUI and add a custom user snippet, only the options set in the snippet will be used.

tina · Oct 27, 2020

mira said:
Using a custom user snippet overrides the complete user config set in the GUI or config. This means you can't combine those 2 together. If you set ciuser, cipassword or sshkeys in the GUI and add a custom user snippet, only the options set in the snippet will be used.

Of course i know that I can't use both options at the same time...
But when i take a look into the cloud-init image i think i've seen my problem:

while the autogenerated config of proxmox by setting ciuser / cipassword looks like this:

YAML:

#cloud-config
hostname: cli-test-cos1
manage_etc_hosts: true
user: k3s
password: $5$i/sHvvmO$fsvuXzdWohMkFX0HZ13SO0/OwcRQhhpT0NhcobjAA74
chpasswd:
  expire: False
users:
  - default
package_upgrade: true

mine looks like this ( i followed the docs here)

YAML:

chpasswd:
  expire: False
users:
  - k3s
    passwd: "$5$=4096$utiyPEPcWUUEiS7O33k2tmFP55zf2rK7KEYGuZn3zf6"
    sudo: "ALL=(ALL) NOPASSWD:ALL"
    ssh_authorized_keys:
      - "ssh-rsa AAAAB3NzaC1yc-....."

tina · Oct 27, 2020

@mira I've played around a little bit the the options:

I've set ciuser/cipasswd/sshkeys/nameserver/ipconfig0 options and all is working as expected. So i've take a look into the cloud-init iso and the configuration files:

meta-data

YAML:

instance-id: d57a4b93a50e48122186ddbe185efe1935a47741

network-config

YAML:

version: 1
config:
    - type: physical
      name: eth0
      mac_address: 'de:e8:f6:ef:13:fc'
      subnets:
      - type: static
        address: '192.168.20.150'
        netmask: '255.255.255.0'
        gateway: '192.168.20.1'
    - type: nameserver
      address:
      - '192.168.20.200'
      - '192.168.20.1'
      search:
      - 'domain.tld'

user-data

YAML:

hostname: cli-test-cos
manage_etc_hosts: true
fqdn: cli-test-cos.domain.tld
user: k3s
password: $5$98FxBt7m$h62.GEDIc4m1rcWt1IDjFsvVfh8g5WanTkGTdk1H0M3
ssh_authorized_keys:
  - ssh-rsa AAAA.....
chpasswd:
  expire: False
users:
  - default
package_upgrade: true

I've copied the code of the network-config and the user-data and put them under /var/lib/vz/snippets/(user|network).yaml
Then i've deleted the container and created it again:

Bash:

 qm clone 9000 300 --full true --name cli-test-cos --storage local-lvm
 qm set 300 -cicustom "user=local:snippets/user.yaml,network=local:snippets/network.yaml"
 qm start 300 # told me that the cloud-init iso is generated...

So the config matches exactly the one of the manual settings, but the config doesn't seems to be respected.
I've mounted the cloud-init iso on the pve host to check the content, but all configs are there and correct.
So it seems that the -set options do anything different to the vm OR the -cicustom option is broken.

Dacesilian · Jan 8, 2021

I have the same problem - providing config files with

Code:

qm set 138 --cicustom "network=local:snippets/138-network.yml,user=local:snippets/138-user.yml"

and Debian gets stucked on:

Code:

[    4.122675] cloud-init[209]: SHA256:vnpWXRTWMVUM59cjPChKnfxp0g3YVj+pcrez0Fik7tk root@debian
[    4.124201] cloud-init[209]: The key's randomart image is:
[    4.125326] cloud-init[209]: +---[ECDSA 256]---+
[    4.126355] cloud-init[209]: |             .+BB|
[    4.127431] cloud-init[209]: |        o + +.oo*|[  OK  ] Started Initial cloud-init job (pre-networking).
[  OK  ] Reached target Network (Pre).


[  OK  ] Started ifup for eth0.
[    4.130813] cloud-init[209]: |       . * = = *+|
         Starting Raise network interfaces...


[   ***] A start job is running for Raise ne…rk interfaces (1min 33s / 5min 1s)

Why network settings is not set? I don't have DHCP so booting is stucked. Thanks.

tina · Jan 14, 2021

Any idea @mira ?

mira · Jan 14, 2021

Did you run cloud-init clean and then reboot?

mira · Jan 14, 2021

Dacesilian said:

I have the same problem - providing config files with

Code:

qm set 138 --cicustom "network=local:snippets/138-network.yml,user=local:snippets/138-user.yml"

and Debian gets stucked on:

Code:

[    4.122675] cloud-init[209]: SHA256:vnpWXRTWMVUM59cjPChKnfxp0g3YVj+pcrez0Fik7tk root@debian
[    4.124201] cloud-init[209]: The key's randomart image is:
[    4.125326] cloud-init[209]: +---[ECDSA 256]---+
[    4.126355] cloud-init[209]: |             .+BB|
[    4.127431] cloud-init[209]: |        o + +.oo*|[  OK  ] Started Initial cloud-init job (pre-networking).
[  OK  ] Reached target Network (Pre).


[  OK  ] Started ifup for eth0.
[    4.130813] cloud-init[209]: |       . * = = *+|
         Starting Raise network interfaces...


[   ***] A start job is running for Raise ne…rk interfaces (1min 33s / 5min 1s)

Why network settings is not set? I don't have DHCP so booting is stucked. Thanks.

This is a common problem if you set IPv6 to DHCP without having a DHCP server.

pawanshrestha · Apr 10, 2021

@tina

I know it's little old thread but I was having similar problem. It would not pick up my custom snippet. What did the trick in my case is the addition of "#cloud-config" as the first line of the yml file:

A sample that is working on my end is as following:

YAML:

#cloud-config
hostname: pkshr
manage_etc_hosts: true
user: ubuntu
chpasswd:
  expire: False
users:
  - default
ssh_authorized_keys:
  - ssh-rsa XXXXXXXXXXXXXXXXXXXXXx
package_upgrade: true

eset · Jan 5, 2023

pawanshrestha said:
@tina

I know it's little old thread but I was having similar problem. It would not pick up my custom snippet. What did the trick in my case is the addition of "#cloud-config" as the first line of the yml file:

A sample that is working on my end is as following:

YAML:

#cloud-config hostname: pkshr manage_etc_hosts: true user: ubuntu chpasswd: expire: False users: - default ssh_authorized_keys: - ssh-rsa XXXXXXXXXXXXXXXXXXXXXx package_upgrade: true

This worked ?

eset · Jan 5, 2023

Ok it worked for me also but dump still shows stupid things that aren't related with configs.

Code:

root@pve:/var/lib/vz/snippets# qm cloudinit dump 100 user
#cloud-config
hostname: srv-app-1
manage_etc_hosts: true
chpasswd:
  expire: False
users:
  - default
package_upgrade: true

root@pve:/var/lib/vz/snippets# qm cloudinit dump 100 network
version: 1
config:
    - type: nameserver
      address:
      - '10.0.1.1'
      - '1.1.1.1'
      search:
      - 'hw.sonic'

For example my network-config looks like this

Code:

#cloud-config
version: 1
config:
    - type: physical
      name: ens18
      subnets:
        - type: static
          address: 10.0.101.1/24
          gateway: 10.0.101.254
    - type: nameserver
      address: ["10.0.1.30","1.1.1.1"]
      search: ["pve.sonic","hw.sonic","sonic"]

bbgeek17 · Jan 5, 2023

eset said:
Ok it worked for me also but dump still shows stupid things that aren't related with configs.

the "qm cloudinit dump" only shows the PVE native cloudinit config, it does not look at or show custom files. If you are using custom cloudinit, the output of dump is completely irrelevant.

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

bbgeek17 · Jan 5, 2023

eset said:
This worked ?

https://cloudinit.readthedocs.io/en/latest/topics/format.html

Code:

See the Cloud config examples section for a commented set of examples of supported cloud config formats.

Begins with: #cloud-config or Content-Type: text/cloud-config when using a MIME archive.

Note

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

eset · Jan 5, 2023

bbgeek17 said:
the "qm cloudinit dump" only shows the PVE native cloudinit config, it does not look at or show custom files. If you are using custom cloudinit, the output of dump is completely irrelevant.

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

and how the native one is set? Where exactly or how?

bbgeek17 · Jan 5, 2023

eset said:
and how the native one is set?

via PVE CLI, GUI or API

eset said:
Where exactly or how?

Utilizing the Cloudinit section of VM properties, using "qm set" options (see "man qm" , section "Deploying Cloud-Init Templates" and " Cloud-Init specific Options"), or corresponding API.

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

lewinernst · Jul 11, 2024

mira said:
Using a custom user snippet overrides the complete user config set in the GUI or config. This means you can't combine those 2 together. If you set ciuser, cipassword or sshkeys in the GUI and add a custom user snippet, only the options set in the snippet will be used.

Hi - does this only apply when using a custom userconfig or also when using a custom vendor config?

mira · Jul 11, 2024

We don't generate a vendor config at the moment. So adding a custom vendor config won't override any other config.
Any custom config will only override just this specific generated one.
Custom User Config -> Generated User Config
Custom Network Config -> Generated Network Config
Custom Metadata Config -> Generated Metadata Config

lewinernst · Jul 11, 2024

Hi - thanks for the quick Feedback! Since there are 4 configs i might as well ask: which ones are "free" and which ones are used by pve?

Proxmox ignores custom cloud-init from --cicustom paramter (tested with Ubuntu/CentOS)

Member

Proxmox Staff Member

Member

Proxmox Staff Member

Member

Member

Member

Member

Proxmox Staff Member

Proxmox Staff Member

New Member

Well-Known Member

Well-Known Member

Distinguished Member

Distinguished Member

Well-Known Member

Distinguished Member

Member

Proxmox Staff Member

Member