Cloud init drives change to cdrom drive

[jimmycav]

I use terraform to provision my VMs, and each is created with a cloudinit drive (cloudinit.png). They are each replicated to other nodes for HA.

At some point during the life of the VM the cloudinit drives seem to change to cdrom drives in the UI (cloudinit-cdrom.png) and a new disk is created in zfs (cloudinit-zfs.png). While replication still works, when I attempt to migrate the VM to another node it fails because of the cdrom drive (cloudinit-error.png).

So I have to go to each VM, delete the cdrom drive and recreate the cloud init drive, which sometimes has to be deleted from zfs first.

It happens to all of my provisioned VMs

What is causing this, and how can i fix it?

 

[fba]

The screenshot of the error is of VM 102. The screenshots with the configuration are from VM 104 and VM 108. Would you like to share the configuration of VM 102, too?

 

[jimmycav]

Unfortunately I have since fixed each of the VMs (remove the 'cdrom' drive, shutdown, add cloudinit drive, start). That error was just one I had in the logs when a migration failed.

I'm trying to see how to reproduce the issue, but it simply happens randomly - most weeks I find one where the cloudinit drive has changed to a cdrom drive. Sometimes there is no cdrom or cloudinit drive in the UI, but a zfs cloudinit drive exists

 

[fba]

The config for a cloud-init drive is like
ide3: local-zfs:vm-108-cloudinit,media=cdrom,size=4M

The config in your examples is
ide3: local-zfs:vm-108-disk-3,media=cdrom,size=4M
So something created a new zvol on the storage and changed the vm config.

As Proxmox VE doesn't do any changes by itself, the cause is an external one. Are you sure about Terraform's actions?

 

[jimmycav]

Ok, I can reproduce it (kind of). So I've noticed that in my cluster of 5 nodes with zfs replication (each vm is only replicated up to 3 times), there are some VMs that have a cloudinit drive on multiple hosts. I assume is not normal, and only actual drives should exist, the cloudinit drives are recreated.

E.g.

For a 'good' VM, it has:

Node1 (where it is currently running):
local-zfs/vm-102-disk-cloudinit
local-zfs/vm-102-disk-0
local-zfs/vm-102-disk-1

Node2:
local-zfs/vm-102-disk-0
local-zfs/vm-102-disk-1

Note that there is no cloudinit drive on Node2. When I migrate that VM, everything is as expected.

But somehow, I have VMs that look like this:


Node1 (where it is currently running):
local-zfs/vm-107-disk-cloudinit
local-zfs/vm-107-disk-0
local-zfs/vm-107-disk-1

Node2:
local-zfs/vm-107-disk-cloudinit
local-zfs/vm-107-disk-0
local-zfs/vm-107-disk-1

Note there is a cloudinit on Node 2. When I migrate that VM from Node 1 to Node 2 it succeeds, but an extra zfs disk is created as a cdrom (like the above configuration).

Any ideas how these extra cloudinit drives could come about?

I suppose, it could somehow be terraform related?