PVE 5.4 Ceph Cloud Init Issue

[mira]

As this patch is rather heavy (on storages with lots of images), it's not a great solution. As a result it won't be released. There's a new patch available on the pve-devel list (https://pve.proxmox.com/pipermail/pve-devel/2019-April/036789.html) that uses a different solution.

 

[mira]

The patch (including follow-ups) is available in pvetest currently. You can manually download and install it (http://download.proxmox.com/debian/...ary-amd64/qemu-server-dbgsym_5.0-51_amd64.deb and http://download.proxmox.com/debian/...est/binary-amd64/qemu-server_5.0-51_amd64.deb). To install use 'apt install <path-to-deb>'.
Once it received enough testing it will be released in pve-no-subscription.

 

[amospalla]

We have a licensed cluster with qemu-server-5.0-51_amd64.deb installed on all hosts and bug still persists. Any ETA or recommendation so we can use Cloud-Init?

 

[mira]

Please be more specific. What's the exact error? In which scenario does it happen?
The original error reported in this thread should be solved by qemu-server 5.0-51.
Did you reboot after updating? Or at least restart pvedaemon and pveproxy?

 

[amospalla]

The error is:

task started by HA resource agent
qemu-img: Could not open '/dev/rbd/ssd/vm-345-cloudinit': Could not open '/dev/rbd/ssd/vm-345-cloudinit': No such file or directory
rbd: create error: (17) File exists2019-05-15 10:44:19.560397 7fe42796d100 -1 librbd: rbd image vm-345-cloudinit already exists
TASK ERROR: error with cfs lock 'storage-ceph-ssd': rbd create vm-345-cloudinit' error: rbd: create error: (17) File exists2019-05-15 10:44:19.560397 7fe42796d100 -1 librbd: rbd image vm-345-cloudinit already exists

Virtual Environment 5.4-5

We are using an external CEPH cluster (jewel) with krbd and the issue appeared for us on last thursday when we upgraded from 5.3 to 5.4.

I upgraded qemu-server 5.0-51 on all nodes and restarded pvedaemon (but not pveproxy). I also rebooted one of the nodes. It keeps giving us this error on any node including the rebooted one.

Maybe I should restart pveproxy on all nodes and try again?

 

[Daniel S.]

I confirm that the issue is still present even with the patch installed and daemons restarted - using the latest version from pve-no-subscription with the patch.

After patching when you first create a VM with cloudinit drive on ceph rbd storage it starts/stops without any problem.
If you regenerate the cloudinit image the error throws, VM cannot be migrated/started, etc.

The error is the same for the cloudinit drive: ....error: rbd: create error: (17) File exists

As a note, it will be nice if we could also assing the additional IP addresses for VMs using cloudinit.
And also, it would be nice if we can create a clone from a VM template with lets say disk size of 50GB a new VM with different/increased disk size, as cloudinit comes with the required modules to resize automatically.

 

[mira]

This seems to be a problem with 'krbd' and 'qemu-img info'. Please try without krbd for now.

 

[amospalla]

Yes, that's it, with librbd it works again.

Thank you.

 

[Daniel S.]

We expect it to work with krbd in the nearby future aswell, as some of us might see performance difference when using krbd.

 

[Donny Davis]

Ok maybe I am dull, but I don't see an option to use librbd for cloud-init images.
If I remove the cloud-init drive the issue disappears.

I have a hyperconverged cluster running ceph luminous

 

[Daniel S.]

I believe he was talking about the storage itself, not the drive. You can see and change that from Datacenter -> Storage.
The ceph storage, when editing, should have the option KRBD there.

Though some of us might notice performance difference, thats why I will wait till they will fix this issue.

 

[mira]

It should work with krbd in qemu-server 5.0-52.