Common practices for managing VM images?

[jamesharr]

Hello,

I'm working on some best practices for managing VM disk images for use as templates. I really like how we can do this for LXC, where there's a library to select from and build the container that's independent of any core count/memory sizing.

The specific kind of images I'm wanting to work on are the Cloud init images from Ubuntu, Alma, Rocky Linux (and others) distributed as qcow2.

I think it's worth mentioning that I have little interest in using linked clones, full copy/import of the disk image.

Right now I'm accomplishing this by using the PVE command line. I keep a folder with qcow2 images and import them when needed, but I'd like to graduate to using the API and also make it easier for some of our admins to do it via the UI when applicable. Bonus points if there's a way to upload a qcow2 via the UI. That'd be awesome.

Managing these as VM templates is also less interesting to me as well. VM templates come with a lot of hardware settings that I don't want to tie to the VM image. I'd rather have something similar to LXC image templates if possible.

How are people managing VM images today? Is this something I can submit a feature request for?

Thanks in advance

 

[t.lamprecht]

I think it's worth mentioning that I have little interest in using linked clones, full copy/import of the disk image.

Do you have a specific reason for that? As clone + some provisioning step (cloudinit, ansible, terraform, ...) is one of the simplest things to do. Also, depending on the underlying storage tech in use it'd be one of the most efficient/fastest method on COW based storages (ZFS, BTRFS) or those having reflinks enabled. With qcow2 you'd only get that using a base template and linked clones, if that's not fitting your usecase due to often changing the base images it may be not the best option for you though.

How are people managing VM images today?

Some use tools such as packer or ansible/terraform to setup image clones, or a simple clone + cloudinit config
E.g., in our CI we conditionally (determined if the VEID changed) clone from a base VM (which is not a template), set the respective config settings via the API, boot the VM and let buildbot start off, works pretty well and efficient while still being flexible enough to roll out config or base-image changes, especially as we're using ZFS for the storage which makes the cloning operation itself pretty fast/cheap.

Is this something I can submit a feature request for?

FWIW, there's some ongoing work to extend support for importing existing VMs (be it some qcow2 image or also from other hyper-visors) in the API/GUI. For now the qm importdisk is probably the most straight-forward option.

 

[jamesharr]

Do you have a specific reason for that? As clone + some provisioning step (cloudinit, ansible, terraform, ...) is one of the simplest things to do. Also, depending on the underlying storage tech in use it'd be one of the most efficient/fastest method on COW based storages (ZFS, BTRFS) or those having reflinks enabled. With qcow2 you'd only get that using a base template and linked clones, if that's not fitting your usecase due to often changing the base images it may be not the best option for you though.

The main reason I'm taking this approach is that I'm working on infrastructure bootstrapping scripts right now. So our goal is to make very few assumptions about what stuff is existing in proxmox and/or what other infrastructure exists. IE: don't depend on a CI/CD system.

The infrastructure we're setting up is going to be a smaller number of long-lived VMs, so the storage deduplication benefits of COW don't provide a lot of benefit at this moment. Storage is cheap enough, we don't envision seeing this problem.

Another driver right at the moment is that when each component is provisioned, it manages its own dependencies. IE: which OS image is specified, what packages are installed.

When we talk about hosting common apps, then we'll probably look at building a common template, though VM migration may be desirable in those instances.

Some use tools such as packer or ansible/terraform to setup image clones, or a simple clone + cloudinit config
E.g., in our CI we conditionally (determined if the VEID changed) clone from a base VM (which is not a template), set the respective config settings via the API, boot the VM and let buildbot start off, works pretty well and efficient while still being flexible enough to roll out config or base-image changes, especially as we're using ZFS for the storage which makes the cloning operation itself pretty fast/cheap.

If I wind up using a common set of tools, we'll probably build our own images for sure. Right now, we're working on some infrastructure bootstrap pieces. Thanks for the hints.

FWIW, there's some ongoing work to extend support for importing existing VMs (be it some qcow2 image or also from other hyper-visors) in the API/GUI. For now the qm importdisk is probably the most straight-forward option.

TY - This gives me the answer I need. For now, I'm going to stick to running PVE CLI tools through ansible for (it's working and pretty well), but I'll have a note to revisit when I see work on those API methods. Again, the context is in the realm of infrastructure bootstrapping. For other apps, we might look at using a VM template and/or linked clones.

As a side note -- the CLI tools have been really nice to work with. Consistent, reliable, pretty quick. Much better than a lot of other hypervisor vendors, which has been refreshing.

Thanks for the help and the software

 

[jamesharr]

FWIW, there's some ongoing work to extend support for importing existing VMs (be it some qcow2 image or also from other hyper-visors) in the API/GUI. For now the qm importdisk is probably the most straight-forward option.

@t.lamprecht - I see this is in the [7.2 roadmap](https://pve.proxmox.com/wiki/Roadmap#Proxmox_VE_7.2) now. I remember finding some sparse docs on how to use this via the API, but I for the life of me cannot find this anymore. Do you happen to have a link handy or have the right search string?

 

[jamesharr]

I think I may have found what I was looking for...

https://forum.proxmox.com/threads/new-vm-from-cloud-init-image-via-api.111091/#post-482420

Now if anyone is interested in seeing this in the terraform provider, comment on the github issue

https://github.com/Telmate/terraform-provider-proxmox/issues/605