KVM templates

[msg7086]

Unfortunately I've worked with SolusVM and I'm very used to their VM deployment process. You create a few KVM templates beforehand, place them in a folder just like ISO images (but qcow2 images instead), and deploy them into varies size of VM in one shot.

PVE seems to lack this feature on KVM.

Yes, I know I can use LXC with templates, but I'd like to use KVM.

Yes, I know I can "convert" a VM into templates but I still have to manually resize partitions later.

I guess I can maybe run qemu-img command and clone image to VM disks from SSH.

What do you think? Is this a feature you'd like to have? I'm hesitating to submit this to the bug tracker, so decide to collect some ideas first.

 

[t.lamprecht]

So, AFAICT, you just lack the "resize partitions for update disk sizes" for a clone?

Did SolusVM resized those automatically for you? with libguestfs or the like? (do not recall using SolusVM, sorry)

 

[msg7086]

So, AFAICT, you just lack the "resize partitions for update disk sizes" for a clone?

Did SolusVM resized those automatically for you? with libguestfs or the like? (do not recall using SolusVM, sorry)

Well, "technically" that's the only lacking function that I wanted. But to be honest, having to have a "VM-like template" sitting in the VM list doesn't make me as comfortable.

Comparing to how LXC template works:

You have a few prepared templates, let's say there's 10 Linux flavors. You either download them from official or make your own template.
On the second step of creating CT, you pick a template, and when it's done, you have a working container.

Ideally I'm looking for a similar function, as:

You have a few prepared KVM templates, either by downloading or by making your own.
On the second step, instead of filling OS image, provide a template instead.
When the wizard is done, virt-resize clone the template to the target disk image.
(Optionally, like SolusVM, initialize hostname, passwd, networking, etc., but may not be as important.)

Did SolusVM resized those automatically for you? with libguestfs or the like? (do not recall using SolusVM, sorry)

Yes, so it's essentially

virt-resize --expends /dev/sda1 template.qcow2 /dev/mapper/vg0-vm100-disk0

where sda"1" is specified in template settings (which partition to expand).

 

[oguz]

you might be interested in cloudinit

 

[msg7086]

you might be interested in cloudinit

I know cloud-init, but could you please shed some light on how cloud-init could help to deploy KVM templates just like LXC in one click?

Thanks.

 

[msg7086]

I just had a very quick POC.

# lvchange -ay vg0/vm-101-disk-0
# virt-resize --expand /dev/sda1 my-debian-10-x86_64-2019-12.gz /dev/mapper/vg0-vm--101--disk--0
[   0.0] Examining my-debian-10-x86_64-2019-12.gz
**********

Summary of changes:

/dev/sda1: This partition will be resized from 767.0M to 4.0G.  The
filesystem xfs on /dev/sda1 will be expanded using the ‘xfs_growfs’
method.

**********
[   4.4] Setting up initial partition table on /dev/mapper/vg0-vm--101--disk--0
[   5.2] Copying /dev/sda1
 100% ⟦▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒⟧ --:--
[  11.5] Expanding /dev/sda1 using the ‘xfs_growfs’ method

Resize operation completed with no errors.  Before deleting the old disk,
carefully check that the resized disk boots and works correctly.

Seems to work for me.

 

[t.lamprecht]

But to be honest, having to have a "VM-like template" sitting in the VM list doesn't make me as comfortable.

Why is that uncomfortable? They're clearly marked as template, so not like you could confuse them for a real "VM" or the like..
Proxmox VE guest templates (both, VM and CT ones) and CT installation appliance-templates are two completely different things.
You cannot really make VM templates in a general way, as they're much more flexible and general than a simple Linux Container, where it's already a lot of work to support all those distros in their different version like we do..

For VMs there are "Fully Automated Installation" and configuration management solutions, those are somewhat orthogonal to Proxmox VE.

What I thought about since quite a bit ago is a "Create from (PVE) Template" which is more integrated in the create wizard and allows to overwrite some basic settings from the beginning (memory, CPU, target storage and such), for both VMs and CTs.
I mean one could also integrate this in the clone dialog, but that's then implementation details.

Seems to work for me.

Cool, IMO this could be solved individually with hook scripts. But it seems that for (pre/post) clone or create there are no such scripts executed, but that would be practically IMO. As then you could add a script which triggers such a resize post-clone for VMs, which would be a nice and good work-around for this, IMO.

 

[msg7086]

You see, I said in the first paragraph that SolusVM has been doing this for a long time. It was used in production, for commercial purpose, and it went very well. Having a collection of templates provides a good way to manage all things, one of it being a central template storage for multiple host instances for a VPS provider or a private data center. And as you said there are varies of distributions, and thus will have many potential templates being made. Having them all listed in the tree "pollutes" the resource tree a lot. And it's not even close to easy maintenance. Like, how can I refresh the templates every few months? Do I have to go through all the steps of clone - boot - update - convert? I don't know but I feel like it's cumbersome. With SolusVM we have fully automated scripts to create all templates in one click, and rsync to the storage and call it a day.

Now with CT, everyone can download and deploy templates in one click. Do I have to first create a VM/CT to convert? No. Do I have to upload an ISO and install the OS myself? No. Do I have to play around cloud-init to pre-configure the server? No. It's efficient and convenient, it's a great job done from you guys and I appreciate it.

And I'm seeking the same efficiency and convenience on the KVM side. I'm trying to derive a function from an existing function. "Convert to template" has its use case but I don't think it fits here best.

You cannot really make VM templates in a general way

You can. VMs are not difficult to understand. Note here I'm not saying supporting VM templates in a general way is easy. I'm saying that making a VM template (that follows certain rules) is easy. What are certain rules? For example, you can assume that first Linux data partition is the Linux root partition. You can assume that you always get a MBR partition table. For Windows, you can assume the largest NTFS partition is the system partition. You make templates explicitly according to the rules, and it will work. I don't think it's hard to do because my experience with SVM tells me it can be done and it has been done. I spend my spare time making SVM templates as a part time job, both Windows and Linux (even Arch Linux), and they never disappointed me.

Now, I want to be clear that I'm not demanding you to do things now or any of that. I intended to start a discussion and I'm trying to bring up use cases, pros, cons, do some brainstorming and see if it's worth the effort to implement it. It may sound rude from me when I keep bringing up SVM. I use both products and they have their own pros and cons. I simply think this is a nice feature to have, and I don't think it's very hard to implement. Like I said, having the same UI logic as CT, run that virt-resize command for a user chosen template file is all what's needed.

I don't think this feature request is justified as a high priority, but I also don't think it is not justified at all.

 

[t.lamprecht]

Having them all listed in the tree "pollutes" the resource tree a lot.

So it rather sounds like you want an option to hide those. Or use the global search field (CTRL+SHIFT+F) to get to your desired VM quicker.

Like, how can I refresh the templates every few months? Do I have to go through all the steps of clone - boot - update - convert? I don't know but I feel like it's cumbersome. With SolusVM we have fully automated scripts to create all templates in one click, and rsync to the storage and call it a day.

I mean, we have an API. One can relatively easily use it to clone and start a VM, then updating (either by credentials setup in the VM or using the guest agent, which is included in our API), shutdown and create new template, drop old one. Great thing, building blocks are here and one can put them together to really fit their specific needs, already.

You can

Maybe can was the wrong word, naturally on can, one can do almost everything with enough effort and resources. This can be also done with lots of work and headache to get some specific things to work, were then those things that do not work are the first things user request As there are already solutions for (fully) automated installations and deployments I do not think it's worth to take up on those headaches.

For Windows

That against Microsofts TOS, we could never provide Windows templates just like that (and also do not want to ever do so).

I don't think it's hard to do because my experience with SVM tells me it can be done and it has been done.

Great, we're open-source after all, see https://pve.proxmox.com/wiki/Developer_Documentation for how to start contributing

I don't think this feature request is justified as a high priority, but I also don't think it is not justified at all.

I also don't think so. The "expand guest with libguestfs" part would be a feature enhancement request I could imagine to have more widely use, if you want to make such request at https://git.proxmox.com/
I made already one to add hooks for the "create", "clone", "remove" events ( https://bugzilla.proxmox.com/show_bug.cgi?id=2530 ) which would make automating such, and a lot of other things, also more easily.
As said, and don't take me wrong I appreciate that your sharing your ideas and argue for them, pre-build templates for VMs is something we're not going to start work on anytime soon. I'd rather try to document how some of such scenarios can be done with existing tools.

 

[msg7086]

you want an option to hide those

Just want to be clear I'm not trying to accomplish the work using existing functions. Those sounds "workaround" to me, like, it can be done in a nice and clean way but somehow it can also be done using what we have, and you have been suggesting that. I understand that there are workarounds but my feature request is something "new". (Sorry English is not my native language so my wording may be incorrect please excuse my English.)

we could never provide Windows templates

Definitely not, however vendors can do that if they are allowed. (Hint: VPS providers may use it.)

for how to start contributing

Actually I checked the source code and contribution document before I made the post. If I have time (or being hired to contribute) I'd love to participate. Unfortunately I already have a few open source projects on my hand and I don't want to irresponsibly start a feature and abandon it later. But we'll see if I can get a chance.

It has been nice talking to you, Thomas. Have a merry Christmas.

 

[screenie]

Well, "technically" that's the only lacking function that I wanted. But to be honest, having to have a "VM-like template" sitting in the VM list doesn't make me as comfortable.

Comparing to how LXC template works:

You have a few prepared templates, let's say there's 10 Linux flavors. You either download them from official or make your own template.
On the second step of creating CT, you pick a template, and when it's done, you have a working container.

Ideally I'm looking for a similar function, as:

You have a few prepared KVM templates, either by downloading or by making your own.
On the second step, instead of filling OS image, provide a template instead.
When the wizard is done, virt-resize clone the template to the target disk image.
(Optionally, like SolusVM, initialize hostname, passwd, networking, etc., but may not be as important.)


Yes, so it's essentially

virt-resize --expends /dev/sda1 template.qcow2 /dev/mapper/vg0-vm100-disk0

where sda"1" is specified in template settings (which partition to expand).

A VM Template to download from someone?
I would never ever do that for production use - only someone without any security concern would do that.... how would you decide if you can trust the creator/uploader and which 'extra' tools are included?
You should think twice if you 'really' need that form of deployment.

I also do not want to have templates in my VM listing - so i created several template VM's including all 'basic' software/settings i need and then backup them do a dedicated NFS export 'VM-Templates' and delete them on the PVE host.
Everytime i need a new VM i do a restore and change necessary parameters, that's more or less the same procedure as deploying an containe. With the container you have to set them while you click through it, with the restore you set it afterwards.
If i need to update the template VM's i restore them - perform the updates/changes and create a new backup and remove the old one.

Even with multiple PVE clusters - i have the NFS share mounted and automatically have the same template VM's available on all of them.... can't be simpler than that.

 

[msg7086]

A VM Template to download from someone?
I would never ever do that for production use - only someone without any security concern would do that.... how would you decide if you can trust the creator/uploader and which 'extra' tools are included?
You should think twice if you 'really' need that form of deployment.

Currently LXC templates in PVE can be downloaded by just a few clicks. The functionality has been there for a long time.
For commercial and production use, sure you can choose to build your own templates, and that's what we did with our SolusVM installation.

But think of this. You installed PVE from Proxmox team, and you say you don't trust their templates? If you don't think their dev team is worth the trust, why would you even use their products in the first place?

Why would you download ANY software or ANY operating system from someone?

The same concept applies to us as well. The company I worked for sells VPS products. If a customer comes to us and say I don't trust your template, we would definitely question why you would buy the product in the first place. (Full disk encryption is a valid reason, otherwise it doesn't make any sense to us.)

 

[screenie]

You haven't previously specified which templates from whom you want to use.
If you chose to use a software from a vendor of course you have to trust them, but with PVE you are not limited to use only templates provided by Proxmox.
You can download and use templates from anywhere - that's what i wanted to highlight from a security perspective as you were not that specific where these templates came from.

 

[msg7086]

screenie: From the word You either download them from official I assumed the KVM templates shall come from official just like LXC temlpates. Sorry if that caused confusion.

 

[screenie]

screenie: From the word You either download them from official I assumed the KVM templates shall come from official just like LXC temlpates. Sorry if that caused confusion.

Honestly, i do not know if the various LXC containers available to download are all created by the Proxmox guys - but the turnkey containers which are offered to download are for sure created by someone other...

 

[creslin]

Hi - Nutanix have this also, their image server.

When creating a VM a hard disk can be new, or images of pre-installed VMs.
These cloud images are provided by Ubuntu, Redhat, Centos etc and ship with SHA fingerprints to ensure they are wholesome.
as example: https://cloud-images.ubuntu.com/
Its the preffered way to deploy a cloud vm, as no install is required. img + cloud-init, == new vm in under 1 second.

Our nutanix will check for new checksums each few hours and update the images available if needed.

A user then creating a VM picks their flavour of OS, the image, adds username/ip address for cloud-init, then its instant new kvm.