I've made a tool to import Cloud Images

[Master of the Wind]

Hello guys!

I've made a Python script that makes importing Cloud Images easy.

Instead of manually search and download distros' cloud ready images, and then do the steps in the documentation, this script gives you a list to pick a distro, and then automatically download and imports the image.

I've tried to do the same that Proxmox does with Container images.

The script runs local on the server, basically it sends "qm" commands when need to interact with Proxmox. It does not use the API.

I've uploaded to Github, feel free to use it, it's public: https://github.com/ggMartinez/Proxmox-Cloud-Image-Importer . Also, it has an installer script to add Python PIP, Git, and a few python packages.

Runs well on Proxmox 7 and Proxmox 8.

I've created a public gists that it's a JSON file with the name and link for each of the images, it's also public. Later I'll look for a better way to keep the list, at least something that's not that manual.

Any feedback is appreciated!!!

 

[LnxBil]

It does not use the API.

That's a pity. Maybe look into that and pack your importer into a Docker container image. I would newer run stuff from the internet on my hypervisor and wouldn't install additional packages, especially development packages either.

I was under the impression, that those cloud images are tailored for Azure, AWS and so on and do not work as well as they could with Proxmox VE due to specialized kernel versions.

Who provides thoses images? How often are they updated?

We use our own infrastructure for LX(C) templates that are updated regularly and are 100% under our own control, for KVM VMs we fully rely on network installation and automation to always have the newest packages and configuration on install. I always thought, that the template system in general is too limited (especially KVM templates, which are not updateable).

 

[Master of the Wind]

I truly don't understad your scrutiny.

Your impresion was wrong. The images are generic cloud images from the providers that, as I wrote on the readme, was tested on both Proxmox 7 and 8, and works. You thought that was not tested? The ones tailored to AWS or Azure are other files. And also, if you check, the images are from their official providers. I only made a JSON file with the URLs.

The script uses this Github Gist as a source. These are the sources for the images:

https://repo.almalinux.org/almalinux/9/cloud/x86_64/images/AlmaLinux-9-GenericCloud-latest.x86_64.qcow2
https://repo.almalinux.org/almalinux/9/cloud/x86_64/images/AlmaLinux-9-GenericCloud-latest.x86_64.qcow2
https://repo.almalinux.org/almalinux/8/cloud/x86_64/images/AlmaLinux-8-GenericCloud-latest.x86_64.qcow2
https://cloud.centos.org/centos/9-stream/x86_64/images/CentOS-Stream-GenericCloud-9-latest.x86_64.qcow2
https://cloud-images.ubuntu.com/noble/current/noble-server-cloudimg-amd64.img
https://cloud-images.ubuntu.com/jammy/current/jammy-server-cloudimg-amd64.img
https://download.opensuse.org/repositories/Cloud:/Images:/Leap_15.6/images/openSUSE-Leap-15.6.x86_64-NoCloud.qcow2
https://download.opensuse.org/repositories/Cloud:/Images:/Leap_15.5/images/openSUSE-Leap-15.5.x86_64-NoCloud.qcow2
https://download.opensuse.org/repositories/Cloud:/Images:/Leap_15.4/images/openSUSE-Leap-15.4.x86_64-NoCloud.qcow2
https://download.fedoraproject.org/pub/fedora/linux/releases/40/Cloud/x86_64/images/Fedora-Cloud-Base-Generic.x86_64-40-1.14.qcow2

As you can see, all came from their vendors' side. Not hosted on some random server in nowhere to be known. Just a plain JSON file. And all of them are generic, usually they have the cloud provider tailored to on the name (For example, for Fedora's EC2 image the filename is "Fedora-Cloud-Base-AmazonEC2.x86_64-40-1.14.raw.xz", and for Ubuntu's Azure image the filename is "jammy-server-cloudimg-amd64-azure.vhd.tar.gz")

Did you even check the repository?

You would not run stuff from the internet on your Hypervisor, but you are open to run a tool on a Docker container that connects and does stuff on your Hypervisor, that also comes from the internet?

Doing net installs takes time. Time that would not be needed to take if I can have a template ready to be cloned, and then provision stuff with Ansible. Maybe it's a habit aquired from the ages that download speed was crap in my country.

I don't work that much with LXC. I don't even work that much with Proxmox (or any plain hypervisor), in my 8hs job we use AWS at big scale, and for other projects I'm in, we use CloudStack. I work mostly with VMs for network services (call me old fashioned, but I don't trust LXC's isolation 100%), and then for my apps I tend to work on an environment focused on Kubernetes Clusters.

It's seems that you are more focused on LXC. That's not bad, just that this script would be of not much use for you. There's no point focusing on LXC for this script, Proxmox already has the option to import LXC templates on the UI.

Finally, I made the script for my Proxmox Server in on my Homelab. And after using it, I polished it a little bit and thought that could be useful to someone. The script is open. To use it, or to don't use it.

 

[LnxBil]

I wrote on the readme, was tested on both Proxmox 7 and 8

I'm not talking about a one-time test if they work. Regular testing by the vendor if they work. One thing that does not work with most cloud images I tried is the use of the virtualized watchdog. Most Debian-based cloud images use the cloud kernel and the driver for the watchdog is missing. Also a lot of vmware drivers for high precision timing and vmware framebuffer console support is not included in the cloud kernel. These are thing people care about and that is what I was talking about.

Did you even check the repository?

The repository yes and I did not find a repo file. Having the list outside of the repo is confusing and potentially a security risk. Why not provide the list with the repo?

You would not run stuff from the internet on your Hypervisor, but you are open to run a tool on a Docker container that connects and does stuff on your Hypervisor, that also comes from the internet?

The API is much more secure and restrictive than running shell commands.

Doing net installs takes time. Time that would not be needed to take if I can have a template ready to be cloned, and then provision stuff with Ansible. Maybe it's a habit aquired from the ages that download speed was crap in my country.

Updating the images afterwards is better? Normally they don't get updates regularly and need packages thereafter. The network install time is therefore more pedicable than using the cloud image. If the cloud image is new, the install time may be shorter, yet while time goes one and update pile up, things may change. Also having a local mirror will solve the speed problems (for both setups). You can also still go with ansible after installing via network.

 

[hvisage]

I've been trying to use FAI https://fai-project.org/FAIme/ to create cloud-init images, but ran into troubles (can't recall was 1.5years ago) but if somebody could make that "simple" for server type installation, it could handle about all the concerns above

 

[proxuser77]

And it's exactly threads like this why I will never share my scripts publicly.

Obviously, logging in as root and start Bash- or Python-scripts to download or deploy VMs is not suitable for an enterprise environment. But honestly, for my homelab I'm not going to learn Terraform or Ansible just to deploy a quick VM, and in that context it’s absolutely legitimate to do it this way. I mean, why does a system even have a CLI interface if you're not “allowed” to use it? Come on.

You might be right from a strict enterprise perspective, but bursting into the thread like that was really unnecessary. And you don’t exactly come across as the ultimate pro either if you think cloud images are only meant for Azure, AWS, and the like. The “cloud” in the name comes from cloud-init, and of course there are generic images that can be used on Proxmox.

Also, you can customize them with virt-customize if a feature is missing. Combined with the qm commands and virt-customize, you can easily build a nice Bash or Python script that asks a few questions and then sets up a fully configured VM, incl. user accounts, SSH keys, Docker, LAMP stack, whatever. It’s perfect for quickly testing things, and btw, I see no reason why cloud images shouldn’t also be used on production systems.

I mean, Hetzner, Linode, and countless other VPS providers use these images as well (sometimes more, sometimes less customized). And again: you can customize them yourself, just let your script run `virt-customize` after downloading the image, and boom, you’ve got your custom image in one go.

Sure, doing it via scripts directly on the host isn’t the highest art of DevOps, but using anecdotal arguments to talk down OP’s approach isn’t exactly the highest art either. At the very least, you could have taken the time to actually look at the script, instead of dismissing everything outright from the start.

By the way, the generic Debain cloud images, for example, use exactly the same kernel as a manual installation from the net-install ISO.

 

[proxuser77]

Updating the images afterwards is better?

Yes, at least compared to installing manually from the net-installer ISO, because with the cloud-image it only downloads the packages that actually have updates available — obviously. And that happens automatically.

I definitely have no desire to go through the Debian installer every single time I want to spin up a quick VM. With the cloud images and my Bash scripts, I just choose the OS I want, how much memory, etc., and less than two minutes later I can SSH into the VM and start testing whatever I want. At that point, if I were using the regular installer, I’d be lucky if I had already reached the “participate in the popularity contest?” question, and I’d probably still be busy unchecking the GNOME desktop and checking the SSH server option.

Oh, and before all that, you also have to click through the Proxmox wizard. So by the time the Debian ISO has even finished booting, the VM created with my script is most likely already rebooting because it has finished installing whatever updates were available.