[SOLVED] Cloning template to zvol questions

[blueduckdock]

I did some searches but didn't quite see what I'm looking for.

Right now I use the terraform proxmox provider to provision Fedora CoreOS nodes (among other things.) This all works decently well save the fact that clone is quite slow (when doing multiple at once). This basically (I think) is due to the fact I take the CoreOS qemu img, make a new disk that uses it as a backing disk, then create the template. The template then does a full clone to a VM where the ignition runs and does all the actual install and config. I am cloning to a zvol. So it takes the backing disk (10GB IIRC) and then lays it down to a zvol.

My question- how does cloning work when cloning to a new zvol? Is there someway with what I'm doing to clone a zvol quickly? I'm imagining having a raw file of some sort (or writing the raw file to a zvol), attaching the zvol to a template, then proxmox does some kind of zfs clone to a new zvol which should be near instant? Is this a thing at all or is my qemu clone to zvol the best I can do?

 

[aaron]

I am not 100% clear how you do it. Do you clone a normal VM? Then yeah, that is going to take some time as the disk is copied.

If you convert a VM to a template, then you can create "linked clones" which will be almost instantaneous as the linked clones will reference the template as their base disk image. Anything that isn't yet on their own disk image will be read from the base disk image of the template.

The longer the VMs keep running, the more data they will write into their own disk images and therefore rely less and less on the base image.

Once a VM is converted to a template, you cannot boot it up or change the disk image as that would corrupt the linked clones.

 

[blueduckdock]

Thanks for the reply aaron. I did get some time to mess with it and I think I've "solved" it (not really- it's just impossible at this time to do what I'm looking to do.)

For reference for anyone wishing to do the same- you can template out and clone qcow2 easily enough (using linked clones even to speed it up).

This is done by using the following "args" parameter either in your terraform code or command line.

args                   = "-fw_cfg name=opt/com.coreos/config,file=some/host/path/here/${var.name}.ign"

I use var.name in my terraform code so the templated ignition is named for the node it was configured for (with static IP, hostname, users, etc. etc.)

Qcow2 should work fine on some storages (notably LVM, etc.) BUT wouldn't be recommended for zvols where you are writing to a raw image. What I have done (and works) up to this point is to take the Fedora CoreOS qcow2 image and then convert it to a template. I can then have terraform clone that to a zvol and the qemu arg still works to provision. This works well except save you're doing a full clone every time which takes the qcow2 and writes it to a zvol.

My goal at this point was to have a templated VM that uses a zvol which I can then near instantly clone using linked clones. I figured I could take the provided Fedora CoreOS raw file, dd it to the zvol of the VM that will be the template, then template it and be done. The config would still need to be passed though. This is the issue though- Fedora CoreOS as of January 2023 has NO cloud-init support and qemu args (as I've tried) do not work here with the raw file.

Ticket to track for support is here- https://github.com/coreos/fedora-coreos-tracker/issues/736

I wanted to pass it hopefully using "cicustom" with the terraform proxmox provider (or you could use that via API/CLI as well.) It just simply doesn't support it at this time.

So, in summary, if you're doing qcow2 to qcow2 (ie. to a directory storage, etc.) then you should be fine to do linked clones and just use the args parameter. If you're laying down VMs to zvols or some other raw, you're going to have to use full clones off a qcow2 template until the CoreOS team adds support of some kind for cloud-init.

Thanks again aaron.

Edit- What I did try that might seem to work (as close as I can get) is to

1. download the qcow2
2. use qemu-img to convert that to raw -

   qemu-img convert -f qcow -O raw downloaded.qcow2 fcos37.raw

3. dd the raw file to the soon-to-be-templated VM's zvol -

   dd if=fcos37.raw of=/dev/zvol/<poolname>/data/vm-<vmid>-disk-0 bs=1M

4. convert it to template

It can still use the qemu args provided above and using linked clones seems to work now which is near instant. Hoping this allows me to also up the parallel operations in terraform (which had issues with timeouts as multiple clones take time and apparently each needs a lock when done, or at least I've only ever been able to do 2 at once before it stalls.)