Concurrent cloning of VM?

H

Hyien

Member
Jun 18, 2021
95
3
13
35
is it possible to clone a VM via concurrent requests?
I'm getting 500 can't lock file '/var/lock/qemu-server/lock-100.conf' - got timeout when I try.
 
no, a clone requires to lock the source to prevent modification, sharing this lock is not possible.
 
but it's not making modifications, only reading the source. :confused:
in any case, is there a way to wait/queue the request until the lock becomes available rather than failing?
 
the lock ensures no-one else can modify the guest during the clone. only one task can hold it. it will be queued for up to 60s IIRC, beyond that there is no queuing mechanism.
 
There are several other things that are happening during the clone operation:
- system has to find next available vmid, confirm its not in use
- a disk/slice/file/etc needs to be allocated from the storage, some backends are faster than others. Even in thin clone operation there could be backend contention.
- If you have a cloud-init disk, its not cloned and needs to be generated
- Even with template, although the disk is RO you are not prevented from modifying the configuration of the Template, ie adding/removing a device.

And this is just on the surface. Although I am not a PVE developer, I can see many potentials for a race condition. Faster systems may fare better, but may behave differently under load. The only way to have predictable result across variety of systems is to use a lock.


Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox
 
bbgeek17 said:
There are several other things that are happening during the clone operation:
- system has to find next available vmid, confirm its not in use
Click to expand...
yes, but the vmid is reserved (config file is created) when the clone start.
(I don't talk about launching clone exactly at the same time, only with 1 or 2s interval)


bbgeek17 said:
- a disk/slice/file/etc needs to be allocated from the storage, some backends are faster than others. Even in thin clone operation there could be backend contention.
Click to expand...
I don't care, I have full nvme ceph cluster ;)

bbgeek17 said:
- If you have a cloud-init disk, its not cloned and needs to be generated
Click to expand...
not related, it's generated at vm startup

bbgeek17 said:
- Even with template, although the disk is RO you are not prevented from modifying the configuration of the Template, ie adding/removing a device.
Click to expand...
adding a write only lock could prevent change, and still allow read.

bbgeek17 said:
And this is just on the surface. Although I am not a PVE developer, I can see many potentials for a race condition. Faster systems may fare better, but may behave differently under load. The only way to have predictable result across variety of systems is to use a lock.
Click to expand...
Anyway, you can create 20 templates, and clone them in parallel. you still don't have any protection about the load.


(for my personnal usecase, I need to clone sometimes around 500vms, 2 min by vm, sequentially it's really too slow, so I create multiple templates as workaround)
 
Last edited:
I think creating 500VMs is probably not something that 90% of users do. But unique requirements require unique solutions. The trick, of course, is to use API directly to avoid the time cost of the language interpreter startup. This also avoids some general locking. Here is a procedure to prepare 500 VMs in under 20 minutes. The backend storage is Blockbridge . For the case below thin clones are used so the footprint is as minimal as it gets.

Create a template as you wish (~20sec):
Bash: 
#!/usr/bin/bash
VMID=${1-"100"}
NAME=vm$VMID
OSIMAGE=cirros-0.5.2-x86_64-disk.img
STORAGE=blockbridge

qm create $VMID --memory 128 --name template --bootdisk virtio0 --socket 1 --onboot no
qm importdisk $VMID ./$OSIMAGE $STORAGE --format raw
qm set $VMID --scsihw virtio-scsi-pci --scsi0 $STORAGE:vm-$VMID-disk-0
qm set $VMID -net0 e1000,bridge=vmbr0,firewall=1
qm set $VMID --serial0 socket --vga serial0
qm template $VMID

Create 500 VMs (32 seconds) :
for i in {200..700};do ./5_vm_create.api $i;done
Bash: 
#!/usr/bin/bash
curl  --insecure --cookie "$(<./.cookie)" --header "$(<./.csrftoken)" -X POST https://localhost:8006/api2/json/nodes/pve7test1/qemu \
--data-urlencode "onboot=0" \
--data-urlencode "bootdisk=virtio0" \
--data-urlencode "name=vm$1" \
--data-urlencode "serial0=socket" \
--data-urlencode "vga=serial0" \
--data-urlencode "virtio0=blockbridge:vm-$1-disk-0" \
--data memory=128 \
--data vmid=$1

Clone the base disk directly on storage via API. I am using Blockbridge SDS (12min on a node running in a VM with limited resources):
for i in {200..700};do ./6_disk_clone.sh $i;done
Bash: 
#!/usr/bin/bash
API_NODE=blockbridge-api
TOKEN=$(curl -sk -X POST -H 'Content-Type: application/json' -d '{"username":"proxmox","password":"proxmox"}' "https://"$API_NODE"/api/oauth2/token" | jq -r .access_token)
snapshotid=$(curl -skX GET https://$API_NODE/api/snapshot -H "Authorization: Bearer $TOKEN"|jq -r .[].serial)
RESERVE=$(curl -skX GET https://$API_NODE/api/snapshot -H "Authorization: Bearer $TOKEN"|jq -r .[].capacity)

token=$(curl -skX POST https://$API_NODE/api/product/search \
   -H 'Content-Type: application/json' \
   -H "Authorization: Bearer $TOKEN" \
   -d '{"capacity":"'$RESERVE'"}' | jq -r .products[].token )

curl -skX POST https://$API_NODE/api/vss \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $TOKEN" \
  -d \
'{
  "token":"'$token'",
  "vss":{"label": "blockbridge:vm-'$1'-disk-0"},
  "disk":{"create": true,
  "label": "base",
  "src": "snapshot",
  "snapshot_id": "'$snapshotid'"}
}'

Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox
 
Last edited:
You must log in or register to reply here.
Top Bottom