Concurrent cloning of VM?
- Thread starter Hyien
- Start date
@Hyien - there are many ways to avoid failure, it really depends on what you are trying to achieve. Some are prettier than others. You can:
- sleep 120
- use API to initiate the clone, get the task back, keep checking for task status
- put the command in a loop and run it until it succeeds
Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox
- sleep 120
- use API to initiate the clone, get the task back, keep checking for task status
- put the command in a loop and run it until it succeeds
Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox
There are several other things that are happening during the clone operation:
- system has to find next available vmid, confirm its not in use
- a disk/slice/file/etc needs to be allocated from the storage, some backends are faster than others. Even in thin clone operation there could be backend contention.
- If you have a cloud-init disk, its not cloned and needs to be generated
- Even with template, although the disk is RO you are not prevented from modifying the configuration of the Template, ie adding/removing a device.
And this is just on the surface. Although I am not a PVE developer, I can see many potentials for a race condition. Faster systems may fare better, but may behave differently under load. The only way to have predictable result across variety of systems is to use a lock.
Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox
- system has to find next available vmid, confirm its not in use
- a disk/slice/file/etc needs to be allocated from the storage, some backends are faster than others. Even in thin clone operation there could be backend contention.
- If you have a cloud-init disk, its not cloned and needs to be generated
- Even with template, although the disk is RO you are not prevented from modifying the configuration of the Template, ie adding/removing a device.
And this is just on the surface. Although I am not a PVE developer, I can see many potentials for a race condition. Faster systems may fare better, but may behave differently under load. The only way to have predictable result across variety of systems is to use a lock.
Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox
yes, but the vmid is reserved (config file is created) when the clone start.
(I don't talk about launching clone exactly at the same time, only with 1 or 2s interval)
I don't care, I have full nvme ceph cluster
not related, it's generated at vm startup
adding a write only lock could prevent change, and still allow read.
Anyway, you can create 20 templates, and clone them in parallel. you still don't have any protection about the load.
(for my personnal usecase, I need to clone sometimes around 500vms, 2 min by vm, sequentially it's really too slow, so I create multiple templates as workaround)
Last edited:
I think creating 500VMs is probably not something that 90% of users do. But unique requirements require unique solutions. The trick, of course, is to use API directly to avoid the time cost of the language interpreter startup. This also avoids some general locking. Here is a procedure to prepare 500 VMs in under 20 minutes. The backend storage is Blockbridge . For the case below thin clones are used so the footprint is as minimal as it gets.
Create a template as you wish (~20sec):
Create 500 VMs (32 seconds) :
for i in {200..700};do ./5_vm_create.api $i;done
Clone the base disk directly on storage via API. I am using Blockbridge SDS (12min on a node running in a VM with limited resources):
for i in {200..700};do ./6_disk_clone.sh $i;done
Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox
Create a template as you wish (~20sec):
Bash:
#!/usr/bin/bash
VMID=${1-"100"}
NAME=vm$VMID
OSIMAGE=cirros-0.5.2-x86_64-disk.img
STORAGE=blockbridge
qm create $VMID --memory 128 --name template --bootdisk virtio0 --socket 1 --onboot no
qm importdisk $VMID ./$OSIMAGE $STORAGE --format raw
qm set $VMID --scsihw virtio-scsi-pci --scsi0 $STORAGE:vm-$VMID-disk-0
qm set $VMID -net0 e1000,bridge=vmbr0,firewall=1
qm set $VMID --serial0 socket --vga serial0
qm template $VMIDCreate 500 VMs (32 seconds) :
for i in {200..700};do ./5_vm_create.api $i;done
Bash:
#!/usr/bin/bash
curl --insecure --cookie "$(<./.cookie)" --header "$(<./.csrftoken)" -X POST https://localhost:8006/api2/json/nodes/pve7test1/qemu \
--data-urlencode "onboot=0" \
--data-urlencode "bootdisk=virtio0" \
--data-urlencode "name=vm$1" \
--data-urlencode "serial0=socket" \
--data-urlencode "vga=serial0" \
--data-urlencode "virtio0=blockbridge:vm-$1-disk-0" \
--data memory=128 \
--data vmid=$1Clone the base disk directly on storage via API. I am using Blockbridge SDS (12min on a node running in a VM with limited resources):
for i in {200..700};do ./6_disk_clone.sh $i;done
Bash:
#!/usr/bin/bash
API_NODE=blockbridge-api
TOKEN=$(curl -sk -X POST -H 'Content-Type: application/json' -d '{"username":"proxmox","password":"proxmox"}' "https://"$API_NODE"/api/oauth2/token" | jq -r .access_token)
snapshotid=$(curl -skX GET https://$API_NODE/api/snapshot -H "Authorization: Bearer $TOKEN"|jq -r .[].serial)
RESERVE=$(curl -skX GET https://$API_NODE/api/snapshot -H "Authorization: Bearer $TOKEN"|jq -r .[].capacity)
token=$(curl -skX POST https://$API_NODE/api/product/search \
-H 'Content-Type: application/json' \
-H "Authorization: Bearer $TOKEN" \
-d '{"capacity":"'$RESERVE'"}' | jq -r .products[].token )
curl -skX POST https://$API_NODE/api/vss \
-H "Content-Type: application/json" \
-H "Authorization: Bearer $TOKEN" \
-d \
'{
"token":"'$token'",
"vss":{"label": "blockbridge:vm-'$1'-disk-0"},
"disk":{"create": true,
"label": "base",
"src": "snapshot",
"snapshot_id": "'$snapshotid'"}
}'Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox
Last edited: