ZFS: Linked clone feature is not supported for drive 'virtio0' (500)

[jf21]

Hello,

i've got the following setup:
Proxmox 6.3 cluster with 2 servers, both running on ZFS.
SiteA is replicating its VM to SiteB. I've created a VM template on SiteB which i'd like to use to create clones (both full and linked). I can create full clones without problems
but once i try linked clones i do get an error:
Linked clone feature is not supported for drive 'virtio0' (500)

Afaik linked clones should work fine on ZFS. I am not sure if the replication is a problem in this setup or what could cause this. Any ideas?

Br,
Jonas

 

[fabian]

please post the full VM and storage configs..

 

[jf21]

(This is from the replication target, SiteB):

zfspool: vm1
pool zpool1/vm1
content rootdir,images
mountpoint /zpool1/vm1
sparse 1

and the VM to-be-cloned itself:

agent: 1
boot: order=ide2;net0
cores: 3
ide2: none,media=cdrom
memory: 15258
name: Server-Replicated
net0: e1000=42:37:6E:C0:76:9A,bridge=vmbr0
numa: 0
ostype: win10
scsihw: virtio-scsi-pci
smbios1: uuid=bf6f8214-cdf6-494f-be4c-90d00e20f1b3
sockets: 2
template: 1
virtio0: vm1:vm-100-disk-0,discard=on,size=110G
virtio1: vm1:vm-100-disk-1,discard=on,size=110G
vmgenid: a3655e79-6153-49d6-af8b-f65152aae4e0

 

[fabian]

that is not a proper template - how did you create it?

 

[jf21]

I converted a VM to the template. Whats wrong with it?

 

[fabian]

the disks aren't base disks - conversion to a template would normally convert the disks as well. creating a linked clone checks whether the disks are base disks..

 

[jf21]

Ok, ive deleted the template and recreated it. Now i can see the base disks and i can create a linked clone. However now it appears the replication is broken. The replication still replicates to vm1:vm-100-disk-0 while the base disk is vm1:base-100-disk-0. So the linked or full clones will always be outdated as the template is using its own (base) disks which arent the target of the replication. Or am i missing something?

 

[jf21]

Just an update: after i recreate the template and the replication started again the base disks are gone. They are no longer on the ZFS but the template still references them. Now if you try to use the template to create a clone (full or linked) it will result in an error:
TASK ERROR: clone failed: timeout: no zvol device link for 'base-100-disk-1' found after 300 sec found.

Which makes sense as this disk is no longer there.
So the basic question is: are clones really working/supported with replicated VM's on ZFS [on the the replicated target]?

 

[fabian]

I am not quite sure what you are doing - the VM does not exist on the replicaton target, just the disks? so there is no config to create a linked clone from..

 

[jf21]

Well, this is how to reproduce:
(Create cluster of SiteA and SiteB)
SiteA: Create a VM1 with ZFS storage
SiteA: Configure replication of VM disks of VM1 to SiteB
SiteB: Create same VM with replicated disks now present
SiteB: Convert VM to template (this changes disks from "vm-100-disk-0" to "base-100-disk-0")
Now its possible to do full clones and linked clones.
However after time x (x beeing the replication interval) the base-disk's disappear and the vm-disk's reappear (likely because of the replication).
Now the template can no longer be used as it references the base-disk's which no longer exists.

 

[fabian]

yeah, that is definitely not a supported use case, sorry.. the replication target is not supposed to be modified on the target side

 

[jf21]

Ok, thats what i feared. Well...are there plans to implent this or is this a too uncommon use-case (i am not a pro on ZFS so i am not sure if its technically possible)?

 

[fabian]

no plans, as that is not what replication is for. it will always overwrite changes you do on the target side to keep the target in sync with the source..

 

[jf21]

Ok, i get that. Our use case is that we want to have data replicated to another site. If something goes wrong with siteA (initial site) then we still have data on SiteB. Now we do not want to touch that data (forensics) but would like to create clones for e.g. data recovery. Key is to not change the replicated data at all. Full clones work (as i said) but linked clones would be alot faster due to the size. But we'll work with full clones then.