Unable to migrate disk with ZFS over iSCSI (Linux Guest)

[bkvamme]

Hi,
I have been playing around with ZFS over iSCSI, and got it working properly a couple of days ago, and was able to start moving the existing disks over without any issue. Container disks are hosted over NFS.

System info:
PVE 5.2, latest no-subscription updates installed
iSCSI host is one of the proxmox hosts using LIO (3 in a cluster, no HA)

I am able to migrate disks used on Windows VMs just fine, but the disks that are used on my Linux VMs (All Ubuntu Server 18.04.1) refuse. It makes no different if the VM is powered on or off. I get the error message:

create full clone of drive scsi0 (vmtank-disks:vm-127-disk-0)
TASK ERROR: storage migration failed: error with cfs lock 'storage-iscsi-x10srlf': Device /dev/vmtank/disks/vm-127-disk-1 is not a TYPE_DISK block device

Example on Windows VM:

()
create full clone of drive virtio1 (vmtank-disks:vm-123-disk-1)
drive mirror is starting for drive-virtio1
drive-virtio1: transferred: 0 bytes remaining: 1073741824 bytes total: 1073741824 bytes progression: 0.00 % busy: 1 ready: 0
drive-virtio1: transferred: 718274560 bytes remaining: 355467264 bytes total: 1073741824 bytes progression: 66.89 % busy: 1 ready: 0
drive-virtio1: transferred: 1073741824 bytes remaining: 0 bytes total: 1073741824 bytes progression: 100.00 % busy: 1 ready: 0
drive-virtio1: transferred: 1073741824 bytes remaining: 0 bytes total: 1073741824 bytes progression: 100.00 % busy: 1 ready: 0
drive-virtio1: transferred: 1073741824 bytes remaining: 0 bytes total: 1073741824 bytes progression: 100.00 % busy: 0 ready: 1
all mirroring jobs are ready
drive-virtio1: Completing block job...
drive-virtio1: Completed successfully.
drive-virtio1 : finished
TASK OK

I get a similar error message when trying to add a disk to a Linux VM. Windows VMs work fine, no issue adding disks with any bus.

VirtIO Block

update VM 129: -virtio2 iscsi-x10srlf:1
TASK ERROR: error with cfs lock 'storage-iscsi-x10srlf': Device /dev/vmtank/disks/vm-129-disk-1 is not a TYPE_DISK block device

VirtIO SCSI

()
update VM 129: -scsi2 iscsi-x10srlf:1
TASK ERROR: error with cfs lock 'storage-iscsi-x10srlf': Device /dev/vmtank/disks/vm-129-disk-2 is not a TYPE_DISK block device

Adding disks with SATA or IDE works fine.
IDE

update VM 129: -ide1 iscsi-x10srlf:1
TASK OK

SATA

update VM 129: -sata2 iscsi-x10srlf:1
TASK OK

What have I stumbled onto here?

 

[mir]

This /dev/vmtank/disks/vm-129-disk-1 is a local disk so you are not using ZFS over iSCSI. ZFS over iSCSI requires a remote server exposing a ZFS pool for which there is support in the ZFS over iSCSI storage type. You cannot migrate a local disk exposed via iscsi.