ZFS Storage Replication...

Hi... I have here just one PowerEdge Dell 2950 as a lab server...
With this server, I enable nested virtulization, in order to create a 3 node cluster.
I have deploy 3 VM with Proxmox 5, inside Proxmox 5 server...
I also try Storage Replication.
Offline, the migration is ok.
But online, I always get this error:

qm migrate 100 prox02 --online --with-local-disks
2017-08-26 21:31:57 starting migration of VM 100 to node 'prox02' (10.1.1.20)
2017-08-26 21:31:58 found local disk 'stg-zfs:vm-100-disk-1' (in current VM config)
2017-08-26 21:31:58 copying disk images
2017-08-26 21:31:58 starting VM 100 on remote node 'prox02'
22017-08-26 21:32:05 start remote tunnel
2017-08-26 21:32:06 ssh tunnel ver 1
2017-08-26 21:32:06 starting storage migration
2017-08-26 21:32:06 scsi0: start migration to to nbd:10.1.1.20:60000:exportname=drive-scsi0
drive mirror is starting for drive-scsi0
drive-scsi0: transferred: 0 bytes remaining: 34359738368 bytes total: 34359738368 bytes progression: 0.00 % busy: 1 ready: 0
drive-scsi0: transferred: 102760448 bytes remaining: 34256977920 bytes total: 34359738368 bytes progression: 0.30 % busy: 1 ready: 0
drive-scsi0: transferred: 205520896 bytes remaining: 34154217472 bytes total: 34359738368 bytes progression: 0.60 % busy: 1 ready: 0
drive-scsi0: transferred: 373293056 bytes remaining: 33986445312 bytes total: 34359738368 bytes progression: 1.09 % busy: 1 ready: 0
drive-scsi0: transferred: 525336576 bytes remaining: 33834401792 bytes total: 34359738368 bytes progression: 1.53 % busy: 1 ready: 0
drive-scsi0: transferred: 649068544 bytes remaining: 33710669824 bytes total: 34359738368 bytes progression: 1.89 % busy: 1 ready: 0
drive-scsi0: transferred: 693108736 bytes remaining: 33666629632 bytes total: 34359738368 bytes progression: 2.02 % busy: 1 ready: 0
drive-scsi0: transferred: 703594496 bytes remaining: 33656143872 bytes total: 34359738368 bytes progression: 2.05 % busy: 1 ready: 0
drive-scsi0: transferred: 716177408 bytes remaining: 33643560960 bytes total: 34359738368 bytes progression: 2.08 % busy: 1 ready: 0
drive-scsi0: transferred: 728760320 bytes remaining: 33630978048 bytes total: 34359738368 bytes progression: 2.12 % busy: 1 ready: 0
drive-scsi0: transferred: 736100352 bytes remaining: 33623638016 bytes total: 34359738368 bytes progression: 2.14 % busy: 1 ready: 0
drive-scsi0: transferred: 751828992 bytes remaining: 33607909376 bytes total: 34359738368 bytes progression: 2.19 % busy: 1 ready: 0
drive-scsi0: transferred: 760217600 bytes remaining: 33599520768 bytes total: 34359738368 bytes progression: 2.21 % busy: 1 ready: 0
drive-scsi0: transferred: 775946240 bytes remaining: 33583792128 bytes total: 34359738368 bytes progression: 2.26 % busy: 1 ready: 0
drive-scsi0: transferred: 789577728 bytes remaining: 33570160640 bytes total: 34359738368 bytes progression: 2.30 % busy: 1 ready: 0
drive-scsi0: transferred: 810549248 bytes remaining: 33549189120 bytes total: 34359738368 bytes progression: 2.36 % busy: 1 ready: 0
drive-scsi0: transferred: 818937856 bytes remaining: 33540800512 bytes total: 34359738368 bytes progression: 2.38 % busy: 1 ready: 0
drive-scsi0: transferred: 835715072 bytes remaining: 33524023296 bytes total: 34359738368 bytes progression: 2.43 % busy: 1 ready: 0
drive-scsi0: transferred: 854589440 bytes remaining: 33505148928 bytes total: 34359738368 bytes progression: 2.49 % busy: 1 ready: 0
drive-scsi0: transferred: 867172352 bytes remaining: 33492566016 bytes total: 34359738368 bytes progression: 2.52 % busy: 1 ready: 0
drive-scsi0: transferred: 882900992 bytes remaining: 33476837376 bytes total: 34359738368 bytes progression: 2.57 % busy: 1 ready: 0
drive-scsi0: transferred: 893386752 bytes remaining: 33466351616 bytes total: 34359738368 bytes progression: 2.60 % busy: 1 ready: 0
drive-scsi0: transferred: 902823936 bytes remaining: 33456914432 bytes total: 34359738368 bytes progression: 2.63 % busy: 1 ready: 0
drive-scsi0: transferred: 915406848 bytes remaining: 33444331520 bytes total: 34359738368 bytes progression: 2.66 % busy: 1 ready: 0
drive-scsi0: transferred: 921698304 bytes remaining: 33438040064 bytes total: 34359738368 bytes progression: 2.68 % busy: 1 ready: 0
drive-scsi0: transferred: 932184064 bytes remaining: 33427554304 bytes total: 34359738368 bytes progression: 2.71 % busy: 1 ready: 0
drive-scsi0: transferred: 942669824 bytes remaining: 33417068544 bytes total: 34359738368 bytes progression: 2.74 % busy: 1 ready: 0
drive-scsi0: transferred: 973078528 bytes remaining: 33386659840 bytes total: 34359738368 bytes progression: 2.83 % busy: 1 ready: 0
drive-scsi0: transferred: 985661440 bytes remaining: 33374076928 bytes total: 34359738368 bytes progression: 2.87 % busy: 1 ready: 0
drive-scsi0: transferred: 1001390080 bytes remaining: 33358348288 bytes total: 34359738368 bytes progression: 2.91 % busy: 1 ready: 0
drive-scsi0: transferred: 1009778688 bytes remaining: 33349959680 bytes total: 34359738368 bytes progression: 2.94 % busy: 1 ready: 0
drive-scsi0: transferred: 1019215872 bytes remaining: 33340522496 bytes total: 34359738368 bytes progression: 2.97 % busy: 1 ready: 0
drive-scsi0: transferred: 1029701632 bytes remaining: 33330036736 bytes total: 34359738368 bytes progression: 3.00 % busy: 1 ready: 0
drive-scsi0: transferred: 1048576000 bytes remaining: 33311162368 bytes total: 34359738368 bytes progression: 3.05 % busy: 1 ready: 0
drive-scsi0: transferred: 1068498944 bytes remaining: 33291239424 bytes total: 34359738368 bytes progression: 3.11 % busy: 1 ready: 0
drive-scsi0: transferred: 1081081856 bytes remaining: 33278656512 bytes total: 34359738368 bytes progression: 3.15 % busy: 1 ready: 0
drive-scsi0: transferred: 1093664768 bytes remaining: 33266073600 bytes total: 34359738368 bytes progression: 3.18 % busy: 1 ready: 0
drive-scsi0: transferred: 1105199104 bytes remaining: 33254539264 bytes total: 34359738368 bytes progression: 3.22 % busy: 1 ready: 0
drive-scsi0: transferred: 1113587712 bytes remaining: 33246150656 bytes total: 34359738368 bytes progression: 3.24 % busy: 1 ready: 0
drive-scsi0: transferred: 1124073472 bytes remaining: 33235664896 bytes total: 34359738368 bytes progression: 3.27 % busy: 1 ready: 0
drive-scsi0: transferred: 1146093568 bytes remaining: 33213644800 bytes total: 34359738368 bytes progression: 3.34 % busy: 1 ready: 0
drive-scsi0: transferred: 1163919360 bytes remaining: 33195819008 bytes total: 34359738368 bytes progression: 3.39 % busy: 1 ready: 0
drive-scsi0: transferred: 1175453696 bytes remaining: 33184284672 bytes total: 34359738368 bytes progression: 3.42 % busy: 1 ready: 0
drive-scsi0: transferred: 1189085184 bytes remaining: 33170653184 bytes total: 34359738368 bytes progression: 3.46 % busy: 1 ready: 0
drive-scsi0: transferred: 1198522368 bytes remaining: 33161216000 bytes total: 34359738368 bytes progression: 3.49 % busy: 1 ready: 0
drive-scsi0: transferred: 1217396736 bytes remaining: 33142341632 bytes total: 34359738368 bytes progression: 3.54 % busy: 1 ready: 0
drive-scsi0: transferred: 1228931072 bytes remaining: 33130807296 bytes total: 34359738368 bytes progression: 3.58 % busy: 1 ready: 0
drive-scsi0: transferred: 1247805440 bytes remaining: 33111932928 bytes total: 34359738368 bytes progression: 3.63 % busy: 1 ready: 0
drive-scsi0: transferred: 1262485504 bytes remaining: 33097252864 bytes total: 34359738368 bytes progression: 3.67 % busy: 1 ready: 0
drive-scsi0: transferred: 1271922688 bytes remaining: 33087815680 bytes total: 34359738368 bytes progression: 3.70 % busy: 1 ready: 0
drive-scsi0: transferred: 1284505600 bytes remaining: 33075232768 bytes total: 34359738368 bytes progression: 3.74 % busy: 1 ready: 0
drive-scsi0: transferred: 1299185664 bytes remaining: 33060552704 bytes total: 34359738368 bytes progression: 3.78 % busy: 1 ready: 0
drive-scsi0: transferred: 1317011456 bytes remaining: 33042726912 bytes total: 34359738368 bytes progression: 3.83 % busy: 1 ready: 0
drive-scsi0: transferred: 1323302912 bytes remaining: 33036435456 bytes total: 34359738368 bytes progression: 3.85 % busy: 1 ready: 0
drive-scsi0: transferred: 1341128704 bytes remaining: 33018609664 bytes total: 34359738368 bytes progression: 3.90 % busy: 1 ready: 0
drive-scsi0: transferred: 1388314624 bytes remaining: 32971423744 bytes total: 34359738368 bytes progression: 4.04 % busy: 1 ready: 0
drive-scsi0: transferred: 1479540736 bytes remaining: 32880197632 bytes total: 34359738368 bytes progression: 4.31 % busy: 1 ready: 0
drive-scsi0: transferred: 1485832192 bytes remaining: 32873906176 bytes total: 34359738368 bytes progression: 4.32 % busy: 1 ready: 0
drive-scsi0: transferred: 1485832192 bytes remaining: 32873906176 bytes total: 34359738368 bytes progression: 4.32 % busy: 1 ready: 0
drive-scsi0: transferred: 1485832192 bytes remaining: 32873906176 bytes total: 34359738368 bytes progression: 4.32 % busy: 1 ready: 0
drive-scsi0: transferred: 1485832192 bytes remaining: 32873906176 bytes total: 34359738368 bytes progression: 4.32 % busy: 1 ready: 0
drive-scsi0: transferred: 1485832192 bytes remaining: 32873906176 bytes total: 34359738368 bytes progression: 4.32 % busy: 1 ready: 0
drive-scsi0: transferred: 1485832192 bytes remaining: 32873906176 bytes total: 34359738368 bytes progression: 4.32 % busy: 1 ready: 0
drive-scsi0: transferred: 1485832192 bytes remaining: 32873906176 bytes total: 34359738368 bytes progression: 4.32 % busy: 1 ready: 0
drive-scsi0: transferred: 1485832192 bytes remaining: 32873906176 bytes total: 34359738368 bytes progression: 4.32 % busy: 1 ready: 0
drive-scsi0: transferred: 1485832192 bytes remaining: 32873906176 bytes total: 34359738368 bytes progression: 4.32 % busy: 1 ready: 0
drive-scsi0: transferred: 1485832192 bytes remaining: 32873906176 bytes total: 34359738368 bytes progression: 4.32 % busy: 1 ready: 0
drive-scsi0: transferred: 1485832192 bytes remaining: 32873906176 bytes total: 34359738368 bytes progression: 4.32 % busy: 1 ready: 0
drive-scsi0: transferred: 1485832192 bytes remaining: 32873906176 bytes total: 34359738368 bytes progression: 4.32 % busy: 1 ready: 0
drive-scsi0: transferred: 1485832192 bytes remaining: 32873906176 bytes total: 34359738368 bytes progression: 4.32 % busy: 1 ready: 0
drive-scsi0: transferred: 1485832192 bytes remaining: 32873906176 bytes total: 34359738368 bytes progression: 4.32 % busy: 1 ready: 0
drive-scsi0: transferred: 1485832192 bytes remaining: 32873906176 bytes total: 34359738368 bytes progression: 4.32 % busy: 1 ready: 0
drive-scsi0: transferred: 1485832192 bytes remaining: 32873906176 bytes total: 34359738368 bytes progression: 4.32 % busy: 1 ready: 0
drive-scsi0: transferred: 1485832192 bytes remaining: 32873906176 bytes total: 34359738368 bytes progression: 4.32 % busy: 1 ready: 0
drive-scsi0: transferred: 1485832192 bytes remaining: 32873906176 bytes total: 34359738368 bytes progression: 4.32 % busy: 1 ready: 0
drive-scsi0: transferred: 1485832192 bytes remaining: 32873906176 bytes total: 34359738368 bytes progression: 4.32 % busy: 1 ready: 0
drive-scsi0: transferred: 1485832192 bytes remaining: 32873906176 bytes total: 34359738368 bytes progression: 4.32 % busy: 1 ready: 0
drive-scsi0: transferred: 1485832192 bytes remaining: 32873906176 bytes total: 34359738368 bytes progression: 4.32 % busy: 1 ready: 0
drive-scsi0: transferred: 1485832192 bytes remaining: 32873906176 bytes total: 34359738368 bytes progression: 4.32 % busy: 1 ready: 0
drive-scsi0: transferred: 1485832192 bytes remaining: 32873906176 bytes total: 34359738368 bytes progression: 4.32 % busy: 1 ready: 0
drive-scsi0: transferred: 1485832192 bytes remaining: 32873906176 bytes total: 34359738368 bytes progression: 4.32 % busy: 1 ready: 0
drive-scsi0: transferred: 1485832192 bytes remaining: 32873906176 bytes total: 34359738368 bytes progression: 4.32 % busy: 1 ready: 0
drive-scsi0: transferred: 1485832192 bytes remaining: 32873906176 bytes total: 34359738368 bytes progression: 4.32 % busy: 1 ready: 0
drive-scsi0: transferred: 1485832192 bytes remaining: 32873906176 bytes total: 34359738368 bytes progression: 4.32 % busy: 1 ready: 0
drive-scsi0: transferred: 1485832192 bytes remaining: 32873906176 bytes total: 34359738368 bytes progression: 4.32 % busy: 1 ready: 0
drive-scsi0: transferred: 1485832192 bytes remaining: 32873906176 bytes total: 34359738368 bytes progression: 4.32 % busy: 1 ready: 0
drive-scsi0: Cancelling block job
2017-08-26 21:33:52 ERROR: online migrate failure - mirroring error: VM 100 not running
2017-08-26 21:33:52 aborting phase 2 - cleanup resources
2017-08-26 21:33:52 migrate_cancel
2017-08-26 21:33:52 migrate_cancel error: VM 100 not running
drive-scsi0: Cancelling block job
2017-08-26 21:33:52 ERROR: VM 100 not running
2017-08-26 21:34:09 ERROR: command '/usr/bin/ssh -o 'BatchMode=yes' -o 'HostKeyAlias=prox02' root@10.1.1.20 pvesm free stg-zfs:vm-100-disk-1' failed: exit code 4
2017-08-26 21:34:12 ERROR: migration finished with problems (duration 00:02:15)
migration problems

I have ran the first syn replication and finish with success!
I do not have any sync schedule between the servers.

What it's wrong??
 
UPDATE!
When I try to migrate the offline VM to PROX01 to PROX02, now I get this error:

qm migrate 100 prox02 --with-local-disks
2017-08-26 21:41:30 starting migration of VM 100 to node 'prox02' (10.1.1.20)
2017-08-26 21:41:30 found local disk 'stg-zfs:vm-100-disk-1' (in current VM config)
2017-08-26 21:41:30 copying disk images
send from @ to ZFS/vm-100-disk-1@__migration__ estimated size is 6.89G
total estimated size is 6.89G
TIME SENT SNAPSHOT
ZFS/vm-100-disk-1 name ZFS/vm-100-disk-1 -
volume 'ZFS/vm-100-disk-1' already exists
command 'zfs send -Rpv -- ZFS/vm-100-disk-1@__migration__' failed: got signal 13
send/receive failed, cleaning up snapshot(s)..
2017-08-26 21:41:32 ERROR: Failed to sync data - command 'set -o pipefail && pvesm export stg-zfs:vm-100-disk-1 zfs - -with-snapshots 0 -snapshot __migration__ | /usr/bin/ssh -o 'BatchMode=yes' -o 'HostKeyAlias=prox02' root@10.1.1.20 -- pvesm import stg-zfs:vm-100-disk-1 zfs - -with-snapshots 0 -delete-snapshot __migration__' failed: exit code 255
2017-08-26 21:41:32 aborting phase 1 - cleanup resources
2017-08-26 21:41:32 ERROR: found stale volume copy 'stg-zfs:vm-100-disk-1' on node 'prox02'
2017-08-26 21:41:32 ERROR: migration aborted (duration 00:00:03): Failed to sync data - command 'set -o pipefail && pvesm export stg-zfs:vm-100-disk-1 zfs - -with-snapshots 0 -snapshot __migration__ | /usr/bin/ssh -o 'BatchMode=yes' -o 'HostKeyAlias=prox02' root@10.1.1.20 -- pvesm import stg-zfs:vm-100-disk-1 zfs - -with-snapshots 0 -delete-snapshot __migration__' failed: exit code 255
migration aborted
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE and Proxmox Mail Gateway. We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get your own in 60 seconds.

Buy now!