ZFS Storage Replication...

Discussion in 'Proxmox VE: Installation and configuration' started by Gilberto Ferreira, Aug 27, 2017.

  1. Gilberto Ferreira

    Joined:
    Aug 21, 2014
    Messages:
    47
    Likes Received:
    0
    Hi... I have here just one PowerEdge Dell 2950 as a lab server...
    With this server, I enable nested virtulization, in order to create a 3 node cluster.
    I have deploy 3 VM with Proxmox 5, inside Proxmox 5 server...
    I also try Storage Replication.
    Offline, the migration is ok.
    But online, I always get this error:

    qm migrate 100 prox02 --online --with-local-disks
    2017-08-26 21:31:57 starting migration of VM 100 to node 'prox02' (10.1.1.20)
    2017-08-26 21:31:58 found local disk 'stg-zfs:vm-100-disk-1' (in current VM config)
    2017-08-26 21:31:58 copying disk images
    2017-08-26 21:31:58 starting VM 100 on remote node 'prox02'
    22017-08-26 21:32:05 start remote tunnel
    2017-08-26 21:32:06 ssh tunnel ver 1
    2017-08-26 21:32:06 starting storage migration
    2017-08-26 21:32:06 scsi0: start migration to to nbd:10.1.1.20:60000:exportname=drive-scsi0
    drive mirror is starting for drive-scsi0
    drive-scsi0: transferred: 0 bytes remaining: 34359738368 bytes total: 34359738368 bytes progression: 0.00 % busy: 1 ready: 0
    drive-scsi0: transferred: 102760448 bytes remaining: 34256977920 bytes total: 34359738368 bytes progression: 0.30 % busy: 1 ready: 0
    drive-scsi0: transferred: 205520896 bytes remaining: 34154217472 bytes total: 34359738368 bytes progression: 0.60 % busy: 1 ready: 0
    drive-scsi0: transferred: 373293056 bytes remaining: 33986445312 bytes total: 34359738368 bytes progression: 1.09 % busy: 1 ready: 0
    drive-scsi0: transferred: 525336576 bytes remaining: 33834401792 bytes total: 34359738368 bytes progression: 1.53 % busy: 1 ready: 0
    drive-scsi0: transferred: 649068544 bytes remaining: 33710669824 bytes total: 34359738368 bytes progression: 1.89 % busy: 1 ready: 0
    drive-scsi0: transferred: 693108736 bytes remaining: 33666629632 bytes total: 34359738368 bytes progression: 2.02 % busy: 1 ready: 0
    drive-scsi0: transferred: 703594496 bytes remaining: 33656143872 bytes total: 34359738368 bytes progression: 2.05 % busy: 1 ready: 0
    drive-scsi0: transferred: 716177408 bytes remaining: 33643560960 bytes total: 34359738368 bytes progression: 2.08 % busy: 1 ready: 0
    drive-scsi0: transferred: 728760320 bytes remaining: 33630978048 bytes total: 34359738368 bytes progression: 2.12 % busy: 1 ready: 0
    drive-scsi0: transferred: 736100352 bytes remaining: 33623638016 bytes total: 34359738368 bytes progression: 2.14 % busy: 1 ready: 0
    drive-scsi0: transferred: 751828992 bytes remaining: 33607909376 bytes total: 34359738368 bytes progression: 2.19 % busy: 1 ready: 0
    drive-scsi0: transferred: 760217600 bytes remaining: 33599520768 bytes total: 34359738368 bytes progression: 2.21 % busy: 1 ready: 0
    drive-scsi0: transferred: 775946240 bytes remaining: 33583792128 bytes total: 34359738368 bytes progression: 2.26 % busy: 1 ready: 0
    drive-scsi0: transferred: 789577728 bytes remaining: 33570160640 bytes total: 34359738368 bytes progression: 2.30 % busy: 1 ready: 0
    drive-scsi0: transferred: 810549248 bytes remaining: 33549189120 bytes total: 34359738368 bytes progression: 2.36 % busy: 1 ready: 0
    drive-scsi0: transferred: 818937856 bytes remaining: 33540800512 bytes total: 34359738368 bytes progression: 2.38 % busy: 1 ready: 0
    drive-scsi0: transferred: 835715072 bytes remaining: 33524023296 bytes total: 34359738368 bytes progression: 2.43 % busy: 1 ready: 0
    drive-scsi0: transferred: 854589440 bytes remaining: 33505148928 bytes total: 34359738368 bytes progression: 2.49 % busy: 1 ready: 0
    drive-scsi0: transferred: 867172352 bytes remaining: 33492566016 bytes total: 34359738368 bytes progression: 2.52 % busy: 1 ready: 0
    drive-scsi0: transferred: 882900992 bytes remaining: 33476837376 bytes total: 34359738368 bytes progression: 2.57 % busy: 1 ready: 0
    drive-scsi0: transferred: 893386752 bytes remaining: 33466351616 bytes total: 34359738368 bytes progression: 2.60 % busy: 1 ready: 0
    drive-scsi0: transferred: 902823936 bytes remaining: 33456914432 bytes total: 34359738368 bytes progression: 2.63 % busy: 1 ready: 0
    drive-scsi0: transferred: 915406848 bytes remaining: 33444331520 bytes total: 34359738368 bytes progression: 2.66 % busy: 1 ready: 0
    drive-scsi0: transferred: 921698304 bytes remaining: 33438040064 bytes total: 34359738368 bytes progression: 2.68 % busy: 1 ready: 0
    drive-scsi0: transferred: 932184064 bytes remaining: 33427554304 bytes total: 34359738368 bytes progression: 2.71 % busy: 1 ready: 0
    drive-scsi0: transferred: 942669824 bytes remaining: 33417068544 bytes total: 34359738368 bytes progression: 2.74 % busy: 1 ready: 0
    drive-scsi0: transferred: 973078528 bytes remaining: 33386659840 bytes total: 34359738368 bytes progression: 2.83 % busy: 1 ready: 0
    drive-scsi0: transferred: 985661440 bytes remaining: 33374076928 bytes total: 34359738368 bytes progression: 2.87 % busy: 1 ready: 0
    drive-scsi0: transferred: 1001390080 bytes remaining: 33358348288 bytes total: 34359738368 bytes progression: 2.91 % busy: 1 ready: 0
    drive-scsi0: transferred: 1009778688 bytes remaining: 33349959680 bytes total: 34359738368 bytes progression: 2.94 % busy: 1 ready: 0
    drive-scsi0: transferred: 1019215872 bytes remaining: 33340522496 bytes total: 34359738368 bytes progression: 2.97 % busy: 1 ready: 0
    drive-scsi0: transferred: 1029701632 bytes remaining: 33330036736 bytes total: 34359738368 bytes progression: 3.00 % busy: 1 ready: 0
    drive-scsi0: transferred: 1048576000 bytes remaining: 33311162368 bytes total: 34359738368 bytes progression: 3.05 % busy: 1 ready: 0
    drive-scsi0: transferred: 1068498944 bytes remaining: 33291239424 bytes total: 34359738368 bytes progression: 3.11 % busy: 1 ready: 0
    drive-scsi0: transferred: 1081081856 bytes remaining: 33278656512 bytes total: 34359738368 bytes progression: 3.15 % busy: 1 ready: 0
    drive-scsi0: transferred: 1093664768 bytes remaining: 33266073600 bytes total: 34359738368 bytes progression: 3.18 % busy: 1 ready: 0
    drive-scsi0: transferred: 1105199104 bytes remaining: 33254539264 bytes total: 34359738368 bytes progression: 3.22 % busy: 1 ready: 0
    drive-scsi0: transferred: 1113587712 bytes remaining: 33246150656 bytes total: 34359738368 bytes progression: 3.24 % busy: 1 ready: 0
    drive-scsi0: transferred: 1124073472 bytes remaining: 33235664896 bytes total: 34359738368 bytes progression: 3.27 % busy: 1 ready: 0
    drive-scsi0: transferred: 1146093568 bytes remaining: 33213644800 bytes total: 34359738368 bytes progression: 3.34 % busy: 1 ready: 0
    drive-scsi0: transferred: 1163919360 bytes remaining: 33195819008 bytes total: 34359738368 bytes progression: 3.39 % busy: 1 ready: 0
    drive-scsi0: transferred: 1175453696 bytes remaining: 33184284672 bytes total: 34359738368 bytes progression: 3.42 % busy: 1 ready: 0
    drive-scsi0: transferred: 1189085184 bytes remaining: 33170653184 bytes total: 34359738368 bytes progression: 3.46 % busy: 1 ready: 0
    drive-scsi0: transferred: 1198522368 bytes remaining: 33161216000 bytes total: 34359738368 bytes progression: 3.49 % busy: 1 ready: 0
    drive-scsi0: transferred: 1217396736 bytes remaining: 33142341632 bytes total: 34359738368 bytes progression: 3.54 % busy: 1 ready: 0
    drive-scsi0: transferred: 1228931072 bytes remaining: 33130807296 bytes total: 34359738368 bytes progression: 3.58 % busy: 1 ready: 0
    drive-scsi0: transferred: 1247805440 bytes remaining: 33111932928 bytes total: 34359738368 bytes progression: 3.63 % busy: 1 ready: 0
    drive-scsi0: transferred: 1262485504 bytes remaining: 33097252864 bytes total: 34359738368 bytes progression: 3.67 % busy: 1 ready: 0
    drive-scsi0: transferred: 1271922688 bytes remaining: 33087815680 bytes total: 34359738368 bytes progression: 3.70 % busy: 1 ready: 0
    drive-scsi0: transferred: 1284505600 bytes remaining: 33075232768 bytes total: 34359738368 bytes progression: 3.74 % busy: 1 ready: 0
    drive-scsi0: transferred: 1299185664 bytes remaining: 33060552704 bytes total: 34359738368 bytes progression: 3.78 % busy: 1 ready: 0
    drive-scsi0: transferred: 1317011456 bytes remaining: 33042726912 bytes total: 34359738368 bytes progression: 3.83 % busy: 1 ready: 0
    drive-scsi0: transferred: 1323302912 bytes remaining: 33036435456 bytes total: 34359738368 bytes progression: 3.85 % busy: 1 ready: 0
    drive-scsi0: transferred: 1341128704 bytes remaining: 33018609664 bytes total: 34359738368 bytes progression: 3.90 % busy: 1 ready: 0
    drive-scsi0: transferred: 1388314624 bytes remaining: 32971423744 bytes total: 34359738368 bytes progression: 4.04 % busy: 1 ready: 0
    drive-scsi0: transferred: 1479540736 bytes remaining: 32880197632 bytes total: 34359738368 bytes progression: 4.31 % busy: 1 ready: 0
    drive-scsi0: transferred: 1485832192 bytes remaining: 32873906176 bytes total: 34359738368 bytes progression: 4.32 % busy: 1 ready: 0
    drive-scsi0: transferred: 1485832192 bytes remaining: 32873906176 bytes total: 34359738368 bytes progression: 4.32 % busy: 1 ready: 0
    drive-scsi0: transferred: 1485832192 bytes remaining: 32873906176 bytes total: 34359738368 bytes progression: 4.32 % busy: 1 ready: 0
    drive-scsi0: transferred: 1485832192 bytes remaining: 32873906176 bytes total: 34359738368 bytes progression: 4.32 % busy: 1 ready: 0
    drive-scsi0: transferred: 1485832192 bytes remaining: 32873906176 bytes total: 34359738368 bytes progression: 4.32 % busy: 1 ready: 0
    drive-scsi0: transferred: 1485832192 bytes remaining: 32873906176 bytes total: 34359738368 bytes progression: 4.32 % busy: 1 ready: 0
    drive-scsi0: transferred: 1485832192 bytes remaining: 32873906176 bytes total: 34359738368 bytes progression: 4.32 % busy: 1 ready: 0
    drive-scsi0: transferred: 1485832192 bytes remaining: 32873906176 bytes total: 34359738368 bytes progression: 4.32 % busy: 1 ready: 0
    drive-scsi0: transferred: 1485832192 bytes remaining: 32873906176 bytes total: 34359738368 bytes progression: 4.32 % busy: 1 ready: 0
    drive-scsi0: transferred: 1485832192 bytes remaining: 32873906176 bytes total: 34359738368 bytes progression: 4.32 % busy: 1 ready: 0
    drive-scsi0: transferred: 1485832192 bytes remaining: 32873906176 bytes total: 34359738368 bytes progression: 4.32 % busy: 1 ready: 0
    drive-scsi0: transferred: 1485832192 bytes remaining: 32873906176 bytes total: 34359738368 bytes progression: 4.32 % busy: 1 ready: 0
    drive-scsi0: transferred: 1485832192 bytes remaining: 32873906176 bytes total: 34359738368 bytes progression: 4.32 % busy: 1 ready: 0
    drive-scsi0: transferred: 1485832192 bytes remaining: 32873906176 bytes total: 34359738368 bytes progression: 4.32 % busy: 1 ready: 0
    drive-scsi0: transferred: 1485832192 bytes remaining: 32873906176 bytes total: 34359738368 bytes progression: 4.32 % busy: 1 ready: 0
    drive-scsi0: transferred: 1485832192 bytes remaining: 32873906176 bytes total: 34359738368 bytes progression: 4.32 % busy: 1 ready: 0
    drive-scsi0: transferred: 1485832192 bytes remaining: 32873906176 bytes total: 34359738368 bytes progression: 4.32 % busy: 1 ready: 0
    drive-scsi0: transferred: 1485832192 bytes remaining: 32873906176 bytes total: 34359738368 bytes progression: 4.32 % busy: 1 ready: 0
    drive-scsi0: transferred: 1485832192 bytes remaining: 32873906176 bytes total: 34359738368 bytes progression: 4.32 % busy: 1 ready: 0
    drive-scsi0: transferred: 1485832192 bytes remaining: 32873906176 bytes total: 34359738368 bytes progression: 4.32 % busy: 1 ready: 0
    drive-scsi0: transferred: 1485832192 bytes remaining: 32873906176 bytes total: 34359738368 bytes progression: 4.32 % busy: 1 ready: 0
    drive-scsi0: transferred: 1485832192 bytes remaining: 32873906176 bytes total: 34359738368 bytes progression: 4.32 % busy: 1 ready: 0
    drive-scsi0: transferred: 1485832192 bytes remaining: 32873906176 bytes total: 34359738368 bytes progression: 4.32 % busy: 1 ready: 0
    drive-scsi0: transferred: 1485832192 bytes remaining: 32873906176 bytes total: 34359738368 bytes progression: 4.32 % busy: 1 ready: 0
    drive-scsi0: transferred: 1485832192 bytes remaining: 32873906176 bytes total: 34359738368 bytes progression: 4.32 % busy: 1 ready: 0
    drive-scsi0: transferred: 1485832192 bytes remaining: 32873906176 bytes total: 34359738368 bytes progression: 4.32 % busy: 1 ready: 0
    drive-scsi0: transferred: 1485832192 bytes remaining: 32873906176 bytes total: 34359738368 bytes progression: 4.32 % busy: 1 ready: 0
    drive-scsi0: transferred: 1485832192 bytes remaining: 32873906176 bytes total: 34359738368 bytes progression: 4.32 % busy: 1 ready: 0
    drive-scsi0: transferred: 1485832192 bytes remaining: 32873906176 bytes total: 34359738368 bytes progression: 4.32 % busy: 1 ready: 0
    drive-scsi0: Cancelling block job
    2017-08-26 21:33:52 ERROR: online migrate failure - mirroring error: VM 100 not running
    2017-08-26 21:33:52 aborting phase 2 - cleanup resources
    2017-08-26 21:33:52 migrate_cancel
    2017-08-26 21:33:52 migrate_cancel error: VM 100 not running
    drive-scsi0: Cancelling block job
    2017-08-26 21:33:52 ERROR: VM 100 not running
    2017-08-26 21:34:09 ERROR: command '/usr/bin/ssh -o 'BatchMode=yes' -o 'HostKeyAlias=prox02' root@10.1.1.20 pvesm free stg-zfs:vm-100-disk-1' failed: exit code 4
    2017-08-26 21:34:12 ERROR: migration finished with problems (duration 00:02:15)
    migration problems

    I have ran the first syn replication and finish with success!
    I do not have any sync schedule between the servers.

    What it's wrong??
     
  2. Gilberto Ferreira

    Joined:
    Aug 21, 2014
    Messages:
    47
    Likes Received:
    0
    UPDATE!
    When I try to migrate the offline VM to PROX01 to PROX02, now I get this error:

    qm migrate 100 prox02 --with-local-disks
    2017-08-26 21:41:30 starting migration of VM 100 to node 'prox02' (10.1.1.20)
    2017-08-26 21:41:30 found local disk 'stg-zfs:vm-100-disk-1' (in current VM config)
    2017-08-26 21:41:30 copying disk images
    send from @ to ZFS/vm-100-disk-1@__migration__ estimated size is 6.89G
    total estimated size is 6.89G
    TIME SENT SNAPSHOT
    ZFS/vm-100-disk-1 name ZFS/vm-100-disk-1 -
    volume 'ZFS/vm-100-disk-1' already exists
    command 'zfs send -Rpv -- ZFS/vm-100-disk-1@__migration__' failed: got signal 13
    send/receive failed, cleaning up snapshot(s)..
    2017-08-26 21:41:32 ERROR: Failed to sync data - command 'set -o pipefail && pvesm export stg-zfs:vm-100-disk-1 zfs - -with-snapshots 0 -snapshot __migration__ | /usr/bin/ssh -o 'BatchMode=yes' -o 'HostKeyAlias=prox02' root@10.1.1.20 -- pvesm import stg-zfs:vm-100-disk-1 zfs - -with-snapshots 0 -delete-snapshot __migration__' failed: exit code 255
    2017-08-26 21:41:32 aborting phase 1 - cleanup resources
    2017-08-26 21:41:32 ERROR: found stale volume copy 'stg-zfs:vm-100-disk-1' on node 'prox02'
    2017-08-26 21:41:32 ERROR: migration aborted (duration 00:00:03): Failed to sync data - command 'set -o pipefail && pvesm export stg-zfs:vm-100-disk-1 zfs - -with-snapshots 0 -snapshot __migration__ | /usr/bin/ssh -o 'BatchMode=yes' -o 'HostKeyAlias=prox02' root@10.1.1.20 -- pvesm import stg-zfs:vm-100-disk-1 zfs - -with-snapshots 0 -delete-snapshot __migration__' failed: exit code 255
    migration aborted
     
  3. Gilberto Ferreira

    Joined:
    Aug 21, 2014
    Messages:
    47
    Likes Received:
    0
    UPDATE!
    When try to migrate online, I receive the error above, and in target host, I get this in /var/log/syslog:

    Aug 26 22:15:53 prox01 kernel: [12036.840122] sd 2:0:0:1: [sdb] tag#0 abort
     
  4. wolfgang

    wolfgang Proxmox Staff Member
    Staff Member

    Joined:
    Oct 1, 2014
    Messages:
    4,763
    Likes Received:
    315
    Live storage migration is not working with storage replication.
    So if you use storage replication you can only migrate guests offline.
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  1. This site uses cookies to help personalise content, tailor your experience and to keep you logged in if you register.
    By continuing to use this site, you are consenting to our use of cookies.
    Dismiss Notice