HA live migration with zfs replication on pve 6.2

M

mbosma

Renowned Member
Dec 3, 2018
124
25
68
30
First of all I'd like to thank the devs for another great update. pve 6.2 has brought some really nice features.

During my testing with 6.2 I was pleased to see that live migration with zfs replication works now, this is a feature I've been waiting for.

I use HA with zfs replication as failover but live migration has always been a bit of a hassle.
HA had to be disabled on a vm as wel as the zfs replication and had to be enabled afterwards.

Now live migration will work with zfs replication but only after disabling HA.
The migration will fail on HA because the HA resource agent doesn't add the "--with-local-disks" flag.

Task viewer: HA 102 - Migrate
Requesting HA migration for VM 102 to node pve2
TASK OK
Click to expand...

Task viewer: VM 102 - Migrate
task started by HA resource agent
2020-05-13 11:38:06 use dedicated network address for sending migration traffic (192.168.100.232)
2020-05-13 11:38:07 starting migration of VM 102 to node 'pve2' (192.168.100.232)
2020-05-13 11:38:07 found local, replicated disk 'local-zfs:vm-102-disk-0' (in current VM config)
2020-05-13 11:38:07 can't migrate local disk 'local-zfs:vm-102-disk-0': can't live migrate attached local disks without with-local-disks option
2020-05-13 11:38:07 ERROR: Failed to sync data - can't migrate VM - check log
2020-05-13 11:38:07 aborting phase 1 - cleanup resources
2020-05-13 11:38:07 ERROR: migration aborted (duration 00:00:01): Failed to sync data - can't migrate VM - check log
TASK ERROR: migration aborted
Click to expand...

Here's the log for a migration using zfs replication with HA disabled.

Task viewer: VM 102 - Migrate
2020-05-12 22:45:16 use dedicated network address for sending migration traffic (192.168.100.233)
2020-05-12 22:45:16 starting migration of VM 102 to node 'pve3' (192.168.100.233)
2020-05-12 22:45:16 found local, replicated disk 'local-zfs:vm-102-disk-0' (in current VM config)
2020-05-12 22:45:16 scsi0: start tracking writes using block-dirty-bitmap 'repl_scsi0'
2020-05-12 22:45:16 replicating disk images
2020-05-12 22:45:16 start replication job
2020-05-12 22:45:16 guest => VM 102, running => 76265
2020-05-12 22:45:16 volumes => local-zfs:vm-102-disk-0
2020-05-12 22:45:17 create snapshot '__replicate_102-0_1589316316__' on local-zfs:vm-102-disk-0
2020-05-12 22:45:17 using secure transmission, rate limit: none
2020-05-12 22:45:17 incremental sync 'local-zfs:vm-102-disk-0' (__replicate_102-0_1589316300__ => __replicate_102-0_1589316316__)
2020-05-12 22:45:18 send from @__replicate_102-0_1589316300__ to rpool/data/vm-102-disk-0@__replicate_102-0_1589316316__ estimated size is 70.0K
2020-05-12 22:45:18 total estimated size is 70.0K
2020-05-12 22:45:18 TIME SENT SNAPSHOT rpool/data/vm-102-disk-0@__replicate_102-0_1589316316__
2020-05-12 22:45:18 rpool/data/vm-102-disk-0@__replicate_102-0_1589316300__ name rpool/data/vm-102-disk-0@__replicate_102-0_1589316300__ -
2020-05-12 22:45:18 successfully imported 'local-zfs:vm-102-disk-0'
2020-05-12 22:45:18 delete previous replication snapshot '__replicate_102-0_1589316300__' on local-zfs:vm-102-disk-0
2020-05-12 22:45:19 (remote_finalize_local_job) delete stale replication snapshot '__replicate_102-0_1589316300__' on local-zfs:vm-102-disk-0
2020-05-12 22:45:19 end replication job
2020-05-12 22:45:19 copying local disk images
2020-05-12 22:45:19 starting VM 102 on remote node 'pve3'
2020-05-12 22:45:20 start remote tunnel
2020-05-12 22:45:21 ssh tunnel ver 1
2020-05-12 22:45:21 starting storage migration
2020-05-12 22:45:21 scsi0: start migration to nbd:unix:/run/qemu-server/102_nbd.migrate:exportname=drive-scsi0
drive mirror re-using dirty bitmap 'repl_scsi0'
drive mirror is starting for drive-scsi0
all mirroring jobs are ready
2020-05-12 22:45:21 volume 'local-zfs:vm-102-disk-0' is 'local-zfs:vm-102-disk-0' on the target
2020-05-12 22:45:21 starting online/live migration on unix:/run/qemu-server/102.migrate
2020-05-12 22:45:21 set migration_caps
2020-05-12 22:45:21 migration speed limit: 8589934592 B/s
2020-05-12 22:45:21 migration downtime limit: 100 ms
2020-05-12 22:45:21 migration cachesize: 268435456 B
2020-05-12 22:45:21 set migration parameters
2020-05-12 22:45:21 start migrate command to unix:/run/qemu-server/102.migrate
2020-05-12 22:45:22 migration status: active (transferred 428878266, remaining 42119168), total 2165121024)
2020-05-12 22:45:22 migration xbzrle cachesize: 268435456 transferred 0 pages 0 cachemiss 0 overflow 0
2020-05-12 22:45:23 migration speed: 1024.00 MB/s - downtime 57 ms
2020-05-12 22:45:23 migration status: completed
all mirroring jobs are ready
drive-scsi0: Completing block job...
drive-scsi0: Completed successfully.
drive-scsi0 : finished
2020-05-12 22:45:24 # /usr/bin/ssh -e none -o 'BatchMode=yes' -o 'HostKeyAlias=pve3' root@192.168.100.233 pvesr set-state 102 \''{"local/pve2":{"last_try":1589316316,"fail_count":0,"last_sync":1589316316,"duration":2.748475,"storeid_list":["local-zfs"],"last_node":"pve2","last_iteration":1589316316}}'\'
2020-05-12 22:45:25 stopping NBD storage migration server on target.
2020-05-12 22:45:28 migration finished successfully (duration 00:00:12)
TASK OK
Click to expand...

Would it be possible to add this flag to the migration command used by the HA resource agent?

Using live migration on a ceph backed vm works flawlessly, here is the log for that migration as reference.

Task viewer: HA 100 - Migrate
Requesting HA migration for VM 100 to node pve2
TASK OK
Click to expand...

Task viewer: VM 100 - Migrate
task started by HA resource agent
2020-05-13 11:46:26 use dedicated network address for sending migration traffic (192.168.100.232)
2020-05-13 11:46:26 starting migration of VM 100 to node 'pve2' (192.168.100.232)
2020-05-13 11:46:27 starting VM 100 on remote node 'pve2'
2020-05-13 11:46:28 start remote tunnel
2020-05-13 11:46:29 ssh tunnel ver 1
2020-05-13 11:46:29 starting online/live migration on unix:/run/qemu-server/100.migrate
2020-05-13 11:46:29 set migration_caps
2020-05-13 11:46:29 migration speed limit: 8589934592 B/s
2020-05-13 11:46:29 migration downtime limit: 100 ms
2020-05-13 11:46:29 migration cachesize: 268435456 B
2020-05-13 11:46:29 set migration parameters
2020-05-13 11:46:29 start migrate command to unix:/run/qemu-server/100.migrate
2020-05-13 11:46:30 migration speed: 2048.00 MB/s - downtime 58 ms
2020-05-13 11:46:30 migration status: completed
2020-05-13 11:46:33 migration finished successfully (duration 00:00:07)
TASK OK
Click to expand...
 
I just got around to reading the announcement comments and saw this was already talked about.
Thanks!
 
I patched 2 clusters myself for testing and it works great so far.
Thanks devs!!
 
I applied the patch this way:
Code: 
vim /usr/share/perl5/PVE/HA/Resources/PVEVM.pm

Enter ":108" to get to the correct line
Press "i" to insert and paste the new line in.
Make it look like this:

Code: 
    my $params = {
        node => $nodename,
        vmid => $id,
        # bug #2241 forces is for local resource only, people can ensure that
        # different host have the same hardware, so this can be fine, and qemu
        # knows when not, so can only win here
        force => 1,
        with-local-disks => 1,
        target => $target,
        online => $online,
    };
 
mbosma said:
I applied the patch this way:
Code: 
vim /usr/share/perl5/PVE/HA/Resources/PVEVM.pm

Enter ":108" to get to the correct line
Press "i" to insert and paste the new line in.
Make it look like this:

Code: 
    my $params = {
        node => $nodename,
        vmid => $id,
        # bug #2241 forces is for local resource only, people can ensure that
        # different host have the same hardware, so this can be fine, and qemu
        # knows when not, so can only win here
        force => 1,
        with-local-disks => 1,
        target => $target,
        online => $online,
    };
Click to expand...

Thank you so much! In addiction, just just needed to use that parameter between apostrophes ('with-local-disks' => 1,), because at first try it crashed.

Anyway, now it works like a charm even in GUI... ;-)
 
Nice!

Can't wait for the proxmox devs to release this into the repo so I can upgrade production systems, this fix has been working like a charm on my dev machines for a while now.
 
  • Like
Reactions: edsenos
I do not think this has been fixed.

I am running PVE 8.2 and live migration still errors for those HA VMs with zfs replication



Code: 
task started by HA resource agent
2024-08-09 15:03:24 use dedicated network address for sending migration traffic (10.21.250.106)
2024-08-09 15:03:25 starting migration of VM 104 to node 'ozone-set00-j09-svr06' (10.21.250.106)
2024-08-09 15:03:25 found local, replicated disk 'hdd-2m-data:vm-104-disk-0' (attached)
2024-08-09 15:03:25 found generated disk 'local-zfs:vm-104-cloudinit' (in current VM config)
2024-08-09 15:03:25 virtio0: start tracking writes using block-dirty-bitmap 'repl_virtio0'
2024-08-09 15:03:25 replicating disk images
2024-08-09 15:03:25 start replication job
QEMU Guest Agent is not running - VM 104 qmp command 'guest-ping' failed - got timeout
2024-08-09 15:03:28 guest => VM 104, running => 1749892
2024-08-09 15:03:28 volumes => hdd-2m-data:vm-104-disk-0
2024-08-09 15:03:30 create snapshot '__replicate_104-0_1723183405__' on hdd-2m-data:vm-104-disk-0
2024-08-09 15:03:30 using secure transmission, rate limit: none
2024-08-09 15:03:30 incremental sync 'hdd-2m-data:vm-104-disk-0' (__replicate_104-0_1723183203__ => __replicate_104-0_1723183405__)
2024-08-09 15:03:32 send from @__replicate_104-0_1723183203__ to hdd-2m-data/vm-104-disk-0@__replicate_104-0_1723183405__ estimated size is 170K
2024-08-09 15:03:32 total estimated size is 170K
2024-08-09 15:03:32 TIME        SENT   SNAPSHOT hdd-2m-data/vm-104-disk-0@__replicate_104-0_1723183405__
2024-08-09 15:03:33 successfully imported 'hdd-2m-data:vm-104-disk-0'
2024-08-09 15:03:33 delete previous replication snapshot '__replicate_104-0_1723183203__' on hdd-2m-data:vm-104-disk-0
2024-08-09 15:03:34 (remote_finalize_local_job) delete stale replication snapshot '__replicate_104-0_1723183203__' on hdd-2m-data:vm-104-disk-0
2024-08-09 15:03:35 end replication job
2024-08-09 15:03:35 copying local disk images
2024-08-09 15:03:36 full send of rpool/data/vm-104-cloudinit@__migration__ estimated size is 65.3K
2024-08-09 15:03:36 total estimated size is 65.3K
2024-08-09 15:03:36 TIME        SENT   SNAPSHOT rpool/data/vm-104-cloudinit@__migration__
2024-08-09 15:03:36 successfully imported 'local-zfs:vm-104-cloudinit'
2024-08-09 15:03:36 volume 'local-zfs:vm-104-cloudinit' is 'local-zfs:vm-104-cloudinit' on the target
2024-08-09 15:03:36 starting VM 104 on remote node 'ozone-set00-j09-svr06'
2024-08-09 15:03:39 volume 'hdd-2m-data:vm-104-disk-0' is 'hdd-2m-data:vm-104-disk-0' on the target
2024-08-09 15:03:39 start remote tunnel
2024-08-09 15:03:40 ssh tunnel ver 1
2024-08-09 15:03:40 starting storage migration
2024-08-09 15:03:40 virtio0: start migration to nbd:unix:/run/qemu-server/104_nbd.migrate:exportname=drive-virtio0
drive mirror re-using dirty bitmap 'repl_virtio0'
drive mirror is starting for drive-virtio0
channel 3: open failed: connect failed: open failed
drive-virtio0: Cancelling block job
drive-virtio0: Done.
2024-08-09 15:03:40 ERROR: online migrate failure - mirroring error: VM 104 qmp command 'drive-mirror' failed - Failed to read initial magic: Unexpected end-of-file before all data were read
2024-08-09 15:03:40 aborting phase 2 - cleanup resources
2024-08-09 15:03:40 migrate_cancel


Here is my output of pveversion -v

Code: 
root@ozone-set00-j09-svr06:~/.ssh# pveversion -v
proxmox-ve: 8.2.0 (running kernel: 6.8.4-2-pve)
pve-manager: 8.2.2 (running version: 8.2.2/9355359cd7afbae4)
proxmox-kernel-helper: 8.1.0
proxmox-kernel-6.8: 6.8.4-2
proxmox-kernel-6.8.4-2-pve-signed: 6.8.4-2
ceph-fuse: 17.2.7-pve3
corosync: 3.1.7-pve3
criu: 3.17.1-2
glusterfs-client: 10.3-5
ifupdown2: 3.2.0-1+pmx8
ksm-control-daemon: 1.5-1
libjs-extjs: 7.0.0-4
libknet1: 1.28-pve1
libproxmox-acme-perl: 1.5.0
libproxmox-backup-qemu0: 1.4.1
libproxmox-rs-perl: 0.3.3
libpve-access-control: 8.1.4
libpve-apiclient-perl: 3.3.2
libpve-cluster-api-perl: 8.0.6
libpve-cluster-perl: 8.0.6
libpve-common-perl: 8.2.1
libpve-guest-common-perl: 5.1.1
libpve-http-server-perl: 5.1.0
libpve-network-perl: 0.9.8
libpve-rs-perl: 0.8.8
libpve-storage-perl: 8.2.1
libspice-server1: 0.15.1-1
lvm2: 2.03.16-2
lxc-pve: 6.0.0-1
lxcfs: 6.0.0-pve2
novnc-pve: 1.4.0-3
proxmox-backup-client: 3.2.0-1
proxmox-backup-file-restore: 3.2.0-1
proxmox-kernel-helper: 8.1.0
proxmox-mail-forward: 0.2.3
proxmox-mini-journalreader: 1.4.0
proxmox-offline-mirror-helper: 0.6.6
proxmox-widget-toolkit: 4.2.1
pve-cluster: 8.0.6
pve-container: 5.0.10
pve-docs: 8.2.1
pve-edk2-firmware: 4.2023.08-4
pve-esxi-import-tools: 0.7.0
pve-firewall: 5.0.5
pve-firmware: 3.11-1
pve-ha-manager: 4.0.4
pve-i18n: 3.2.2
pve-qemu-kvm: 8.1.5-5
pve-xtermjs: 5.3.0-3
qemu-server: 8.2.1
smartmontools: 7.3-pve1
spiceterm: 3.3.0
swtpm: 0.8.0+pve1
vncterm: 1.8.0
zfsutils-linux: 2.2.3-pve2
root@ozone-set00-j09-svr06:~/.ssh# pveversion -v
 
that is a different problem:

Code: 
channel 3: open failed: connect failed: open failed
 
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom
Back