How to migrate VM from one PVE cluster to another

This is an awesome feature, thanks y'all! Is there a way to get it to work with ceph? Currently trying to migrate with Ceph disks hits "ERROR: no export formats for 'cephtwo:vm-1046-disk-0' - check storage plugin support!"
 
not at the moment, but we will extend the volume handling code to allow moving a volume from and to RBD.
 
Ironically it works for live migration, just not for offline migration. In any case, just managed to move a full cluster over a frustratingly slow WAN link, awesome feature! Took a week, but there were zero hiccups outside of having to comment out the checks that prevent migration if the VM was based on a snapshot of an image (which presumably just meant more disk usage on the target cluster, but otherwise appears to have left disks totally fine on the other side).
 
BlueMatt said:
Ironically it works for live migration, just not for offline migration. In any case, just managed to move a full cluster over a frustratingly slow WAN link, awesome feature! Took a week, but there were zero hiccups outside of having to comment out the checks that prevent migration if the VM was based on a snapshot of an image (which presumably just meant more disk usage on the target cluster, but otherwise appears to have left disks totally fine on the other side).
Click to expand...

great to hear :) RBD (offline) support is definitely on the todo list, and (including) snapshots is also up on the table since we want to take a stab at snapshot-based RBD replication at some point as well.
 
  • Like
Reactions: hellfire
the cross cluster ("remote") migration feature exists, but it is still marked as experimental.
 
fabian said:
the cross cluster ("remote") migration feature exists, but it is still marked as experimental.
Click to expand...
Thanks for the update. I am using proxmox in my homelab to get away from some non-genuine vmware keys and loving pve so far. I am also at an org the is very close to going with proxmox or xen and has 2 large data centers. The only concerns they had were native support for iscsi and the single pane of glass multi-cluster management. Which isn't really the need. The need was cross cluster migration. once that makes enterprise and can be tested, our org will likely make the jump over.
 
it works quite well for many cases already, but there are still some kinks to be ironed out :)
 
I've been runnig the command-line migration for a while now in various projects, seems to work really well, even migration between a cluster and single host (and vice versa) works. Hopefully we get a nice GUI for this at some point. :)
 
fabian said:
it's included as a preview/experimental feature, see the commands pct remote_migrate and qm remote_mgirate. you'll need an API token with the relevant privileges (the command will error out if your are missing them) and I would strongly suggest playing around with it in a test lab setting before letting it near a production environment - as I said, it's a preview/experimental feature and might still have bugs and rough edges.

among the things not yet supported are:
- snapshots (this requires some refactoring of our privilege checks, nothing else blocking it since we re-use the same storage migration code)
- pending changes (this requires some refactoring of our privilege checks, nothing else blocking it)
- replication (this one is a bit of a bigger feature, but definitely planned)
- non-dir based shared storages for offline/container migration (this one just lacks some implemented functions in the storage plugins and should be easiest of them all to implement)
Click to expand...
old thread, but fyi :
I have successfully used the 'qm remote_migrate' tool between prox7.4 clusters, but it seems to fail when migrating a vm form a 7.4 to a 8.4 cluster
I wouldn't be surprised if this is expected behavior (since different major release) but this might be interesting to someone
Using the backup-restore (using a shared storage) between 7.4 and 8.4 works as expected
 
thijz said:
old thread, but fyi :
I have successfully used the 'qm remote_migrate' tool between prox7.4 clusters, but it seems to fail when migrating a vm form a 7.4 to a 8.4 cluster
I wouldn't be surprised if this is expected behavior (since different major release) but this might be interesting to someone
Using the backup-restore (using a shared storage) between 7.4 and 8.4 works as expected
Click to expand...
could you post the exact versions on both ends and share the migration task log? we've recently backported a few compat fixes, maybe you were still missing those?
 
sure :
7.4-18 -> 8.4.1

This is the error :

# qm remote-migrate 101 1111 'apitoken=PVEAPIToken=root@pam!migrate=<token of target node>,host=10.X.Y.Z,fingerprint=<fingerprint>' --target-bridge vmbr1 --target-storage local-lvm
remote: storage 'local-lvm' does not exist!

I tried with other options and other storage locations but i keep getting the same 'storage does not exist' message


on the target host :

# pvesm status
Name Type Status Total Used Available %
FN02_NFS003 nfs active 3770391552 306011392 3464380160 8.12%
FN02_NFS004 nfs active 3770412672 2513310592 1257102080 66.66%
FN02_NFS006-RO nfs active 3770667520 388459904 3382207616 10.30%
TN05_NFS001 nfs active 1127268352 245147648 882120704 21.75%
local dir active 98497780 4148848 89299384 4.21%
local-lvm lvmthin active 448917504 0 448917504 0.00%
 
could you post "pveversion -v" from both sides? thanks!
 
Hi,
thijz said:
sure :
7.4-18 -> 8.4.1

This is the error :

# qm remote-migrate 101 1111 'apitoken=PVEAPIToken=root@pam!migrate=<token of target node>,host=10.X.Y.Z,fingerprint=<fingerprint>' --target-bridge vmbr1 --target-storage local-lvm
remote: storage 'local-lvm' does not exist!

I tried with other options and other storage locations but i keep getting the same 'storage does not exist' message


on the target host :

# pvesm status
Name Type Status Total Used Available %
FN02_NFS003 nfs active 3770391552 306011392 3464380160 8.12%
FN02_NFS004 nfs active 3770412672 2513310592 1257102080 66.66%
FN02_NFS006-RO nfs active 3770667520 388459904 3382207616 10.30%
TN05_NFS001 nfs active 1127268352 245147648 882120704 21.75%
local dir active 98497780 4148848 89299384 4.21%
local-lvm lvmthin active 448917504 0 448917504 0.00%
Click to expand...
does your token have permissions to access the storage?

EDIT: also, there should be upgrades available, 7.4-18 is not the latest PVE 7 version.
 
Last edited:
  • Like
Reactions: fabian
fabian said:
could you post "pveversion -v" from both sides? thanks!
Click to expand...
# pveversion -v
proxmox-ve: 7.4-1 (running kernel: 5.15.143-1-pve)
pve-manager: 7.4-17 (running version: 7.4-17/513c62be)
pve-kernel-5.15: 7.4-11
pve-kernel-5.15.143-1-pve: 5.15.143-1
pve-kernel-5.15.102-1-pve: 5.15.102-1
ceph-fuse: 15.2.17-pve1
corosync: 3.1.7-pve1
criu: 3.15-1+pve-1
glusterfs-client: 9.2-1
ifupdown2: 3.1.0-1+pmx4
ksm-control-daemon: 1.4-1
libjs-extjs: 7.0.0-1
libknet1: 1.24-pve2
libproxmox-acme-perl: 1.4.4
libproxmox-backup-qemu0: 1.3.1-1
libproxmox-rs-perl: 0.2.1
libpve-access-control: 7.4.3
libpve-apiclient-perl: 3.2-1
libpve-common-perl: 7.4-2
libpve-guest-common-perl: 4.2-4
libpve-http-server-perl: 4.2-3
libpve-rs-perl: 0.7.7
libpve-storage-perl: 7.4-3
libspice-server1: 0.14.3-2.1
lvm2: 2.03.11-2.1
lxc-pve: 5.0.2-2
lxcfs: 5.0.3-pve1
novnc-pve: 1.4.0-1
proxmox-backup-client: 2.4.6-1
proxmox-backup-file-restore: 2.4.6-1
proxmox-kernel-helper: 7.4-1
proxmox-mail-forward: 0.1.1-1
proxmox-mini-journalreader: 1.3-1
proxmox-widget-toolkit: 3.7.3
pve-cluster: 7.3-3
pve-container: 4.4-6
pve-docs: 7.4-2
pve-edk2-firmware: 3.20230228-4~bpo11+2
pve-firewall: 4.3-5
pve-firmware: 3.6-6
pve-ha-manager: 3.6.1
pve-i18n: 2.12-1
pve-qemu-kvm: 7.2.0-8
pve-xtermjs: 4.16.0-2
qemu-server: 7.4-4
smartmontools: 7.2-pve3
spiceterm: 3.2-2
swtpm: 0.8.0~bpo11+3
vncterm: 1.7-1
zfsutils-linux: 2.1.14-pve1


# pveversion -v
proxmox-ve: 8.4.0 (running kernel: 6.8.12-11-pve)
pve-manager: 8.4.1 (running version: 8.4.1/2a5fa54a8503f96d)
proxmox-kernel-helper: 8.1.1
proxmox-kernel-6.8.12-11-pve-signed: 6.8.12-11
proxmox-kernel-6.8: 6.8.12-11
proxmox-kernel-6.8.4-2-pve-signed: 6.8.4-2
ceph-fuse: 17.2.7-pve3
corosync: 3.1.9-pve1
criu: 3.17.1-2+deb12u1
glusterfs-client: 10.3-5
ifupdown2: 3.2.0-1+pmx11
ksm-control-daemon: 1.5-1
libjs-extjs: 7.0.0-5
libknet1: 1.30-pve2
libproxmox-acme-perl: 1.6.0
libproxmox-backup-qemu0: 1.5.1
libproxmox-rs-perl: 0.3.5
libpve-access-control: 8.2.2
libpve-apiclient-perl: 3.3.2
libpve-cluster-api-perl: 8.1.0
libpve-cluster-perl: 8.1.0
libpve-common-perl: 8.3.1
libpve-guest-common-perl: 5.2.2
libpve-http-server-perl: 5.2.2
libpve-network-perl: 0.11.2
libpve-rs-perl: 0.9.4
libpve-storage-perl: 8.3.6
libspice-server1: 0.15.1-1
lvm2: 2.03.16-2
lxc-pve: 6.0.0-1
lxcfs: 6.0.0-pve2
novnc-pve: 1.6.0-2
proxmox-backup-client: 3.4.1-1
proxmox-backup-file-restore: 3.4.1-1
proxmox-firewall: 0.7.1
proxmox-kernel-helper: 8.1.1
proxmox-mail-forward: 0.3.2
proxmox-mini-journalreader: 1.4.0
proxmox-offline-mirror-helper: 0.6.7
proxmox-widget-toolkit: 4.3.11
pve-cluster: 8.1.0
pve-container: 5.2.6
pve-docs: 8.4.0
pve-edk2-firmware: 4.2025.02-3
pve-esxi-import-tools: 0.7.4
pve-firewall: 5.1.1
pve-firmware: 3.15-4
pve-ha-manager: 4.0.7
pve-i18n: 3.4.4
pve-qemu-kvm: 9.2.0-5
pve-xtermjs: 5.5.0-2
qemu-server: 8.3.12
smartmontools: 7.3-pve1
spiceterm: 3.3.0
swtpm: 0.8.0+pve1
vncterm: 1.8.0
zfsutils-linux: 2.2.7-pve2
 
please upgrade to the last versions available on the source side (in particular qemu-server 7.4-7 and libpve-guest-common-perl 4.2-5)
 
You must log in or register to reply here.
Top Bottom