Framework for remote migration to cluster-external Proxmox VE hosts

[elter]

1) Is there a plan to add support for cloned disks? Most of my VMs are clones.

Trying to migrate a clone leads to this error:

2023-02-06 13:11:48 ERROR: Problem found while scanning volumes - can't migrate 'ceph-data:base-100-disk-0/vm-103-disk-0' as it's a clone of 'base-100-disk-0' at /usr/share/perl5/PVE/QemuMigrate.pm line 503, <GEN24> line 2.

2) Cloud-Init enabled VMs appear to be failing too:

2023-02-06 13:18:42 ERROR: migration aborted (duration 00:00:01): no export formats for 'ceph-data:vm-103-cloudinit' - check storage plugin support!

3) Custom CPU types not defined on a target lead to a failure:

failed - failed to handle 'config' command - vm 103 - unable to parse value of 'cpu' - Custom cputype 'Cascadelake-Server-noTSX-noCLWB' not found

(Side note, it's pretty exciting functionality)

 

[elter]

One issue that I'm seeing, after the migration the origin cluster is leaving an artifact behind. I've not looked at the files as of yet. Is there any data that would help determine the cause of this? I've not done any cleanup or anything like that on these in case data is requested.

It's actually documented here: https://pve.proxmox.com/pve-docs/qm.1.html , see the "delete" parameter. For "artifacts", you can qm unlock <VMID> and then delete or do whatever you need to do.

 

[Darkk]

This qm remote-migrate very cool to move VMs between clusters but what if I just want it to sync the VMs as cold standby? Something like qm remote-sync where it'll sync via ZFS snapshots? This way I can have a working copy as cold standby on the remote cluster in case the primary cluster fails.

I am looking at PVE-zsync but am still using ZFS replication between the nodes on the same cluster so didn't want to break those if I have to migrate the VMs to another node which is a limitation of PVE-zsync. I love it when I mirgrate the VM to another node and replication automatically flips it to the other direction.

 

[Kosh]

Hi

help find the error

qm remote-migrate 172 172 --target-endpoint'10.20.87.201,apitoken=PVEAPIToken=root@pam=monitoring!296f2fee-0e39-4e14-bbb4-6f5b417aff18' --target-bridge=vmbr0 --target-storage local-lvm5 --online

what should the team look like in the end?


I have

Spoiler

 

[t.lamprecht]

help find the error

What's the output, i.e., error message you get when executing the command..?

 

[Kosh]

What's the output, i.e., error message you get when executing the command..?

thanks i figured it out

correct command in my case

qm remote-migrate 172 172  'apitoken=PVEAPIToken=root@pam!monitoring=296f2fee-0e39-4e14-bbb4-6f5b417aff18,host=10.20.87.201,fingerprint=E1:D8:2D:88:00:2C:CE:62:46:92:DA:56:F3:CC:C5:14:C8:02:2B:8F:05:5F:69:CF:D2:73:77:01:06:8C:BE:A6' --target-bridge vmbr0 --target-storage local-lvm5 --online

it was not clear with fingerprint it was necessary to execute the command without this parameter in order to see the correct fingerprint and then insert it into the original command

 

[fettfoen]

Really good features and works quite well. It would be nice to define the 'migration network.' In my case, I have two separate clusters. However, the migration only works through the mgmt interface. In my old cluster, the management interface is only 1GB. But both clusters have the same 10GB storage network. Unfortunately, the bottleneck for my qm remote-migrate is that the migration network operates by default only through the management interface and cannot be defined. The directive --migration_network didnt work in my case.

 

[Jean-Paul van Hamond]

The only problem I have now is that I think it is impossible to give a vlan tag to a vlan-aware bridge.

I can migrate a VM to vmbr0 but not provide the necessary vlan tag. Or am I missing something?

 

[croso]

the qm-remote migrate command does not function clearly as defined, or could be more properly defined (I shouldnt have to enter an API token to try this command, maybe prompt for password on remote system instead?)

 

[t.lamprecht]

the qm-remote migrate command does not function clearly as defined, or could be more properly defined (I shouldnt have to enter an API token to try this command, maybe prompt for password on remote system instead?)

In general, it functions as defined as user feedback shows here, or what specific problem do you have? Seeing the actual command invocation (with token/IPs censored minimally) and error messages would better allow helping you.

This command is designed as an internal command that will be used by higher level tooling, that's also why it targets the use of API tokens. We exposed it now as it can be already useful for more experienced admins, and it can also be scripted to fit your needs in a more convenient way.

 

[croso]

is there a token free way to do this? I cannot seem to find the right syntax

 

[static302]

1) Is there a plan to add support for cloned disks? Most of my VMs are clones.

Trying to migrate a clone leads to this error:

2023-02-06 13:11:48 ERROR: Problem found while scanning volumes - can't migrate 'ceph-data:base-100-disk-0/vm-103-disk-0' as it's a clone of 'base-100-disk-0' at /usr/share/perl5/PVE/QemuMigrate.pm line 503, <GEN24> line 2.

2) Cloud-Init enabled VMs appear to be failing too:

2023-02-06 13:18:42 ERROR: migration aborted (duration 00:00:01): no export formats for 'ceph-data:vm-103-cloudinit' - check storage plugin support!

3) Custom CPU types not defined on a target lead to a failure:

failed - failed to handle 'config' command - vm 103 - unable to parse value of 'cpu' - Custom cputype 'Cascadelake-Server-noTSX-noCLWB' not found

(Side note, it's pretty exciting functionality)

Hello,
Sorry for necroposting, but have you found a solution for 2nd problem with cloud-init?
Thanks.

Edit: Found this thread: https://forum.proxmox.com/threads/remote-migration-cloudinit.148674. It's unfortunate that this issue makes the remote migration feature almost unusable for me atm. I’m really hoping for improvements soon. Thanks.