[SOLVED] Proxmox Migration Non-Cluster

C

cshill

Member
May 8, 2024
59
8
8
Hi Everyone,
I have learned to migrate just fine within clusters and moving VM's via HA but this question is in regards to migration to other Proxmox hosts without a cluster. A straight P2P method. I suggested using the backup option to a Proxmox Backup server then restoring it but that just doubles the time for transfer with a middle man however much cleaner.

I also have tried SCP from one proxmox to another proxmox but the challenge is that the config file would need to be manually edited. Example, I make a VM on data3 of server1. Server 2 does not have a data3 storage but has a data2. I make a directory and everything on data2 and the files transfer but the config file of the VMs says they should be on data3. Server2 also doesn't initially recognize the disks and if you qm -rescan the disks show up as hard disks even though one is a hard disk, one is a TPM, and one is an EFI. This breaks the natural boot and instead I have to go to bios and change the boot order.

The problem with the method above is the amount of "in the shell" work.

In addition, does anyone know of a clean method to upload random different disk images to Proxmox and for it to format effectively?
 
Hi @cshill , have tried the command below?

man qm
Code: 
       qm remote-migrate <vmid> [<target-vmid>] <target-endpoint> --target-bridge <string> --target-storage <string> [OPTIONS]
       Migrate virtual machine to a remote cluster. Creates a new migration task. EXPERIMENTAL feature!

       <vmid>: <integer> (100 - 999999999)
           The (unique) ID of the VM.

       <target-vmid>: <integer> (100 - 999999999)
           The (unique) ID of the VM.

       <target-endpoint>: apitoken=<PVEAPIToken=user@realm!token=SECRET> ,host=<ADDRESS> [,fingerprint=<FINGERPRINT>] [,port=<PORT>]
           Remote target endpoint

       --bwlimit <integer> (0 - N) (default = migrate limit from datacenter or storage config)
           Override I/O bandwidth limit (in KiB/s).

       --delete <boolean> (default = 0)
           Delete the original VM and related data after successful migration. By default the original VM is kept on the source cluster in a stopped state.

       --online <boolean>
           Use online/live migration if VM is running. Ignored if VM is stopped.

       --target-bridge <string>
           Mapping from source to target bridges. Providing only a single bridge ID maps all source bridges to that bridge. Providing the special value 1 will map each source bridge to itself.

       --target-storage <string>
           Mapping from source to target storages. Providing only a single storage ID maps all source storages to that storage. Providing the special value 1 will map each source storage to itself.


Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox
 
bbgeek17 said:
Hi @cshill , have tried the command below?

man qm
Code: 
       qm remote-migrate <vmid> [<target-vmid>] <target-endpoint> --target-bridge <string> --target-storage <string> [OPTIONS]
       Migrate virtual machine to a remote cluster. Creates a new migration task. EXPERIMENTAL feature!

       <vmid>: <integer> (100 - 999999999)
           The (unique) ID of the VM.

       <target-vmid>: <integer> (100 - 999999999)
           The (unique) ID of the VM.

       <target-endpoint>: apitoken=<PVEAPIToken=user@realm!token=SECRET> ,host=<ADDRESS> [,fingerprint=<FINGERPRINT>] [,port=<PORT>]
           Remote target endpoint

       --bwlimit <integer> (0 - N) (default = migrate limit from datacenter or storage config)
           Override I/O bandwidth limit (in KiB/s).

       --delete <boolean> (default = 0)
           Delete the original VM and related data after successful migration. By default the original VM is kept on the source cluster in a stopped state.

       --online <boolean>
           Use online/live migration if VM is running. Ignored if VM is stopped.

       --target-bridge <string>
           Mapping from source to target bridges. Providing only a single bridge ID maps all source bridges to that bridge. Providing the special value 1 will map each source bridge to itself.

       --target-storage <string>
           Mapping from source to target storages. Providing only a single storage ID maps all source storages to that storage. Providing the special value 1 will map each source storage to itself.


Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox
Click to expand...
Hi BB I will give that a shot today.
 
I tried to use the following below to start the migration. Let me know what I did wrong as I found an older post and tried it their way and the person asking for help said it worked but did not for me.
-- Any place that says 12345 put in the correct values. The below is exactly how it would have been put in with the commas and the brackets.

qm remote-migrate 100 103 --target-endpoint 'apitoken=PVEAPIToken=root@pam!token=12345 ,host=192.168.XXX.XXX [,fingerprint=12345] [,port=443]'--target-bridge vmbr0 --target-storage DataStore2 --online
 
I edited it to the following and received a different error, this time I received : !token=a8ac8ebb: event not found

qm remote-migrate 100 [103] apitoken=PVEAPIToken=root@pam!token=12345 ,host=192.168.XXX.XXX [,fingerprint=12345] [,port=443] --target-bridge vmbr0 --target-storage DataStore2 --online

I'm not quite sure what I'm doing incorrectly here.
 
The [] and <> are part of Linux documentation convention.
The [] mean that the parameter is optional.
The [<>] means that if you do replace, it must be particular information.
The <> means its required and there is no particular format.

[<target-vmid>] means that you can provide target VM ID but don't have to. If you provide it, it must be in VMID format.

The <target-endpoint> is required. --target-bridge and --target-storage are required.
[fingerprint] and [port] are optional.

Additionally ! has a special meaning in Linux shell. You must quote strings containing the ! and many other characters, i.e. &

Good luck

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox
 
Working this through. Syntax is better but now the fingerprint is not verified. for the pve=ssl.pem on the 2nd server, the receiver of the VM.
I've tried looking on other posts but they are discussing what looks like the same issue but in regards to joining a node to a cluster.

Some people state I should run 'pvecm updatecerts --force' to swap out these certificates. I've tried that and rebooted servers, no change.
I've also removed the API token for the root users on both servers and then rebooted, then added the API back, no change. Tried new fingerprints with the new API tokens, no changes.

I'm just not sure how I verify these fingerprints.
 
I actually tried that after my last response. I first tried SCP to confirm connection and accepted the fingerprint.
 
Next time consider posting actual commands you are using, including output/error.
You verify token by specifying it on the command line

Code: 
 qm remote-migrate 999 9999 'apitoken=PVEAPIToken=root@pam!test=69398265-0806-4401-9503,host=172.16.202.202' --target-bridge vmbr0 --target-storage local-lvm
fingerprint 'A8:C4:B0:24:53:EB:0F:70:21:8F:45:1F:68:06:91:B4:C8:78:67:B0:5B:5C:67:22:AF:7C:3F:0B:AE:83:FB:DD' not verified, abort!

specifying token:
Code: 
qm remote-migrate 999 9999 'apitoken=PVEAPIToken=root@pam!test=69398265-0806-4401-9503-,host=172.16.202.202,fingerprint=A8:C4:B0:24:53:EB:0F:70:21:8F:45:1F:68:06:91:B4:C8:78:67:B0:5B:5C:67:22:AF:7C:3F:0B:AE:83:FB:DD' --target-bridge vmbr0 --target-storage local-lvm
remote: storage 'local-lvm' does not exist (or missing permission)!

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox
 
Code: 
Test1 to Test2 using Test1 apitoken and fingerprint
root@proxmoxtest1:~# qm remote-migrate 100 103 'apitoken=PVEAPIToken=root@pam!token=feed2f23-0d88-427f-8bd4-a1dec2c3c46b ,host=<IP> ,fingerprint=42:EE:CB:30:7C:B3:B7:3B:7C:2C:C6:BF:2B:95:B7:6B:EB6:E2:13:96:F4:9ED:7D:34:0F:38:1E:91:41:E7' --target-bridge vmbr0 --target-storage Datastore2
fingerprint 'DA:9C:20:8F:1A:2A:13:72:1797:64:51:19:53:25:F4:44:0B:21:E2:35:FB:3F:2B:64:36:6C:541:8A:5E' not verified, abort!
Test1 to Test2 using Test1 re-rolled fingerprint
root@proxmoxtest1:~# qm remote-migrate 100 103 'apitoken=PVEAPIToken=root@pam!token=feed2f23-0d88-427f-8bd4-a1dec2c3c46b ,host=<IP> ,fingerprint=9B:6D:7B:36:64:F4:5A:AC5:CC:64:72:18:52:7F:54:2F0:BF:977:37:97:EB:CD:E7:F9:71:CC:5D:78:6C' --target-bridge vmbr0 --target-storage Datastore2
fingerprint 'DA:9C:20:8F:1A:2A:13:72:1797:64:51:19:53:25:F4:44:0B:21:E2:35:FB:3F:2B:64:36:6C:541:8A:5E' not verified, abort!

Test1 to Test2 using Test2 apitoken and Fingerprint
root@proxmoxtest1:~# qm remote-migrate 100 103 'apitoken=PVEAPIToken=root@pam!token=f9ad8dbc-4f9f-407e-a13f-12321695f004 ,host=<IP> ,fingerprint=DA:9C:20:8F:1A:2A:13:72:1797:64:51:19:53:25:F4:44:0B:21:E2:35:FB:3F:2B:64:36:6C:541:8A:5E' --target-bridge vmbr0 --target-storage Datastore2
401 no such token 'token' for user 'root@pam'
Test1 to Test2 using Test2 apitoken and changed !token to !secret
root@proxmoxtest1:~# qm remote-migrate 100 103 'apitoken=PVEAPIToken=root@pam!secret=f9ad8dbc-4f9f-407e-a13f-12321695f004 ,host=<IP> ,fingerprint=DA:9C:20:8F:1A:2A:13:72:1797:64:51:19:53:25:F4:44:0B:21:E2:35:FB:3F:2B:64:36:6C:541:8A:5E' --target-bridge vmbr0 --target-storage Datastore2
401 no such token 'secret' for user 'root@pam'
Test1 to Test2 using Test2 apitoken and changed !token to !Secret - Capital Secret
root@proxmoxtest1:~# qm remote-migrate 100 103 'apitoken=PVEAPIToken=root@pam!Secret=f9ad8dbc-4f9f-407e-a13f-12321695f004 ,host=<IP>,fingerprint=<DA:9C:20:8F:1A:2A:13:72:1797:64:51:19:53:25:F4:44:0B:21:E2:35:FB:3F:2B:64:36:6C:541:8A:5E' --target-bridge vmbr0 --target-storage Datastore2>
401 no such token 'Secret' for user 'root@pam'


I even went through removing and trying a new token and SSH back and forth
 
Last edited:
Hi @cshill
a) use <> CODE tags from the menu of the edit box. This will prevent smiley faces replacement and present readable information
b) replacing a suffix of an IP, which is located within a reserved private network space, hardly helps you with any security posture
c) you have spaces " " in front of the commas "," that break your command line
d) in the first command you specify a fingerprint that ends with 41:E7, yet the remote server's fingerprint that you need to specify ends 8A:5E. As reported by the error message.

I did not read beyond the first two lines, as the output format is extremely distracting.

Cheers

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox
 
I fixed the code based on what you said. I also just didn't put in the IP here in the forum

Code: 
qm remote-migrate 100 103 'apitoken=PVEAPIToken=root@pam!token=f9ad8dbc-4f9f-407e-a13f-12321695f004,host=IP,fingerprint=DA:9C:20:8F:1A:2A:13:72:17:D9:D7:64:51:19:53:25:F4:44:0B:2
1:E2:35:FB:3F:2B:64:36:6C:54:D1:8A:5E' --target-bridge vmbr0 --target-storage Datastore2
401 no such token 'token' for user 'root@pam'
 
cshill said:
I fixed the code based on what you said.
Click to expand...
great to see progress.

cshill said:
Code:
Code: 
qm remote-migrate 100 103 'apitoken=PVEAPIToken=root@pam!token=f9ad8dbc-4f9f-407e-a13f-12321695f004,host=IP,fingerprint=DA:9C:20:8F:1A:2A:13:72:17:D9:D7:64:51:19:53:25:F4:44:0B:2
1:E2:35:FB:3F:2B:64:36:6C:54:D1:8A:5E' --target-bridge vmbr0 --target-storage Datastore2
401 no such token 'token' for user 'root@pam'
Click to expand...
The command and output indicate that you've been able to pass beyond the fingerprint verification and on to the next error.

I presume you meant to ask why you are getting this error now? If so, the simplest explanation is that the token ID does not, indeed, exist.

You can consult the list of tokens via:
pveum user token list root@pam

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox
 
Yes I figured out that part. I'm so sorry for the headache there. The new issue is the target storage. I was using Datastore2 and it was not finding it.

Code: 
root@proxmoxtest1:~# qm remote-migrate 100 103 'apitoken=PVEAPIToken=root@pam!root-token=f9ad8dbc-4f9f-407e-a13f-12321695f004,host=IP,fingerprint=DA:9C:20:8F:1A:2A:13:72:17:D9:D7:64:51:19:53:25:F4:44:0B:21:E2:35:FB:3F:2B:64:36:6C:54:D1:8A:5E' --target-bridge vmbr0 --target-storage Datastore2
remote: storage 'Datastore2' does not exist (or missing permission)!

I read that if it's a 1 to 1, same name storage, to try and use the number 1. This time around it's connecting but no datastore
Code: 
qm remote-migrate 100 103 'apitoken=PVEAPIToken=root@pam!root-token=f9ad8dbc-4f9f-407e-a13f-12321695f004,host=IP ,fingerprint=DA:9C:20:8F:1A:2A:13:72:17:D9:D7:64:51:19:53:25:F4:44:0B:21:E2:35:FB:3F:2B:64:36:6C:54:D1:8A:5E' --target-bridge vmbr0 --target-storage 1
Establishing API connection with remote at 'IP '
remote migration requires explicit storage mapping!
 
Last edited:
I apologize as it sounds I have tested your patience. I figured out that there are unique API special permissions and that CD's have to be removed if they are not found on the receiving end. The migration is running.
 
  • Like
Reactions: Johannes S, Tinwen and bbgeek17
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom
Back