Remote migration of cold VM - EERROR: no export formats for 'VM-pool:vm-9999-disk-0' - check storage plugin support!

[__Peter__]

While doing some testing with qm remote-migrate, I got this error message while trying to migrate a cold VM. However migrating it live works OK:

qm remote-migrate 9999 9999 apitoken='Authorization: PVEAPIToken=root@pam!YY=XX-XX-XX-XX-XX,fingerprint=XX:XX...:XX',host=X.X.X.X --target-bridge vmbr0 --target-storage VM-pool --delete
-> ERROR: no export formats for 'VM-pool:vm-9999-disk-0' - check storage plugin support!

qm remote-migrate 9999 9999 apitoken='Authorization: PVEAPIToken=root@pam!YY=XX-XX-XX-XX-XX,fingerprint=XX:XX...:XX',host=X.X.X.X --target-bridge vmbr0 --target-storage VM-pool --online --delete
-> Works OK, and VM is migrated live

Any ideas??

Cheers,
Peter

 

[spirit]

remote migration of offline vm is not yet supported

 

[__Peter__]

That would do it

 

[fiona]

To be precise, remote offline migration is supported, but only for certain storage combinations: e.g. ZFS -> ZFS works, RBD doesn't (our RBD plugin currently lacks offline export/import functionality).

 

[xiaolin0199]

To be precise, remote offline migration is supported, but only for certain storage combinations: e.g. ZFS -> ZFS works, RBD doesn't (our RBD plugin currently lacks offline export/import functionality).

Is RBD still not supported yet?

 

[fiona]

Hi,

Is RBD still not supported yet?

unfortunately not. But it's still planned of course.

 

[SERvus]

Hi,

unfortunately not. But it's still planned of course.

Is RBD online supportet? I get the same Error, when I try to migrate from Ceph to Ceph.

 

[fiona]

Hi,

Is RBD online supportet? I get the same Error, when I try to migrate from Ceph to Ceph.

yes, (remote) live-migration should work no matter what storage type you are using. Remote offline-migration from RBD to RBD is still not implemented, unfortunately.

 

[SERvus]

Hi,

yes, (remote) live-migration should work no matter what storage type you are using. Remote offline-migration from RBD to RBD is still not implemented, unfortunately.

Is there a different mechanism at work? Offline should be more easier than online, shouldn't it?

 

[fiona]

Is there a different mechanism at work? Offline should be more easier than online, shouldn't it?

Yes, live-migration uses QEMU's NBD export + disk mirroring. Remote migration is still in-development and RBD is shared within a cluster, so there simply was no need to have an export/import mechanism for offline migration. That's the reason it was not already implemented.

 

[deepcloud]

I think RBD to RBD is supported now as I managed to do it just now for 1 VM.
both the clusters have CEPH running and the VM on RBD successfully migrated to a new proxmox cluster on CEPH again.
wonderful

but another VM with TPM state did not work - any ideas for this

 

[fiona]

I think RBD to RBD is supported now as I managed to do it just now for 1 VM.

With online migration, it works. With offline migration, it is not implemented yet.

both the clusters have CEPH running and the VM on RBD successfully migrated to a new proxmox cluster on CEPH again.
wonderful

but another VM with TPM state did not work - any ideas for this

Please share the full migration log and the VM configuration qm config <ID>

 

[deepcloud]

qm config 9997
-----------------------
agent: 1
balloon: 0
bios: seabios
boot: order=scsi0;ide0;net0
cores: 4
cpu: host
description: Server 2022
ide0: none,media=cdrom
machine: pc-q35-8.1
memory: 16384
meta: creation-qemu=8.1.2,ctime=1705037235
name: Win2022-Office
net0: virtio=BC:24:11:A3:F31,bridge=vmbr0,firewall=1
numa: 0
ostype: win11
scsi0: Storage:vm-9997-disk-3,cache=writeback,discard=on,size=70G,ssd=1
scsihw: virtio-scsi-pci
smbios1: uuid=35245ac8-6179-4057-a50f-e461d1dc1a4a
sockets: 1
template: 0
tpmstate0: Storage:vm-9997-disk-2,size=4M,version=v2.0
vmgenid: 8c948c52-a326-48b2-9b3b-be57e6e7e882

------------------------------------------------------------------
command given in shell (this works perfectly for machines without TPM)
---------------------------------------------------------------------------------------------------------
qm remote-migrate 9997 9997 apitoken='PVEAPIToken=root@pam!roottoken=a75a33bf-47g2-2a24-9437-f7a81bb21e59,host=172.16.0.12,fingerprint=32:42:A5:E0:A1:14:86:AC:65:A3:1BA:CC:93:1D6:3E:13:64:33:64:BE:10:BC:11:74:35:58:C8:1A:87:EB' --online --target-bridge vmbr0 --target-storage RBDNVME --delete 1
---------------------------------------------------------------------------------------------------------

Establishing API connection with remote at '172.16.0.12'
2024-06-26 12:29:32 remote: started tunnel worker 'UPID:ww1r1u12:00360062:02EDA200:667BBC54:qmtunnel:9997:root@pam!roottoken:'
tunnel: -> sending command "version" to remote
tunnel: <- got reply
2024-06-26 12:29:32 local WS tunnel version: 2
2024-06-26 12:29:32 remote WS tunnel version: 2
2024-06-26 12:29:32 minimum required WS tunnel version: 2
websocket tunnel started
2024-06-26 12:29:32 starting migration of VM 9997 to node 'w12' (172.16.0.12)
tunnel: -> sending command "bwlimit" to remote
tunnel: <- got reply
tunnel: -> sending command "bwlimit" to remote
tunnel: <- got reply
2024-06-26 12:29:32 found generated disk 'Storage:vm-9997-disk-2' (in current VM config)
2024-06-26 12:29:32 found local disk 'Storage:vm-9997-disk-3' (attached)
2024-06-26 12:29:32 copying local disk images
2024-06-26 12:29:32 ERROR: no export formats for 'Storage:vm-9997-disk-2' - check storage plugin support!
2024-06-26 12:29:32 aborting phase 1 - cleanup resources
tunnel: -> sending command "quit" to remote
tunnel: <- got reply
2024-06-26 12:29:34 ERROR: migration aborted (duration 00:00:02): no export formats for 'Storage:vm-9997-disk-2' - check storage plugin support!
migration aborted

--------------------------------------------------------------------------------------------------------

log is below
------------------

2024-06-26 12:29:32 remote: started tunnel worker 'UPID:ww1r1u12:00360062:02EDA200:667BBC54:qmtunnel:9997:root@pam!roottoken:'
tunnel: -> sending command "version" to remote
tunnel: <- got reply
2024-06-26 12:29:32 local WS tunnel version: 2
2024-06-26 12:29:32 remote WS tunnel version: 2
2024-06-26 12:29:32 minimum required WS tunnel version: 2
websocket tunnel started
2024-06-26 12:29:32 starting migration of VM 9997 to node 'w12' (172.16.0.12)
tunnel: -> sending command "bwlimit" to remote
tunnel: <- got reply
tunnel: -> sending command "bwlimit" to remote
tunnel: <- got reply
2024-06-26 12:29:32 found generated disk 'Storage:vm-9997-disk-2' (in current VM config)
2024-06-26 12:29:32 found local disk 'Storage:vm-9997-disk-3' (attached)
2024-06-26 12:29:32 copying local disk images
2024-06-26 12:29:32 ERROR: no export formats for 'Storage:vm-9997-disk-2' - check storage plugin support!
2024-06-26 12:29:32 aborting phase 1 - cleanup resources
tunnel: -> sending command "quit" to remote
tunnel: <- got reply
2024-06-26 12:29:34 ERROR: migration aborted (duration 00:00:02): no export formats for 'Storage:vm-9997-disk-2' - check storage plugin support!

TASK ERROR: migration aborted

 

[fiona]

I see. The TPM state drive is handled differently, because it is not directly managed by QEMU (but by the swtpm service) and therefore it is not live-migrated using QEMU's drive-mirror functionality, but via the storage plugin. That issue will be resolved once the RBD plugin supports offline export/import.