[SOLVED] remote-migrate fails: found stale volume copy

[sjjh]

I'm trying to migrate a container (id: 104) from a node pve-test to another node PS02 with the new id 113, but receiving an error:

root@pve-test:~# pct remote-migrate 104 113 apitoken='PVEAPIToken=root@pam!migration=XXX',host=10.63.5.21,fingerprint=XXX --target-bridge vmbr1 --target-storage SSD_STORAGE --restart
Establishing API connection with remote at '10.63.5.21'
2025-02-07 14:46:47 remote: started tunnel worker 'UPID:PS02:0036C2B0:03B74410:67A60EC7:vzmtunnel:113:root@pam!migration:'
tunnel: -> sending command "version" to remote
tunnel: <- got reply
2025-02-07 14:46:47 local WS tunnel version: 2
2025-02-07 14:46:47 remote WS tunnel version: 2
2025-02-07 14:46:47 minimum required WS tunnel version: 2
2025-02-07 14:46:47 websocket tunnel started
2025-02-07 14:46:47 shutdown CT 104
2025-02-07 14:46:51 starting migration of CT 104 to node 'PS02' (10.63.5.21)
tunnel: -> sending command "bwlimit" to remote
tunnel: <- got reply
tunnel: -> sending command "bwlimit" to remote
tunnel: <- got reply
2025-02-07 14:46:51 found local volume 'local-lvm:vm-104-disk-0' (in current VM config)
2025-02-07 14:46:51 found local volume 'zfs_at_pve-test_sdb:subvol-104-disk-0' (in current VM config)
tunnel: -> sending command "disk-import" to remote
tunnel: <- got reply
tunnel: accepted new connection on '/run/pve/104.storage'
tunnel: requesting WS ticket via tunnel
tunnel: established new WS for forwarding '/run/pve/104.storage'
full send of zfs_at_pve-test_sdb/subvol-104-disk-0@__migration__ estimated size is 1.36G
total estimated size is 1.36G
TIME        SENT   SNAPSHOT zfs_at_pve-test_sdb/subvol-104-disk-0@__migration__
14:46:53   89.7M   zfs_at_pve-test_sdb/subvol-104-disk-0@__migration__
14:46:54    187M   zfs_at_pve-test_sdb/subvol-104-disk-0@__migration__
14:46:55    294M   zfs_at_pve-test_sdb/subvol-104-disk-0@__migration__
14:46:56    378M   zfs_at_pve-test_sdb/subvol-104-disk-0@__migration__
14:46:57    457M   zfs_at_pve-test_sdb/subvol-104-disk-0@__migration__
14:46:58    519M   zfs_at_pve-test_sdb/subvol-104-disk-0@__migration__
14:46:59    592M   zfs_at_pve-test_sdb/subvol-104-disk-0@__migration__
14:47:00    697M   zfs_at_pve-test_sdb/subvol-104-disk-0@__migration__
14:47:01    796M   zfs_at_pve-test_sdb/subvol-104-disk-0@__migration__
14:47:02    903M   zfs_at_pve-test_sdb/subvol-104-disk-0@__migration__
14:47:03    970M   zfs_at_pve-test_sdb/subvol-104-disk-0@__migration__
14:47:04   1.05G   zfs_at_pve-test_sdb/subvol-104-disk-0@__migration__
14:47:05   1.16G   zfs_at_pve-test_sdb/subvol-104-disk-0@__migration__
14:47:06   1.26G   zfs_at_pve-test_sdb/subvol-104-disk-0@__migration__
14:47:07   1.36G   zfs_at_pve-test_sdb/subvol-104-disk-0@__migration__
tunnel: done handling forwarded connection from '/run/pve/104.storage'
tunnel: -> sending command "query-disk-import" to remote
tunnel: <- got reply
2025-02-07 14:47:08 volume 'zfs_at_pve-test_sdb:subvol-104-disk-0' is 'SSD_STORAGE:subvol-113-disk-0' on the target
tunnel: -> sending command "disk-import" to remote
tunnel: <- got reply
2025-02-07 14:47:08 ERROR: storage migration for 'local-lvm:vm-104-disk-0' to storage 'SSD_STORAGE' failed - error - tunnel command '{"snapshot":null,"cmd":"disk-import","volname":"vm-113-disk-0","migration_snapshot":"","export_formats":"raw+size","allow_rename":"1","with_snapshots":0,"storage":"SSD_STORAGE","format":"raw"}' failed - failed to handle 'disk-import' command - 400 Parameter verification failed.
2025-02-07 14:47:08 ERROR: migration_snapshot: type check ('boolean') failed - got ''
2025-02-07 14:47:08 aborting phase 1 - cleanup resources
2025-02-07 14:47:08 ERROR: found stale volume copy 'SSD_STORAGE:subvol-113-disk-0' on node 'PS02'
2025-02-07 14:47:08 ERROR: found stale volume copy 'local-lvm:vm-104-disk-0' on node 'PS02'
tunnel: -> sending command "quit" to remote
tunnel: <- got reply
2025-02-07 14:47:10 start final cleanup
2025-02-07 14:47:10 start container on source node
2025-02-07 14:47:12 ERROR: migration aborted (duration 00:00:25): storage migration for 'local-lvm:vm-104-disk-0' to storage 'SSD_STORAGE' failed - error - tunnel command '{"snapshot":null,"cmd":"disk-import","volname":"vm-113-disk-0","migration_snapshot":"","export_formats":"raw+size","allow_rename":"1","with_snapshots":0,"storage":"SSD_STORAGE","format":"raw"}' failed - failed to handle 'disk-import' command - 400 Parameter verification failed.
2025-02-07 14:47:12 ERROR: migration_snapshot: type check ('boolean') failed - got ''
migration aborted

I have migrated already two containers like this, thus assuming that the command itself is working correct.
I did not find the mentioned stale volume copies in the web UI storage section of node PS02 (it even does not have a storage local-lvm). I found a blog post recommending to rescan to uncover the volume copies. But this (naturally) also leads to an error:

root@PS02:~# pct rescan -vmid 113
rescan volumes...
Configuration file 'nodes/PS02/lxc/113.conf' does not exist

I thus copied over the config /etc/pve/nodes/pve-test/lxc/104.conf to /etc/pve/nodes/PS02/lxc/113.conf,edited the storage lines and rerun the rescan command -- still without success (the volume copies do not show up).
Now I'm stuck. Any idea how to solve the issue?

 

[sjjh]

Problem solved. After removing the storage local-lvm:vm-104-disk-0 on the source node, which was marked as unused, the migration worked without any issue.