tldr;
If you start a VM migration whos disks reside on a zfs volume and for whatever reason cancel in the middle of its progress, proxmox will not properly clean itself up resulting in failures of consequent attempts to migrate that same VM. Solution is to manually remove the zfs snapshots proxmox left behind on the ORIGINATING host.
Detailed version...
Running pve 4.3, cluster of 10 nodes, if you cancel an offline migration of zfs local storage, proxmox will fail to clean up the volume snapshot(s) that were created to migrate the VM on the originating host. The next time you try to migrate that VM, you'll get an error like this:
So in this case I migrated vmid 103 , from node prx027 to prx019 and purposely canceled the migration. The procedure has been confirmed to be repeatable across any node.
The error above is also deceiving and incorrect. The "stale volume" actually resides on the originating node(prx027 in this case) and not the receiver(prx019) as the error indicates.
Migration and migration cancellation
So heres what happens when you go to migrate that same VM again:
Note in the above error, there are 2 conflicting messages
1) Which is accurate
2) The error people will likely read since its the most explicit and is trying to point you in the direction of where the failure is, but is wrong.
Here are those snapshots, the error is warning about, however, again they reside on the originating node, not the receiver.
The solution is to simply remove these snapshots at shell from the originating host.
Might want to fix that in later releases.
If you start a VM migration whos disks reside on a zfs volume and for whatever reason cancel in the middle of its progress, proxmox will not properly clean itself up resulting in failures of consequent attempts to migrate that same VM. Solution is to manually remove the zfs snapshots proxmox left behind on the ORIGINATING host.
Detailed version...
Running pve 4.3, cluster of 10 nodes, if you cancel an offline migration of zfs local storage, proxmox will fail to clean up the volume snapshot(s) that were created to migrate the VM on the originating host. The next time you try to migrate that VM, you'll get an error like this:
So in this case I migrated vmid 103 , from node prx027 to prx019 and purposely canceled the migration. The procedure has been confirmed to be repeatable across any node.
The error above is also deceiving and incorrect. The "stale volume" actually resides on the originating node(prx027 in this case) and not the receiver(prx019) as the error indicates.
Migration and migration cancellation
So heres what happens when you go to migrate that same VM again:
Note in the above error, there are 2 conflicting messages
1) Which is accurate
2) The error people will likely read since its the most explicit and is trying to point you in the direction of where the failure is, but is wrong.
Here are those snapshots, the error is warning about, however, again they reside on the originating node, not the receiver.
Code:
root@prx027:/home# zfs list -t snapshot
NAME USED AVAIL REFER MOUNTPOINT
zfsStoragePool001/vm-103-disk-2@__migration__ 0 - 17.3M -
zfsStoragePool001/vm-103-disk-4@__migration__ 0 - 19.6M -The solution is to simply remove these snapshots at shell from the originating host.
Code:
root@prx027:/home# zfs list -t snapshot
NAME USED AVAIL REFER MOUNTPOINT
zfsStoragePool001/vm-103-disk-2@__migration__ 0 - 17.3M -
zfsStoragePool001/vm-103-disk-4@__migration__ 0 - 19.6M -
root@prx027:/home# zfs destroy zfsStoragePool001/vm-103-disk-2@__migration__
root@prx027:/home# zfs destroy zfsStoragePool001/vm-103-disk-4@__migration__
root@prx027:/home# zfs list -t snapshot
no datasets availableMight want to fix that in later releases.
