Unable to Migrate VM

[forbin]

Greetings,

I failed to migrate a Windows Server 2022 guest from PVE 7.3.3 to PVE 8.0.3. It failed with the following message.

drive-ide0: transferred 80.2 GiB of 2.1 TiB (3.82%) in 6m 8s
drive-ide0: transferred 80.2 GiB of 2.1 TiB (3.82%) in 6m 9s
drive-ide0: transferred 80.2 GiB of 2.1 TiB (3.82%) in 6m 10s
drive-ide0: Cancelling block job
drive-ide0: Done.
2023-07-01 20:52:10 ERROR: online migrate failure - block job (mirror) error: drive-ide0: 'mirror' has been cancelled
2023-07-01 20:52:10 aborting phase 2 - cleanup resources
2023-07-01 20:52:10 migrate_cancel
2023-07-01 20:52:15 ERROR: migration finished with problems (duration 00:06:24)
TASK ERROR: migration problems

We noticed that the disk in question is the OS/System drive, but also has an unallocated 2050.00 GB partition. We do not need that partition.

Here is the config of the VM in question.
root@vmhost52a:~# qm config 4802
bios: ovmf
boot: order=ide0;ide2;net0
cores: 4
efidisk0: zpool0:vm-4802-disk-0,efitype=4m,pre-enrolled-keys=1,size=528K
ide0: local-lvm:vm-4802-disk-0,size=2100G
ide2: none,media=cdrom
machine: pc-q35-7.1
memory: 256000
meta: creation-qemu=7.1.0,ctime=1675423459
name: mssql02
net0: e1000=DE:F9:24:E2:93:8E,bridge=vmbr0,firewall=1
numa: 0
ostype: win11
scsi0: zpool0:vm-4802-disk-2,iothread=1,size=6500G
scsihw: virtio-scsi-single
smbios1: uuid=c09633e4-04dd-4e30-a9bf-eebb0c744ea7
sockets: 4
tpmstate0: zpool0:vm-4802-disk-1,size=4M,version=v2.0
vmgenid: cce0dfc7-e4dd-426f-b949-e8f1b90cfcd5

root@vmhost52a:~# cat /etc/pve/storage.cfg
dir: local
path /var/lib/vz
content backup,iso,vztmpl

lvmthin: local-lvm
thinpool data
vgname pve
content rootdir,images

zfspool: zpool0
pool zpool0
content images,rootdir
mountpoint /zpool0
nodes vmhost50a,vmhost52a,vmhost50b,vmhost52b,vmhost51b,vmhost51a
sparse 1

nfs: store08a-nfs
export /zpool0/proxmox_storage
path /mnt/pve/store08a-nfs
server store08a
content backup,iso,images
prune-backups keep-all=1


root@vmhost52a:~# pvesm status
Name Type Status Total Used Available %
local dir active 67286400 2857332 60965288 4.25%
local-lvm lvmthin active 136929280 32630247 104299032 23.83%
store08a-nfs nfs active 17837517824 9226263552 8611254272 51.72%
zpool0 zfspool active 18028429312 4716540163 13311889148 26.16%


The full task log is attached.

 

[VictorSTS]

Do you have enough free disk space in the "local-lvm" storage in the destination PVE?

 

[forbin]

I believe so. Here are the details from the destination ProxMox.

root@vmhost52b:~# pvesm status
Name Type Status Total Used Available %
local dir active 67157384 4433040 59267116 6.60%
local-lvm lvmthin active 136544256 52432994 84111261 38.40%
store08a-nfs nfs active 17837517824 9226288128 8611229696 51.72%
zpool0 zfspool active 18028429312 411123739 17617305572 2.28%

root@vmhost52b:~# zfs list
NAME USED AVAIL REFER MOUNTPOINT
zpool0 392G 16.4T 140K /zpool0
zpool0/vm-2530-disk-0 80.7G 16.4T 80.7G -
zpool0/vm-2555-disk-0 80.0G 16.4T 80.0G -
zpool0/vm-2951-disk-0 151K 16.4T 151K -
zpool0/vm-7002-disk-0 231G 16.4T 231G -

 

[VictorSTS]

Sorry, but no:

local-lvm lvmthin active 136544256 52432994 84111261 38.40%

You only have a 136GB volume in the destination server, with just 84GB available.
I think you are trying to live migrate the VM. Even if the disk is mostly empty, using aprox 50GB from a 2TB disk, when doing a live migration the disk has to be fully copied bit by bit, losing the thin provisioning on the destination PVE. Thus, you need 2TB on the destination PVE on local-lvm storage. There are ways to recover such thin provisioning once the migration is done, of course.

You may try:

- Do a storage migration of the 2TB disk from local-lvm to zpool0 before migrating the VM. You will lose thin-provisioning for that disk at this point.
- Stop the VM and migrate to PVE8 host with the VM off.

 

[forbin]

Thank you for pointing that out. I thought the destination was zpool0, which has plenty of space. The guest has 2 disks. One of them is 6TB and that one migrates fine because the destination is zpool0. The OS disk is the problem because it is trying to go to local-lvm.

That said, I recall that the first time I tried the migration, I saw a message about the disk being resized from 51G to 2100G. That's probably the problem. There's a 2TB unformatted partition that got added but is not being used. Is there an easy way to remove that unused partition and shrink the disk?

2023-07-01 20:45:52 starting migration of VM 4802 to node 'vmhost52b' (192.168.8.110)
2023-07-01 20:45:52 found local disk 'local-lvm:vm-4802-disk-0' (in current VM config)
2023-07-01 20:45:52 found local disk 'zpool0:vm-4802-disk-0' (in current VM config)
2023-07-01 20:45:52 found generated disk 'zpool0:vm-4802-disk-1' (in current VM config)
2023-07-01 20:45:52 found local disk 'zpool0:vm-4802-disk-2' (in current VM config)
2023-07-01 20:45:52 drive 'ide0': size of disk 'local-lvm:vm-4802-disk-0' updated from 51G to 2100G
2023-07-01 20:45:52 copying local disk images
2023-07-01 20:45:53 full send of zpool0/vm-4802-disk-1@__migration__ estimated size is 45.5K
2023-07-01 20:45:53 total estimated size is 45.5K
2023-07-01 20:45:53 successfully imported 'zpool0:vm-4802-disk-1'
2023-07-01 20:45:53 volume 'zpool0:vm-4802-disk-1' is 'zpool0:vm-4802-disk-1' on the target
2023-07-01 20:45:53 starting VM 4802 on remote node 'vmhost52b'
2023-07-01 20:45:58 volume 'zpool0:vm-4802-disk-0' is 'zpool0:vm-4802-disk-0' on the target
2023-07-01 20:45:58 volume 'local-lvm:vm-4802-disk-0' is 'local-lvm:vm-4802-disk-0' on the target
2023-07-01 20:45:58 volume 'zpool0:vm-4802-disk-2' is 'zpool0:vm-4802-disk-2' on the target
2023-07-01 20:45:58 start remote tunnel
2023-07-01 20:45:59 ssh tunnel ver 1
2023-07-01 20:45:59 starting storage migration
2023-07-01 20:45:59 ide0: start migration to nbd:unix:/run/qemu-server/4802_nbd.migrate:exportname=drive-ide0
drive mirror is starting for drive-ide0
drive-ide0: transferred 0.0 B of 2.1 TiB (0.00%) in 0s
drive-ide0: transferred 256.0 MiB of 2.1 TiB (0.01%) in 1s
drive-ide0: transferred 507.0 MiB of 2.1 TiB (0.02%) in 2s
drive-ide0: transferred 768.0 MiB of 2.1 TiB (0.04%) in 3s
drive-ide0: transferred 1.0 GiB of 2.1 TiB (0.05%) in 4s
drive-ide0: transferred 1.3 GiB of 2.1 TiB (0.06%) in 5s
drive-ide0: transferred 1.5 GiB of 2.1 TiB (0.07%) in 6s
<snip>
drive-ide0: transferred 80.2 GiB of 2.1 TiB (3.82%) in 5m 58s
drive-ide0: transferred 80.2 GiB of 2.1 TiB (3.82%) in 5m 59s
drive-ide0: transferred 80.2 GiB of 2.1 TiB (3.82%) in 6m
drive-ide0: transferred 80.2 GiB of 2.1 TiB (3.82%) in 6m 1s
drive-ide0: transferred 80.2 GiB of 2.1 TiB (3.82%) in 6m 2s
drive-ide0: transferred 80.2 GiB of 2.1 TiB (3.82%) in 6m 3s
drive-ide0: transferred 80.2 GiB of 2.1 TiB (3.82%) in 6m 4s
drive-ide0: transferred 80.2 GiB of 2.1 TiB (3.82%) in 6m 5s
drive-ide0: transferred 80.2 GiB of 2.1 TiB (3.82%) in 6m 6s
drive-ide0: transferred 80.2 GiB of 2.1 TiB (3.82%) in 6m 7s
drive-ide0: transferred 80.2 GiB of 2.1 TiB (3.82%) in 6m 8s
drive-ide0: transferred 80.2 GiB of 2.1 TiB (3.82%) in 6m 9s
drive-ide0: transferred 80.2 GiB of 2.1 TiB (3.82%) in 6m 10s
drive-ide0: Cancelling block job
drive-ide0: Done.
2023-07-01 20:52:10 ERROR: online migrate failure - block job (mirror) error: drive-ide0: 'mirror' has been cancelled
2023-07-01 20:52:10 aborting phase 2 - cleanup resources
2023-07-01 20:52:10 migrate_cancel
2023-07-01 20:52:15 ERROR: migration finished with problems (duration 00:06:24)
TASK ERROR: migration problems

 

[niteshadow]

Did you try to first move your local-lvm:vm-4802-disk-0 disk to the storage zpool0 ?
Interesting that even your EFI disk is on zpool0.

Or add a disk (to zpool0), clone system disk to new disk, and swap the disks (detach original, attach new disk to ide0).
Generally I clone/restore using a live bootable iso, such as Clonezilla Live.
You can even clone to a smaller disk (but maybe not with Clonezilla).