Unable to Migrate VM

forbin

Member
Dec 16, 2021
30
5
8
44
Greetings,

I failed to migrate a Windows Server 2022 guest from PVE 7.3.3 to PVE 8.0.3. It failed with the following message.

drive-ide0: transferred 80.2 GiB of 2.1 TiB (3.82%) in 6m 8s
drive-ide0: transferred 80.2 GiB of 2.1 TiB (3.82%) in 6m 9s
drive-ide0: transferred 80.2 GiB of 2.1 TiB (3.82%) in 6m 10s
drive-ide0: Cancelling block job
drive-ide0: Done.
2023-07-01 20:52:10 ERROR: online migrate failure - block job (mirror) error: drive-ide0: 'mirror' has been cancelled
2023-07-01 20:52:10 aborting phase 2 - cleanup resources
2023-07-01 20:52:10 migrate_cancel
2023-07-01 20:52:15 ERROR: migration finished with problems (duration 00:06:24)
TASK ERROR: migration problems

We noticed that the disk in question is the OS/System drive, but also has an unallocated 2050.00 GB partition. We do not need that partition.

Here is the config of the VM in question.
root@vmhost52a:~# qm config 4802
bios: ovmf
boot: order=ide0;ide2;net0
cores: 4
efidisk0: zpool0:vm-4802-disk-0,efitype=4m,pre-enrolled-keys=1,size=528K
ide0: local-lvm:vm-4802-disk-0,size=2100G
ide2: none,media=cdrom
machine: pc-q35-7.1
memory: 256000
meta: creation-qemu=7.1.0,ctime=1675423459
name: mssql02
net0: e1000=DE:F9:24:E2:93:8E,bridge=vmbr0,firewall=1
numa: 0
ostype: win11
scsi0: zpool0:vm-4802-disk-2,iothread=1,size=6500G
scsihw: virtio-scsi-single
smbios1: uuid=c09633e4-04dd-4e30-a9bf-eebb0c744ea7
sockets: 4
tpmstate0: zpool0:vm-4802-disk-1,size=4M,version=v2.0
vmgenid: cce0dfc7-e4dd-426f-b949-e8f1b90cfcd5

root@vmhost52a:~# cat /etc/pve/storage.cfg
dir: local
path /var/lib/vz
content backup,iso,vztmpl

lvmthin: local-lvm
thinpool data
vgname pve
content rootdir,images

zfspool: zpool0
pool zpool0
content images,rootdir
mountpoint /zpool0
nodes vmhost50a,vmhost52a,vmhost50b,vmhost52b,vmhost51b,vmhost51a
sparse 1

nfs: store08a-nfs
export /zpool0/proxmox_storage
path /mnt/pve/store08a-nfs
server store08a
content backup,iso,images
prune-backups keep-all=1


root@vmhost52a:~# pvesm status
Name Type Status Total Used Available %
local dir active 67286400 2857332 60965288 4.25%
local-lvm lvmthin active 136929280 32630247 104299032 23.83%
store08a-nfs nfs active 17837517824 9226263552 8611254272 51.72%
zpool0 zfspool active 18028429312 4716540163 13311889148 26.16%


The full task log is attached.
 

Attachments

  • full_task_log.txt
    38.7 KB · Views: 3
I believe so. Here are the details from the destination ProxMox.

root@vmhost52b:~# pvesm status
Name Type Status Total Used Available %
local dir active 67157384 4433040 59267116 6.60%
local-lvm lvmthin active 136544256 52432994 84111261 38.40%
store08a-nfs nfs active 17837517824 9226288128 8611229696 51.72%
zpool0 zfspool active 18028429312 411123739 17617305572 2.28%

root@vmhost52b:~# zfs list
NAME USED AVAIL REFER MOUNTPOINT
zpool0 392G 16.4T 140K /zpool0
zpool0/vm-2530-disk-0 80.7G 16.4T 80.7G -
zpool0/vm-2555-disk-0 80.0G 16.4T 80.0G -
zpool0/vm-2951-disk-0 151K 16.4T 151K -
zpool0/vm-7002-disk-0 231G 16.4T 231G -
 
Sorry, but no:

Code:
local-lvm lvmthin active 136544256 52432994 84111261 38.40%

You only have a 136GB volume in the destination server, with just 84GB available.
I think you are trying to live migrate the VM. Even if the disk is mostly empty, using aprox 50GB from a 2TB disk, when doing a live migration the disk has to be fully copied bit by bit, losing the thin provisioning on the destination PVE. Thus, you need 2TB on the destination PVE on local-lvm storage. There are ways to recover such thin provisioning once the migration is done, of course.

You may try:

- Do a storage migration of the 2TB disk from local-lvm to zpool0 before migrating the VM. You will lose thin-provisioning for that disk at this point.
- Stop the VM and migrate to PVE8 host with the VM off.
 
Thank you for pointing that out. I thought the destination was zpool0, which has plenty of space. The guest has 2 disks. One of them is 6TB and that one migrates fine because the destination is zpool0. The OS disk is the problem because it is trying to go to local-lvm.

That said, I recall that the first time I tried the migration, I saw a message about the disk being resized from 51G to 2100G. That's probably the problem. There's a 2TB unformatted partition that got added but is not being used. Is there an easy way to remove that unused partition and shrink the disk?

2023-07-01 20:45:52 starting migration of VM 4802 to node 'vmhost52b' (192.168.8.110)
2023-07-01 20:45:52 found local disk 'local-lvm:vm-4802-disk-0' (in current VM config)
2023-07-01 20:45:52 found local disk 'zpool0:vm-4802-disk-0' (in current VM config)
2023-07-01 20:45:52 found generated disk 'zpool0:vm-4802-disk-1' (in current VM config)
2023-07-01 20:45:52 found local disk 'zpool0:vm-4802-disk-2' (in current VM config)
2023-07-01 20:45:52 drive 'ide0': size of disk 'local-lvm:vm-4802-disk-0' updated from 51G to 2100G
2023-07-01 20:45:52 copying local disk images
2023-07-01 20:45:53 full send of zpool0/vm-4802-disk-1@__migration__ estimated size is 45.5K
2023-07-01 20:45:53 total estimated size is 45.5K
2023-07-01 20:45:53 successfully imported 'zpool0:vm-4802-disk-1'
2023-07-01 20:45:53 volume 'zpool0:vm-4802-disk-1' is 'zpool0:vm-4802-disk-1' on the target
2023-07-01 20:45:53 starting VM 4802 on remote node 'vmhost52b'
2023-07-01 20:45:58 volume 'zpool0:vm-4802-disk-0' is 'zpool0:vm-4802-disk-0' on the target
2023-07-01 20:45:58 volume 'local-lvm:vm-4802-disk-0' is 'local-lvm:vm-4802-disk-0' on the target
2023-07-01 20:45:58 volume 'zpool0:vm-4802-disk-2' is 'zpool0:vm-4802-disk-2' on the target
2023-07-01 20:45:58 start remote tunnel
2023-07-01 20:45:59 ssh tunnel ver 1
2023-07-01 20:45:59 starting storage migration
2023-07-01 20:45:59 ide0: start migration to nbd:unix:/run/qemu-server/4802_nbd.migrate:exportname=drive-ide0
drive mirror is starting for drive-ide0
drive-ide0: transferred 0.0 B of 2.1 TiB (0.00%) in 0s
drive-ide0: transferred 256.0 MiB of 2.1 TiB (0.01%) in 1s
drive-ide0: transferred 507.0 MiB of 2.1 TiB (0.02%) in 2s
drive-ide0: transferred 768.0 MiB of 2.1 TiB (0.04%) in 3s
drive-ide0: transferred 1.0 GiB of 2.1 TiB (0.05%) in 4s
drive-ide0: transferred 1.3 GiB of 2.1 TiB (0.06%) in 5s
drive-ide0: transferred 1.5 GiB of 2.1 TiB (0.07%) in 6s
<snip>
drive-ide0: transferred 80.2 GiB of 2.1 TiB (3.82%) in 5m 58s
drive-ide0: transferred 80.2 GiB of 2.1 TiB (3.82%) in 5m 59s
drive-ide0: transferred 80.2 GiB of 2.1 TiB (3.82%) in 6m
drive-ide0: transferred 80.2 GiB of 2.1 TiB (3.82%) in 6m 1s
drive-ide0: transferred 80.2 GiB of 2.1 TiB (3.82%) in 6m 2s
drive-ide0: transferred 80.2 GiB of 2.1 TiB (3.82%) in 6m 3s
drive-ide0: transferred 80.2 GiB of 2.1 TiB (3.82%) in 6m 4s
drive-ide0: transferred 80.2 GiB of 2.1 TiB (3.82%) in 6m 5s
drive-ide0: transferred 80.2 GiB of 2.1 TiB (3.82%) in 6m 6s
drive-ide0: transferred 80.2 GiB of 2.1 TiB (3.82%) in 6m 7s
drive-ide0: transferred 80.2 GiB of 2.1 TiB (3.82%) in 6m 8s
drive-ide0: transferred 80.2 GiB of 2.1 TiB (3.82%) in 6m 9s
drive-ide0: transferred 80.2 GiB of 2.1 TiB (3.82%) in 6m 10s
drive-ide0: Cancelling block job
drive-ide0: Done.
2023-07-01 20:52:10 ERROR: online migrate failure - block job (mirror) error: drive-ide0: 'mirror' has been cancelled
2023-07-01 20:52:10 aborting phase 2 - cleanup resources
2023-07-01 20:52:10 migrate_cancel
2023-07-01 20:52:15 ERROR: migration finished with problems (duration 00:06:24)
TASK ERROR: migration problems
 
Did you try to first move your local-lvm:vm-4802-disk-0 disk to the storage zpool0 ?
Interesting that even your EFI disk is on zpool0.

Or add a disk (to zpool0), clone system disk to new disk, and swap the disks (detach original, attach new disk to ide0).
Generally I clone/restore using a live bootable iso, such as Clonezilla Live.
You can even clone to a smaller disk (but maybe not with Clonezilla).
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!