online migrate failure - block job (mirror) error: Source and target image have different sizes

[Mabene]

I just ran into an issue trying to live migrate a VM; source and target are zfs, version is 8.4.1

Error message: online migrate failure - block job (mirror) error: drive-scsi0: Source and target image have different sizes

The disk was created by importing from a Hyper-V vhd image, that turns out to have a size not rounded to the zfs block size of 128k.

qm importdisk 103 /mnt/pve/hv1-d/vm/virtual1/Virtual\ Hard\ Disks/virtual1.vhd zfs -format raw
zfs get volsize zfs/vm-103-disk-0 -p
NAME               PROPERTY  VALUE    SOURCE
zfs/vm-103-disk-0  volsize   68719493120  local

Trying to migrate results in the above error; adjusting the size to the next GB boundary fixed the issue

zfs set volsize=69793218560 zfs/vm-103-disk-0

The question is if this could be caught at an earlier point, i.e could/should qm importdisk round up the size to match target blocksize?

 

[fabian]

do you still have the original file? if so, could you post the output of "qemu-img info --output-format json /path/to/file"?

the ZFS storage plugin should align to the next 1M boundary.. and it seems to do that on the target side (hence the size mismatch)

you just did "qm importdisk" directly on the .vhd file onto the ZFS storage?

 

[Mabene]

let me see.. I still have the original vhd file, so I can recreate the situation. And yes, I first created the vm on VE without disks and then ran the qm importdisk command from the original post.

-rwxr-xr-x 1 root root 68719477248 May 26 14:20 '/mnt/pve/hv1-d/vm/virtual1/Virtual Hard Disks/virtual1.vhd'

That is 64G (64x1024x1024x1024) raw image size + 512 Bytes vhd metadata.

Here's the vhd info from the windows side:

PS D:\vm\virtual1\Virtual Hard Disks> Get-VHD -Path .\virtual1.vhd

ComputerName            : HV1
Path                    : D:\vm\virtual1\Virtual Hard Disks\virtual1.vhd
VhdFormat               : VHD
VhdType                 : Fixed
FileSize                : 68719477248
Size                    : 68719476736
MinimumSize             : 68715037696
LogicalSectorSize       : 512
PhysicalSectorSize      : 512
BlockSize               : 0
ParentPath              :
DiskIdentifier          : 5C057645-3DB2-48C1-9332-24F5670FA567
FragmentationPercentage : 0
Alignment               : 1
Attached                : False
DiskNumber              :
IsPMEMCompatible        : False
AddressAbstractionType  : None
Number                  :

qemu-img won't work as the target is a zfs zvol, not a regular file

zfs get recordsize
NAME                                              PROPERTY    VALUE    SOURCE
zfs                                               recordsize  128K     default

zfs recordsize is 128k; it looks like the result of qm importdisk is the actual (raw) image + the 512 byte vhd metadata block, padded to the nearest 16K boundary.

 

[fabian]

I meant qemu-img info for the vhd file FWIW, you can also pass it the zvol path (/dev/zvol/...), but I am not interested in that..

 

[Mabene]

OOps, sorry about that.

root@ve2:~# qemu-img info /mnt/pve/hv1-d/vm/virtual1/Virtual\ Hard\ Disks/virtual1.vhd
image: /mnt/pve/hv1-d/vm/virtual1/Virtual Hard Disks/virtual1.vhd
file format: raw
virtual size: 64 GiB (68719477248 bytes)
disk size: 64 GiB
Child node '/file':
    filename: /mnt/pve/hv1-d/vm/virtual1/Virtual Hard Disks/virtual1.vhd
    protocol type: file
    file length: 64 GiB (68719477248 bytes)
    disk size: 64 GiB

 

[fabian]

thanks!

could you also post the output of

perl -e 'use strict; use warnings; use PVE::Storage; my $size = PVE::Storage::file_size_info("/mnt/pve/hv1-d/vm/virtual1/Virtual Hard Disks/virtual1.vhd", undef, "auto-detect"); print "$size\n";'

 

[Mabene]

68719477248, same as qemu-img. Looks like vhd is not recognized and treated as raw, so we get 68719476736 (64x1024x1024x1024) + 512 byte vhd metadata; import only works at all because it's a fixed size vhd that consists of raw image + metadata tacked onto the end.

 

[Mabene]

qemu-img actually produces correct info if I specify the format:

qemu-img info "/mnt/pve/hv1-d/vm/virtual1/Virtual Hard Disks/virtual1.vhd" -f vpc
image: /mnt/pve/hv1-d/vm/virtual1/Virtual Hard Disks/virtual1.vhd
file format: vpc
virtual size: 64 GiB (68719476736 bytes)
disk size: 64 GiB
Child node '/file':
    filename: /mnt/pve/hv1-d/vm/virtual1/Virtual Hard Disks/virtual1.vhd
    protocol type: file
    file length: 64 GiB (68719477248 bytes)
    disk size: 64 GiB

 

[fabian]

interesting - so the format is not properly auto-detected! thanks

 

[Mabene]

So we're actually seeing several issues:

• vhd autodetect issue
• no parameter for source format for qm importdisk (?)
• rounding of target size

src size (raw image)	zvol size	padding	live migration
1 073 741 824 (1G)	1 073 741 824	0	OK
1 073 742 848 (1G + 1024)	1 074 790 400	1 047 552 (to 1M)	OK
1 073 742 336 (1G + 512)	1 073 758 208	15 872 (to 16K)	Error

so a raw size aligned to 512 Bytes results in a 16K aligned image that can't be life migrated

 

[fabian]

the storage layer operates in kilobytes, so the 0.5kB there is not handled properly (and yes, we'd need to pass the source format explicitly as well).