Proxmox V4.1: Moving Disk of VM leads to data corruption

[Stefan Partner]

I used the "Move Disk" function in the "Hardware" tab of a Windows-VM (offline) but after starting the machine, the installed SQLServer failed to start with the following (fatal) error:

Could not open error log file Operating system error = 1392

Running a chkdsk inside the VM revealed several files system errors but even after chkdsk /f, SQLServer still fails to start with sql server error 9004...

Switching back to the original disk (raw format), resolves all this filesystem problems and SQLServer starts fine. No disk problems reported with chkdsk (in the guest).

So something destructive must have happened on my "move" of the disks from a local directory, to another local directory (on a different disk) on the same server. Examining what is happening when executing a "Move disk", ps revealed the following command line:

/usr/bin/qemu-img convert -t writeback -p -n -f raw -O raw /var/lib/vz/images/501/vm-501-disk-3.raw /mnt/pve/prox02-ssd-lv/images/501/vm-501-disk-2.raw

So what seems to happen, is not a simple fs move/copy, but a conversion from raw to raw...
The resulting file is also different from the original:

cmp /mnt/pve/prox02-ssd-lv/images/501/vm-501-disk-2.raw vm-501-disk-3.raw
vm-501-disk-2.raw vm-501-disk-3.raw differ: byte 1228841, line 737

I don't understand why a "convert" is happening at all (same image format) but the at the moment the result is a binary different image which is causing fatal filesystem errors in the (Windows) guest.

When I simply copy (cp) the disk to the new directory, the files are (thank god) identical and everything is working fine. If I use the "Move Disk" feature, I end up with a corrupted filesystem in the guest.

Am I missing something fundamental, am I doing something wrong or is this a well known or new bug/problem?

Thank you for your advise, the problem is reproduceable on my system!

 

[Stefan Partner]

as additional info, the (condensed) output of the qmmove task:

create full clone of drive virtio0 (local:501/vm-501-disk-3.raw)
Formatting '/mnt/pve/prox02-ssd-lv/images/501/vm-501-disk-2.raw', fmt=raw size=34359738368
transferred: 0 bytes remaining: 34359738368 bytes total: 34359738368 bytes progression: 0.00 %
transferred: 347033357 bytes remaining: 34012705011 bytes total: 34359738368 bytes progression: 1.01 %
transferred: 694066715 bytes remaining: 33665671653 bytes total: 34359738368 bytes progression: 2.02 %
...
transferred: 33621003993 bytes remaining: 738734375 bytes total: 34359738368 bytes progression: 97.85 %
transferred: 34359738368 bytes remaining: 0 bytes total: 34359738368 bytes progression: 100.00 %
TASK OK

 

[Stefan Partner]

After doing some more testing this seems to be a serious and reproducable bug.

Bottom line - Beware:
If you use the "Move Disk" feature to move a disk in .raw format to a different storage - tested with lvm and directory on the same proxmox host - you end up with critical file system errors, at least in a Windows guest.

 

[dcsapak]

hi,

i cannot reproduce this here,
i took a windows 10 vm, shut it down, and moved the disk image from one local storage (raw format) to another (raw format)
cmp showed no difference and chkdsk in the vm showed no errors

could you post the output of

pveversion -v

to be sure you run with the latest version?

also maybe you have bad ram or a bad disk?
this would explain why you get filesystem errors

 

[Stefan Partner]

Output of pveversion is

# pveversion
pve-manager/4.1-15/8cd55b52 (running kernel: 4.2.8-1-pve)

Storage and RAM are running rock solid, error only happens on "Disk move".
No problems with any other things, if the image is copied manually I get equal images and also no fs errrors in guest.

 

[Marek Hobler]

I just wanted to confirm that moving disks local -> NFS -> (offline migration to another machine) -> local means disk corruption.

We're using 4.1, hardware passes memtest86, no other signs of problems.

 

[dcsapak]

can you update to the latest version and try again?

 

[Marek Hobler]

Unfortunately, not at this time. Waiting for approval to buy subscription.

 

[dcsapak]

you can still update without subscription (but to the less stable no-subscription repository) see
https://pve.proxmox.com/wiki/Package_repositories