multiple of volume block size error during vm migration

ycomet

New Member
Jul 16, 2020
5
1
3
40
Hello,

I am trying to move a VM from one node to another inside the the same Proxmox VE cluster.
Storage is rather large but the destination node has enough disk capacity.

I am getting an error when I try to do an online migration:

Code:
2020-07-15 09:20:50 starting migration of VM 101 to node 'prc23' (10.0.0.3)
2020-07-15 09:20:50 found local disk 'HDD:vm-101-disk-0' (in current VM config)
2020-07-15 09:20:50 copying local disk images
2020-07-15 09:20:50 starting VM 101 on remote node 'prc23'
2020-07-15 09:20:50 [prc23] zfs error: cannot create 'hdd/vm-101-disk-0': volume size must be a multiple of volume block size
2020-07-15 09:20:50 ERROR: online migrate failure - remote command failed with exit code 255
2020-07-15 09:20:50 aborting phase 2 - cleanup resources
2020-07-15 09:20:50 migrate_cancel
2020-07-15 09:20:51 ERROR: migration finished with problems (duration 00:00:02)

If I run a lsblk -b inside of the VM I get a disk size of 2903784816640.
However, Proxmox reports the size of the hard disk incorrectly (scsi0: HDD:vm-101-disk-0,cache=writethrough,size=2902710697329)


While trying to change the hard disk configuration (from no cache to write through cache) I noticed the size is automatically updated to 2769265M - which is the same that I got with lsblk -b command in the VM.


After power cycling the VM and the settings updated, tried running the live migration once again but I got the error log below:

Code:
2020-07-15 11:05:07 starting migration of VM 101 to node 'prc23' (10.0.0.3)
2020-07-15 11:05:07 found local disk 'HDD:vm-101-disk-0' (in current VM config)
2020-07-15 11:05:07 size of disk 'HDD:vm-101-disk-0' (scsi0) updated from 2769265M to 2902710697329
2020-07-15 11:05:07 copying local disk images
2020-07-15 11:05:07 starting VM 101 on remote node 'prc23'
2020-07-15 11:05:07 [prc23] zfs error: cannot create 'hdd/vm-101-disk-0': volume size must be a multiple of volume block size
2020-07-15 11:05:07 ERROR: online migrate failure - remote command failed with exit code 255
2020-07-15 11:05:07 aborting phase 2 - cleanup resources
2020-07-15 11:05:07 migrate_cancel
2020-07-15 11:05:08 ERROR: migration finished with problems (duration 00:00:01)


This time I got a message stating that the disk size was reverted back to the incorrect value of 2902710697329, and of course this is not a multiple of the block size so the migration fails once again.

Does anyone have any suggestions as to what I should try next?

I am using pve-manager/6.1-8/806edfe1 (running kernel: 5.3.18-3-pve)

Best regards,
Nuno
 
what kind of storage is the disk on?
 
could you post the output of zfs get all of the zvol?
 
could you post the output of zfs get all of the zvol?

Here it is:
Code:
# zfs get all
NAME               PROPERTY              VALUE                  SOURCE
hdd                type                  filesystem             -
hdd                creation              Mon Apr 13  8:20 2020  -
hdd                used                  2.72T                  -
hdd                available             807G                   -
hdd                referenced            24K                    -
hdd                compressratio         1.00x                  -
hdd                mounted               yes                    -
hdd                quota                 none                   default
hdd                reservation           none                   default
hdd                recordsize            128K                   default
hdd                mountpoint            /hdd                   default
hdd                sharenfs              off                    default
hdd                checksum              on                     default
hdd                compression           off                    default
hdd                atime                 on                     default
hdd                devices               on                     default
hdd                exec                  on                     default
hdd                setuid                on                     default
hdd                readonly              off                    default
hdd                zoned                 off                    default
hdd                snapdir               hidden                 default
hdd                aclinherit            restricted             default
hdd                createtxg             1                      -
hdd                canmount              on                     default
hdd                xattr                 on                     default
hdd                copies                1                      default
hdd                version               5                      -
hdd                utf8only              off                    -
hdd                normalization         none                   -
hdd                casesensitivity       sensitive              -
hdd                vscan                 off                    default
hdd                nbmand                off                    default
hdd                sharesmb              off                    default
hdd                refquota              none                   default
hdd                refreservation        none                   default
hdd                guid                  3157052319888068953    -
hdd                primarycache          all                    default
hdd                secondarycache        all                    default
hdd                usedbysnapshots       0B                     -
hdd                usedbydataset         24K                    -
hdd                usedbychildren        2.72T                  -
hdd                usedbyrefreservation  0B                     -
hdd                logbias               latency                default
hdd                objsetid              54                     -
hdd                dedup                 off                    default
hdd                mlslabel              none                   default
hdd                sync                  standard               default
hdd                dnodesize             legacy                 default
hdd                refcompressratio      1.00x                  -
hdd                written               24K                    -
hdd                logicalused           2.65T                  -
hdd                logicalreferenced     12K                    -
hdd                volmode               default                default
hdd                filesystem_limit      none                   default
hdd                snapshot_limit        none                   default
hdd                filesystem_count      none                   default
hdd                snapshot_count        none                   default
hdd                snapdev               hidden                 default
hdd                acltype               off                    default
hdd                context               none                   default
hdd                fscontext             none                   default
hdd                defcontext            none                   default
hdd                rootcontext           none                   default
hdd                relatime              off                    default
hdd                redundant_metadata    all                    default
hdd                overlay               off                    default
hdd                encryption            off                    default
hdd                keylocation           none                   default
hdd                keyformat             none                   default
hdd                pbkdf2iters           0                      default
hdd                special_small_blocks  0                      default
hdd/vm-101-disk-0  type                  volume                 -
hdd/vm-101-disk-0  creation              Sat Jul 11 11:26 2020  -
hdd/vm-101-disk-0  used                  2.72T                  -
hdd/vm-101-disk-0  available             865G                   -
hdd/vm-101-disk-0  referenced            2.67T                  -
hdd/vm-101-disk-0  compressratio         1.00x                  -
hdd/vm-101-disk-0  reservation           none                   default
hdd/vm-101-disk-0  volsize               2.64T                  local
hdd/vm-101-disk-0  volblocksize          8K                     default
hdd/vm-101-disk-0  checksum              on                     default
hdd/vm-101-disk-0  compression           off                    default
hdd/vm-101-disk-0  readonly              off                    default
hdd/vm-101-disk-0  createtxg             1485370                -
hdd/vm-101-disk-0  copies                1                      default
hdd/vm-101-disk-0  refreservation        2.72T                  local
hdd/vm-101-disk-0  guid                  2100283438499972043    -
hdd/vm-101-disk-0  primarycache          all                    default
hdd/vm-101-disk-0  secondarycache        all                    default
hdd/vm-101-disk-0  usedbysnapshots       0B                     -
hdd/vm-101-disk-0  usedbydataset         2.67T                  -
hdd/vm-101-disk-0  usedbychildren        0B                     -
hdd/vm-101-disk-0  usedbyrefreservation  57.9G                  -
hdd/vm-101-disk-0  logbias               latency                default
hdd/vm-101-disk-0  objsetid              2179                   -
hdd/vm-101-disk-0  dedup                 off                    default
hdd/vm-101-disk-0  mlslabel              none                   default
hdd/vm-101-disk-0  sync                  standard               default
hdd/vm-101-disk-0  refcompressratio      1.00x                  -
hdd/vm-101-disk-0  written               2.67T                  -
hdd/vm-101-disk-0  logicalused           2.65T                  -
hdd/vm-101-disk-0  logicalreferenced     2.65T                  -
hdd/vm-101-disk-0  volmode               default                default
hdd/vm-101-disk-0  snapshot_limit        none                   default
hdd/vm-101-disk-0  snapshot_count        none                   default
hdd/vm-101-disk-0  snapdev               hidden                 default
hdd/vm-101-disk-0  context               none                   default
hdd/vm-101-disk-0  fscontext             none                   default
hdd/vm-101-disk-0  defcontext            none                   default
hdd/vm-101-disk-0  rootcontext           none                   default
hdd/vm-101-disk-0  redundant_metadata    all                    default
hdd/vm-101-disk-0  encryption            off                    default
hdd/vm-101-disk-0  keylocation           none                   default
hdd/vm-101-disk-0  keyformat             none                   default
hdd/vm-101-disk-0  pbkdf2iters           0                      default
 
That did the trick - thank you Fabian.

Just one note: the ticket mentions the patch was made available since version 6.1-7, but we were on version 6.1-8 and it was still happening. Upgraded to 6.2-10 and it worked
 
  • Like
Reactions: fabian
pve-storage 6.1-7, not pve-manager ;)
 
Right, I incorrectly assumed the versions were aligned. I take it back :) Thanks again
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!