"Move Disk" ZVOL to other zpool, only allocated contents?

[fluidmindorg]

I'm about to replace the HBA card used for my hdd_pool, so I want to temporarily move the ZVOL virtual drives of my VMs from the hdd_pool to the ssd_pool, just to be safe.

I tested with one of the smaller virtual drives that's set to look like a 32G drive to the guest OS and has 7.52G actually stored in it. Before I executed the "Move Disk" operation, the ZVOL showed in "zfs list" as 7.52G used:

NAME                           USED  AVAIL     REFER  MOUNTPOINT
hdd_pool                       286G  10.5T      112K  /hdd_pool
hdd_pool/vm-100-disk-1        7.52G  10.5T     7.52G  -
...
ssd_pool                      90.6G   340G      104K  /ssd_pool

But after the move completed, it shows as 32G in ssd_pool:

NAME USED AVAIL REFER MOUNTPOINT hdd_pool 278G 10.5T 112K /hdd_pool ... ssd_pool 123G 307G 104K /ssd_pool ssd_pool/vm-100-disk-1 32.1G 307G 32.1G -

As you can see, the ssd_pool size shows has going from 90.6G to 123G.
However, in the Proxmox list of ZFS pools, it shows the allocation in the pool as going from 90.7G to 97.24G.

Questions:

1. Why am I seeing this discrepancy?
2. Why didn't the Proxmox Move Disk operation only copy the allocated contents of the ZVOL rather than the entire size of the ZVOL?
3. Most importantly, what is a better way to move a ZVOL (with only its allocated contents) from one pool to another?

Thanks!
--Dan

 

[Dunuin]

Did you check the "Thin provisioning" checkbox of your storage that is pointing to your ssd_pool ZFS pool?
Is any of the ZFS pool using a raidz1/2/3, because then zvols might be bigger because of padding overhead when using a too small volblocksize.
Easiest to move virtual disks between pools would be the "Move Disk" button of the webUI. If you want to move datasets/zvols that re not managed by PVE you could work with the "zfs send" and "zfs recv" commands in CLI.

 

[fluidmindorg]

Thanks for your quick response, @Dunuin . I appreciate it.

Yes, I just double checked. "Thin provisioning" is checked on both pools.
Both pools are comprised of mirrored pairs, no raidz of any kind.

The odd thing is that "zfs list" shows it as taking up all 32G, but the pool allocation correctly went up only 7.5G.

 

[fiona]

Hi,

1. Why am I seeing this discrepancy?
2. Why didn't the Proxmox Move Disk operation only copy the allocated contents of the ZVOL rather than the entire size of the ZVOL?

moving disks is implemented is in a storage-agnostic way using qemu-img convert or QEMU's drive-mirror (when the VM is running). It doesn't know about ZFS's allocation. You can try enabling discard for the VM's disks and running fstrim in the VM to see if you can get the space back.

1. Most importantly, what is a better way to move a ZVOL (with only its allocated contents) from one pool to another?

As @Dunuin already mentioned, you could try using zfs send and zfs recv directly.

 

[fluidmindorg]

Thanks, @fiona . That explains it. That's highly unfortunate for everyone using ZVOLs for their virtual drives.

Yes, discard was on, and fstrim did, in fact, reduce the size of the moved copy of the ZVOL back down to ~7.5G. However, I have a 300G virtual drive that has around 150G or so actually stored in it, and I can't use the Move Disk command to move it to a pool that has around 250G of free space.

I'll see if I can figure out how to use zfs send and zfs recv.

Thanks!
--Dan

 

[fluidmindorg]

Well, poop! I just saw the ArsTechnica 2020 exposé about the WD Red SMR drives and realized that's the model of the six 4TB drives I got for this server...back in 2020!

So, I guess I'll just move the ZVOLs off of this ZFS array, reformat it as an MDRAID instead, then move the virtual disks back.

Do I understand correctly that, when I move them back, I'll be able to specify either QCOW2 or LVM-Thin and the Proxmox "Move Disk" command will convert the ZVOL as it moves it?

 

[_gabriel]

yes as fiona said