Confused about raw image on zfs, qemu-img won't work?

[MrPete]

I am working on a Win10 image. It's too big; want to shrink it. I reduced the partition sizes so I have under 52GB of partitions in a 500GB raw virtual disk.

Config info below.

The hard part: the image (in my local-zfs storage) is actually apparently a set of links to /dev/zd48* which are device files. qemu-img complains that it will not function on device files. I'm confused! Where is the actual file? Have I done something wrong? What's going on here?

in 711.conf: scsi0: local-zfs:vm-711-disk-0,cache=writeback,discard=on,size=500G,ssd=1

ls -lh /dev/zvol/rpool/data/*711*

lrwxrwxrwx 1 root root 13 Mar 14 20:40 vm-711-disk-0 -> ../../../zd48
lrwxrwxrwx 1 root root 15 Mar 14 20:40 vm-711-disk-0-part1 -> ../../../zd48p1
lrwxrwxrwx 1 root root 15 Mar 14 20:40 vm-711-disk-0-part2 -> ../../../zd48p2
lrwxrwxrwx 1 root root 15 Mar 15 05:00 vm-711-disk-0-part3 -> ../../../zd48p3

lsblk /dev/zd48

NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS
zd48 230:48 0 500G 0 disk
├─zd48p1 230:49 0 50M 0 part
├─zd48p2 230:50 0 50G 0 part
└─zd48p3 230:51 0 556M 0 part

qemu-img info vm-711-disk-0

image: vm-711-disk-0
file format: raw
virtual size: 500 GiB (536870912000 bytes)
disk size: 0 B
Child node '/file':
filename: vm-711-disk-0
protocol type: host_device
file length: 500 GiB (536870912000 bytes)
disk size: 0 B

qemu-img resize --shrink -f raw vm-711-disk-0 448G

qemu-img: Cannot resize device files

 

[bbgeek17]

In simplistic terms, ZFS can be a filesystem that stores files as most people understand them: txt, doc, jpeg, qcow.
ZFS can also contain ZVOLs, which are block devices. This is what you have now.
As you can guess the "qemu-img" is designed to work with qemu format, ie qcow, not ZVOL block devices. You need to research ZFS capabilities in terms of shrinking ZVOLs.

Good luck

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[MrPete]

In simplistic terms, ZFS can be a filesystem that stores files as most people understand them: txt, doc, jpeg, qcow.
ZFS can also contain ZVOLs, which are block devices. This is what you have now.

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

Questions:

• How do I avoid creating a ZVOL when setting up a VM??? Perhaps by putting images in local instead of local-zfs?
• Is there a way to convert a ZVOL to a normal image file?

 

[bbgeek17]

• How do I avoid creating a ZVOL when setting up a VM??? Perhaps by putting images in local instead of local-zfs?

Place the images into a FILE based storage https://pve.proxmox.com/wiki/Storage#_storage_types

• Is there a way to convert a ZVOL to a normal image file?

Use "move disk" CLI or GUI option to move VM image/disk

You do realize that shrinking any storage is potentially dangerous and can lead to corruption within a VM?

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[LnxBil]

Is there a way to convert a ZVOL to a normal image file?

Why would you want to do that and technically it is already a linux file, a block file.

 

[Dunuin]

Perhaps by putting images in local instead of local-zfs?

Yes, that would work. But for performance reasons you usually want to work with zvols (so block devices) and not with image files (like qcow2) so not have that overhead of the unneccessary additional filesystem.

 

[MrPete]

Thanks for the thoughts! I've done additional research and for now have solved my issues, and also learned a bunch.

NOTE: ALL of this is applies to GPT (or MBR) Disk/partition management on all typical OS (Linux, MacOS, Windows).

First, a few answers for the curious:
1) Why am I even looking into this?

• It is literally impossible to shrink a block storage VM. Can only be done with file-based.
• Shrinking a disk is a very reasonable thing to do with partitions on a disk: each partition is fully self-contained except for the MBR/GPT data. (Make SURE to preserve about 1MB at the tail end of any disk!!! The secondary GPT table is kept there. It only needs a few dozen disk blocks, but it's more than a little important )
• A few applications of this that I've used over the last 20+ years:
  • Copy a functioning system onto a smaller drive as part of an upgrade
  • Create a minimal-size template
  • Create massive chunks of empty space on storage, for other uses
  • (Note: once you know how to do this well, it really is not a big deal )

2) Isn't it risky (as @bbgeek17 suggests)?

• Given a basic understanding of disk layout, honestly no. Not any more risky than reading and writing data in general.
• The key things that MUST be understood:
  • About a megabyte of space at disk start and end must be preserved for GPT tables and such
  • When you shrink a disk, you (of course) WILL destroy the backup GPT table at the end of the disk. Not an issue if you know what you're doing! See below for a tiny tutorial.

3) Aren't ZVOLs incredibly faster and better? Less overhead etc.

• Not so much. Go look at current benchmarks... yes in some cases they win, a little. Not enough to be a deal breaker.
• In modern systems, IO time is much more significant than CPU time. Thus, data compression can easily provide big advantages over raw disk I/O time.
• Plus, "compressed" qcow2 disks transfer incredibly quickly because all of the compressed-out space is ignored.
• AND, for my purposes, ZVOL has missing functionality that I simply want to have right now: I want a shrinkable VM storage. Move it to a qcow2 and it works. I can always move back to ZVOL if I want.

SOME QUICK HINTS

• Yes, images in "local" can be qcow2. Not in local-zfs. Move it there (or onto shared storage) and voila, done.
• The GUI does NOT permit all possible conversions. But qemu-img allows a ton of flexibility.
• OSS "gdisk" (GPT fdisk) is your friend. Available for Linux, Mac, Win.
  • After using a partition tool to adjust partition sizes and locations...
  • Use gdisk to preserve (and backup) a permanent copy of your GPT Table. This at-risk data comes in VERY handy if that part of a disk is harmed physically, by malware, mistake or otherwise.
  • gdisk easily rebuilds the secondary copy of the GPT table
  • Or any of a number of other functions at that level
  • gdisk also analyzes the condition of the various partition tables and layouts, warning of any issues.

 

[bbgeek17]

Glad you were able to find a solution that works for you.

• It is literally impossible to shrink a block storage VM. Can only be done with file-based.

Thats a very broad statement. We (Blockbridge) can shrink our block storage that is presented to clients (ie LUNs). Obviously the administrator will need to adjust filesystems and partitions first.

If it is software - anything is possible. Whether it makes sense to do and invest in, thats a different question.

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[Dunuin]

It is literally impossible to shrink a block storage VM. Can only be done with file-based.

Its possible but may result in data loss, so its not available with the PVE CLI tools ("qm") and only allowed to extend a virtual disk. But if you really want you could still shrink that virtual disk on a block storage manually. And usually when working with block devices there is thin provisioning (LVM-Thin/ZFS/btrfs) so there are not many situations where it is actually needed to shrink virtual disk as a shrunk disk won't consume less space than a unshrunk one. Only point I see in shrinking a virtual disk would be when using a thick-provisioned storage or in case you vdisk is way too big and you can't make use of dirty-bitmapping when doing backups.

• Given a basic understanding of disk layout, honestly no. Not any more risky than reading and writing data in general.
• The key things that MUST be understood:
  • About a megabyte of space at disk start and end must be preserved for GPT tables and such
  • When you shrink a disk, you (of course) WILL destroy the backup GPT table at the end of the disk. Not an issue if you know what you're doing! See below for a tiny tutorial.

Problem is that people are blindly copy-pasting commands they find in some tutorial. Would be really bad to shrink a virtual disk without shrinking its partitions/VGs/LVs/filesystems first. And not the first time that I have seen people ignoring this.

 

[MrPete]

...We (Blockbridge) can shrink our block storage that is presented to clients (ie LUNs).

... if you really want you could still shrink that virtual disk on a block storage manually.

OK, I do agree -- shouldn't have been quite so dogmatic about impossibilities At some point it's possible to bit-twiddle ANYthing LOL!

... there are not many situations where it is actually needed to shrink virtual disk as a shrunk disk won't consume less space than a unshrunk one.

Interestingly, even with thin provisioning, performance does vary. (My suspicion: if transferring using a "dumb" utility, it may actually move 450GB of zeros ).

Problem is that people are blindly copy-pasting commands they find in some tutorial. Would be really bad to shrink a virtual disk without shrinking its partitions/VGs/LVs/filesystems first. And not the first time that I have seen people ignoring this.

Boy Howdy.

I've had to fix some truly awful messes. The worst was when a Tech Expert™ set up a client with an amazing be-all-end-all network storage unit -- it's FAST! It has built in BACKUP! It crawls on its belly like a reptile! SO AMAZING! NOT. It was actually RAID 0. Fifteen years of EVERYTHING gone when drive #1 died...and then both drives died. Of course. And to make it even more painful: the replacement guy for Tech Expert™ saw the problem, had ordered more storage... which arrived the day after the drives died. Long story short, yes I did get their data back. Truly a miracle of god.

ANYway... thanks all! I'm a happy camper again.

FUTURE readers: please do take note. While those with experience munging drives and partitions DO know how to adjust such things, if new to this please don't assume there are no gotchas hiding. At the very least, Do Your Backups.

 

[LnxBil]

Create massive chunks of empty space on storage, for other uses

for my purposes, ZVOL has missing functionality that I simply want to have right now: I want a shrinkable VM storage. Move it to a qcow2 and it works. I can always move back to ZVOL if I want.

You don't have that problem on thin provisioned space like a zvol with proper trimming. That is the whole purpose in virtualized systems. QCOW2 needs two steps, a ZVOL only one.

 

[MrPete]

You don't have that problem on thin provisioned space like a zvol with proper trimming. That is the whole purpose in virtualized systems. QCOW2 needs two steps, a ZVOL only one.

Yet the (easy CLI) tools for shrinking do not work on block devices. ZVOL is a block device.
QCOW2 does have maybe 10% overhead, but the practical benefits outweigh that small cost IMHO. Others have identified a litany of benefits to QCOW2.

Don't get me wrong: in a fully stable mature environment, migrating to ZVOL does give performance benefits. I'm not there yet.

BTW, yesterday I had a brief demonstration of the overhead involved in even thin provisioned space with proper trimming: did a backup of a new Win10 VM that had not yet been fired up. NOTHING yet installed. But the backup didn't know that of course, so it blindly copied 50GB of... nothingness. Amazing how long it took to create a backup that when done was literally 1.2 MB LOL. We're still talking under a minute but it should only have needed a blip (NVMe storage)

 

[LnxBil]

Yet the (easy CLI) tools for shrinking do not work on block devices. ZVOL is a block device.
QCOW2 does have maybe 10% overhead, but the practical benefits outweigh that small cost IMHO. Others have identified a litany of benefits to QCOW2.

First, I was talking about thin provisioning and second I have no idea where you read this, but it's wrong. a zvol can easily be resized:

root@proxmox ~ > zfs create -s -V $(( 1 * 1024 * 1024 )) rpool/test/vol1

root@proxmox ~ > zfs get volsize rpool/test/vol1
NAME             PROPERTY  VALUE    SOURCE
rpool/test/vol1  volsize   1M       local

root@proxmox ~ > zfs set volsize=2M rpool/test/vol1

root@proxmox ~ > zfs get volsize rpool/test/vol1
NAME             PROPERTY  VALUE    SOURCE
rpool/test/vol1  volsize   2M       local

root@proxmox ~ > zfs set volsize=1M rpool/test/vol1

root@proxmox ~ > zfs get volsize rpool/test/vol1
NAME             PROPERTY  VALUE    SOURCE
rpool/test/vol1  volsize   1M       local

 

[MrPete]

First, I was talking about thin provisioning and second I have no idea where you read this, but it's wrong. a zvol can easily be resized...

Where did I hear this? Just following Fine Documentation

Nice:

root@proxmox:~# qm resize 555 scsi0 +5G
root@proxmox:~# grep scsi0: /etc/pve/nodes/pve2a/qemu-server/555.conf
scsi0: local-zfs:vm-555-disk-0,iothread=1,size=13G,ssd=1

Not So Nice:

root@proxmox:~# qm resize 555 scsi0 -5G
Unknown option: 5g
400 unable to parse option

root@proxmox:~# qm resize 555 scsi0 8G
shrinking disks is not supported

Searching... found a recommended way that works generically on VM images, with special support for shrinking. BUT, not on this:

root@proxmox:~# qemu-img resize -f raw --shrink /dev/zvol/rpool/data/vm-555-disk-0 8G
qemu-img: Cannot resize device files
root@proxmox:~# ls -l /dev/zvol/rpool/data/vm-555-disk-0
lrwxrwxrwx 1 root root 13 Mar 17 15:12 /dev/zvol/rpool/data/vm-555-disk-0 -> ../../../zd32

Yet you are 100% correct:

root@proxmox:~# zfs get volsize rpool/data/vm-555-disk-0
NAME                      PROPERTY  VALUE    SOURCE
rpool/data/vm-555-disk-0  volsize   13G      local

root@proxmox:~# zfs set volsize=8G rpool/data/vm-555-disk-0
root@proxmox:~# zfs get volsize rpool/data/vm-555-disk-0
NAME                      PROPERTY  VALUE    SOURCE
rpool/data/vm-555-disk-0  volsize   8G       local

root@proxmox:~# grep scsi0: /etc/pve/nodes/pve2a/qemu-server/555.conf
scsi0: local-zfs:vm-555-disk-0,iothread=1,size=8G,ssd=1

root@proxmox:~# qemu-img info /dev/zvol/rpool/data/vm-555-disk-0
image: /dev/zvol/rpool/data/vm-555-disk-0
file format: raw
virtual size: 8 GiB (8589934592 bytes)
disk size: 0 B
Child node '/file':
    filename: /dev/zvol/rpool/data/vm-555-disk-0
    protocol type: host_device
    file length: 8 GiB (8589934592 bytes)
    disk size: 0 B

SO:

• Thanks heaps for the zfs set volsize tutorial
• I wonder why neither qm resize nor qemu-image resize can handle this?

 

[Dunuin]

Like already said...wouldn't be a problem to add such a feature to qm resize. We already had this discussion in another thread. They just don't want to, because people will then lose data by accident when using a "qm resize 555 scsi0 -5G" without preparing the virtual disk first, by shrinking partitions and filesystems. As people will think the qm command will do that for them but this would be nothing PVE could do as it all requires administration on the guestOS level.

 

[bbgeek17]

Like already said...wouldn't be a problem to add such a feature to qm resize. We already had this discussion in another thread. They just don't want to, because people will then lose data by accident when using a "qm resize 555 scsi0 -5G" without preparing the virtual disk first, by shrinking partitions and filesystems. As people will think the qm command will do that for them but this would be nothing PVE could do as it all requires administration on the guestOS level.

Essentially the same reason we took reduce option out of Blockbridge GUI and hid it behind an extra special CLI configuration option

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox