[SOLVED] Shrink disk on Ceph

[comfreak]

Hi,

how do I shrink a disk on a Ceph-Cluster (the workflow to reduce the partition beforehand is clear)?

I tried

root@server:~# qm resize 111 scsi1 40G
shrinking disks is not supported

but failed.

Yes, the "qm" manual says that shrinking is not supported but here it sounds as it is supported. Or is "qm" the PVE API mentioned over there? So what would be the manual way instead?

If it is qemu-img, I probably have to mount the Ceph pool manually to access the disk? How is the mount done?

Thanks in advance for help!

Edit: As I unterstand "qm" is not the PVE API mentioned in the wiki as it stands for the "Qemu/KVM Virtual Machine Manager". The PVE API is probably the WebUI (which does not support a shrink, yes) or the REST API.

 

[comfreak]

Ok, got it.

I can access the Ceph RBD directly through qemu-img. Great!

- Shrink partition inside the VM (Windows, gparted, etc.)
- Shutdown VM
- Resize the image: qemu-img resize -f raw --shrink rbd:<pool>/<disk-img> <size> (example: qemu-img resize -f raw --shrink rbd:VMs/vm-111-disk-1 55G)
- Scan the disk: qm rescan --vmid <vmid> (example: qm rescan --vmid 111)
- Start VM and hope that nothing got lost

 

[Superfish1000]

I spent like 10 minutes trying to figure out why this didn't work before realizing I had a typo and put "rdb" instead of "rbd."
I feel dumb. lol
Thank you for posting this. Super helpful.

 

[beathorst]

I cannot boot my VM (ubuntu-test) after shrink it, here is what I did

I test a shrink from 50 GiB -> 30 GiB

$ qemu-img info rbd:ceph-ssd/vm-110-disk-0
image: json:{"driver": "raw", "file": {"pool": "ceph-ssd", "image": "vm-110-disk-0", "driver": "rbd", "namespace": ""}}
file format: raw
virtual size: 50 GiB (53687091200 bytes)
disk size: unavailable
cluster_size: 4194304

Boot gparted and resize the partion /dev/sda1 to 28GB to have some puffer

qemu-img resize --shrink -f raw rbd:ceph-ssd/vm-110-disk-0 30G
qm rescan --vmid 110

Reboot the VM and it stucks

 

[dpeddi]

I cannot boot my VM (ubuntu-test) after shrink it, here is what I did

I test a shrink from 50 GiB -> 30 GiB



Boot gparted and resize the partion /dev/sda1 to 28GB to have some puffer

View attachment 58143



Reboot the VM and it stucks

View attachment 58139

Sorry for late answer..

If your partition table is gpt, doing the qemu-img resize -f raw --shrink rbd:<pool>/<disk-img> <size> you will drop it (usually gpt partition table stay at the end of the disk).
So you need to rebuild it from scratch.

 

[aasami]

So you need to rebuild it from scratch.

How?

 

[comfreak]

You normally don't need to rebuild the GPT partition table completely from scratch.

GPT keeps a primary copy at the beginning of the disk and a backup copy at the end. When shrinking the RBD with qemu-img resize --shrink, the backup GPT at the old end of the disk gets cut off, but the primary GPT should still be intact.

So, provided that all partitions were shrunk beforehand and really fit within the new disk size, you can usually just recreate/move the backup GPT to the new end of the disk.

For example, boot a Linux rescue system / GParted Live and first check the disk:

sgdisk -v /dev/sda
sgdisk -p /dev/sda

If the primary GPT is still valid and only the backup GPT is missing or located beyond the new end of the disk, this should fix it:

sgdisk -e /dev/sda
sgdisk -v /dev/sda
sgdisk -e moves/recreates the backup GPT structures at the actual end of the disk.

So the complete procedure would basically be:

1. Shrink filesystem/partition inside the VM, leaving some margin.
2. Shut down the VM.
3. Shrink the RBD, e.g.:

qemu-img resize -f raw --shrink rbd:<pool>/<disk-img> 30G
qm rescan --vmid <vmid>

1. Boot a rescue system.
2. Run sgdisk -e /dev/sda.
3. Verify with sgdisk -v /dev/sda.
4. Boot the VM normally.

Of course, if a partition still extended beyond the new RBD size when doing the shrink, actual partition/filesystem data may already have been truncated. In that case fixing the GPT alone won't help.