Thin Provisioned Windows-VMs use full capacity in Proxmox Ceph after migration

jsterr

jsterr

Well-Known Member
Jul 24, 2020
678
159
53
32
Is there a way to make them use just the space they are really using? I read alot about trim and discard, but it seems like theres not a easy way-
Any tips? Im trying it myself currently, but yeah if there are any experience you wanna share, I would really appreciate it.

Example: VMware-VM 40GB Thinprovisioned (Used: 16GB)
Example: Imported via OFV-Tool 40GB in qm config AND in OS.

Any prerequisites for this, specific drivers?
 
Last edited:
Thanks! I just installed QEMU Guest Agent and moved disk from one ceph pool to another, but vm is still using all disk space. Your screenshotsays disk move or vm migration, my proxmox 6.3 says on disk clone. Did it change here?

Clone disk how to do that? I only know clone-vm, might work too?

qemu.png

qemu-agent-diskclone.png

Code: 
root@pve-02:~# qm config 103
agent: 1,fstrim_cloned_disks=1
bios: ovmf
boot: order=sata0
cores: 8
memory: 16384
name: JS-PRTG-01
net0: virtio=1E:3A:79:45:03:1F,bridge=vmbr0,firewall=1
sata0: vm_hdd:vm-103-disk-0,discard=on,size=40G,ssd=1
scsihw: virtio-scsi-pci
smbios1: uuid=054f705e-3948-47e3-86b0-7f20c24ba483
vmgenid: bfa2e7c8-b5be-43ce-a71d-fc83e0dbe13e
 
Last edited:
jsterr said:
Your screenshotsays disk move or vm migration, my proxmox 6.3 says on disk clone. Did it change here?
Click to expand...
I'm just running a newer development installation where the text got changed to reflect that it's done for both, clone and migration.

spirit said:
I don't known if the widows guest agent is able to do the trim like the linux guest agent.
Click to expand...
They have at least code in there (just looked), seems like it should work for Windows 8 and newer by calling defrag.exe with respective arguments.

jsterr said:
Clone disk how to do that? I only know clone-vm, might work too?
Click to expand...
That's for a full VM clone or moving the disk to another storage, I do not think you need nor want to do that.


But, you can always trigger it manually, just replace VMID in the following command and run as root on the Proxmox VE host the VM is on (can be done over the webinterface, Node -> Shell):

Bash: 
qm guest cmd VMID fstrim

If that does not work then yeah, try to do it from the inside with native windows tools.
 
  • Like
Reactions: jsterr and spirit
Thanks for all the help. It seems Im missing something, that causes trim not to work.
  1. QEMU Guest Agent installed and activated
  2. VM has discard enabled
  3. qm guest cmd VMID fstrim -> timeout
  4. qm agent 103 fstrim -> timeout
  5. qm agent is working (tried get-host-name) so qemu-guest agent communication works
  6. I use Windows Server 2016 (trim enabled) fsutil behavior query DisableDeleteNotify -> 0)
  7. Defrag.exe /L c: (not freeing up space)
  8. optimize-Volume -DriveLetter C -ReTrim -Verbose (not freeing up space)
  9. I cloned the vm completely for the test, didnot free up space (maybe because timeout of fstrim?)

Just to make sure. THIS should work on Proxmox with Ceph and RBD right? Any specific hard-disk-drivers needed? Thats my current vm-config

Code: 
root@pve-02:~# qm config 103
agent: 1,fstrim_cloned_disks=1
bios: ovmf
boot: order=sata0
cores: 8
memory: 16384
name: JS-PRTG-01
net0: virtio=1E:3A:79:45:03:1F,bridge=vmbr0,firewall=1
sata0: vm_hdd:vm-103-disk-0,discard=on,size=40G,ssd=1
scsihw: virtio-scsi-pci
smbios1: uuid=054f705e-3948-47e3-86b0-7f20c24ba483
vmgenid: bfa2e7c8-b5be-43ce-a71d-fc83e0dbe13e
 
Alwin Antreich said:
Does Windows recognize the drive as SSD? Otherwise it will ignore trim requests.
Click to expand...

I have ssd-emulation = on. Im not really sure howto check if its detected as a ssd though. Windows says QEMU HARD DISK DRIVE. Hardware-Information does not have any specific field where it determines if its a ssd or hdd.

Maybe it has something to do that I imported the disks with qm import after downloading them via ofv-tool. virtio-scsi drivers are installed, disk = virtioscsi. Windows still says. I read something about qemu “Detect zeroes" - is this something that should be activated?


c-trim.png
 
Last edited:
I put my test-vm on a local-zfs now, there the fstrim commands are working (no error after execution) but windows vm still NOT shrinking. I also tried defrag.exe /f c: and optimzing the hard disk and Optimize-Volume -DriveLetter C -ReTrim -Verbose.

does this work with only specific qemu img formats? Im out of ideas now.
 
Last edited:
It may be that trim is deactivated in Windows, us the fsutil behavior query DisableDeleteNotify to check and fsutil behavior set DisableDeleteNotify 0 to enable trim if it is disabled. Then you can run Optimize-Volume -DriveLetter C -ReTrim -Verbose again.
 
Alwin Antreich said:
It may be that trim is deactivated in Windows, us the fsutil behavior query DisableDeleteNotify to check and fsutil behavior set DisableDeleteNotify 0 to enable trim if it is disabled. Then you can run Optimize-Volume -DriveLetter C -ReTrim -Verbose again.
Click to expand...

Both enabled and tried as you can see here:

  1. I use Windows Server 2016 (trim enabled) fsutil behavior query DisableDeleteNotify -> 0)
  2. Defrag.exe /L c: (not freeing up space)
  3. optimize-Volume -DriveLetter C -ReTrim -Verbose (not freeing up space)
Click to expand...
 
Alwin Antreich said:
Now, that I look again at the picture. The disk inside the VM shows already ~39 GB used. There is nothing to free on the underlying storage if it's actually used by the VM.
Click to expand...

The question here is: why is it used by the vm after importing it via ofv tool? On previous hypervisor (vSAN) there are 16GB free of space avaiable inside the windows-VM. I tried it the same way with linux vms and they are working correctly (not more space is used as before) and rbd is also correct size (thin).
 
Last edited:
TRIM can only issue a discard of blocks that are no longer used by the OS. So whatever filled the disks is stored in Windows. VMware uses their own guest-tools, they may have remapped the page file to a separate disk (assuming, since its the amount of memory the VM has).
 
  • Like
Reactions: jsterr
Alwin Antreich said:
TRIM can only issue a discard of blocks that are no longer used by the OS. So whatever filled the disks is stored in Windows. VMware uses their own guest-tools, they may have remapped the page file to a separate disk (assuming, since its the amount of memory the VM has).
Click to expand...

Thanks! Sorry for the confusion thats it! Tried it with another VM and its working! Seems like it takes some time to trim and vm is unsuable until its finished. Thanks everybody for helping!

Code: 
root@pve-01:~# rbd du vm_hdd/vm-105-disk-0
NAME          PROVISIONED USED  
vm-105-disk-0     200 GiB 200 GiB

root@pve-01:~# rbd du vm_hdd/vm-105-disk-0
NAME          PROVISIONED USED  
vm-105-disk-0     200 GiB 199 GiB

root@pve-01:~# rbd du vm_hdd/vm-105-disk-0
NAME          PROVISIONED USED  
vm-105-disk-0     200 GiB 198 GiB

root@pve-01:~# rbd du vm_hdd/vm-105-disk-0
NAME          PROVISIONED USED  
vm-105-disk-0     200 GiB 197 GiB
 
  • Like
Reactions: Alwin Antreich
So it got better but I do not get all free space back, thats the poin where it wont go down further.

  • Disk is 200GB
  • free space in Windows: 168GB.
  • Used: 31GB


    1613113930959.png
Code: 
root@pve-01:~# rbd du vm_hdd/vm-105-disk-0
NAME          PROVISIONED USED   
vm-105-disk-0     200 GiB 128 GiB

So it still uses 4 times the space it should use, I used Optimize-Volume -DriveLetter C -ReTrim -Verbose and optimize function of windows multiple times but space wont go down any further?
 
jsterr said:
So it still uses 4 times the space it should use, I used Optimize-Volume -DriveLetter C -ReTrim -Verbose and optimize function of windows multiple times but space wont go down any further?
Click to expand...
Depending on the alignment of the blocks, Ceph may need to split the data over more objects. You may try the rbd sparsify, this will also remove zeroed extends. Ceph probably tries to minimize read & write IO if the object is not requested.
 
Alwin Antreich said:
Depending on the alignment of the blocks, Ceph may need to split the data over more objects. You may try the rbd sparsify, this will also remove zeroed extends. Ceph probably tries to minimize read & write IO if the object is not requested.
Click to expand...

Thanks! That did not change anything:

root@pve-01:~# rbd sparsify --image vm_hdd/vm-105-disk-0
Image sparsify: 100% complete...done.
root@pve-01:~# rbd du vm_hdd/vm-105-disk-0
NAME PROVISIONED USED
vm-105-disk-0 200 GiB 128 GiB

Optimize-Volume -DriveLetter C -ReTrim -Verbose was used again but still 128GB used (see above)

1613575925788.png
 
Last edited:
Afaik sata doesn't support trim.

You have to use virtio for trim to work.

Download virtio drivers here: https://fedorapeople.org/groups/virt/virtio-win/direct-downloads/stable-virtio/virtio-win.iso

Install the vioscsi driver, after that power down the vm and change the config from "sata0" to "scsi0" in /etc/pve/qemu-server/105.conf

Then power up again and download sdelete https://docs.microsoft.com/en-us/sysinternals/downloads/sdelete

Run "sdelete.exe -z" in admin cmd. Wait for 20 minutes and check used space on the hypervisor.
 
  • Like
Reactions: jsterr
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom
Back