Not sure if this is a bug, setting, or limitation. I'm on the latest version of proxmox, with the latest version of the virtio drivers installed, I have a windows 10 vm with a 1TB hard disk attached. I have it set as an SSD with trim enabled. The issues arises when windows actually tries to trim the drive. It requires 24GB of memory on the VM before it will complete any less and it will throw an out of memory exception. Watching task manager it will use about 21gigs of the 24 while it's trimming. I initialy had a 6TB disk attached to the vm and that requires 64 gigs of memory iirc. Anyone know what is happening here? Doesn't seem to affect my nix VM's.
Windows VM requires lots of ram to trim larger hard disks
- Thread starter whatishappening
- Start date
hi,
that's weird, can you post the VM configuration?
which version of
that's weird, can you post the VM configuration?
is it the
Secure file delete process using that much memory or something else?which version of
sdelete are you using? on my system it never goes beyond a couple of megabytes of RAM while trimming a terabyte virtual disk.Thanks for the reply attaching a screenshot of the config.
I'm not sure what you are referencing with the
Secure file delete vs sdelete, that might be something I'm not aware of. Is that a setting in proxmox or in the VM OS?Attachments
how are you trimming the windows VM? normally
sdelete tool from windows sysinternals is used for zeroing the virtual disk inside the VM, so that's what i was referring to.which process is using up the memory according to windows task manager?
oh i see. i was just letting windows defrag do the optimization/trim that's what is chewing all the memory and erroring out. See error below.
Code:
The storage optimizer couldn't complete slab consolidation on Storage (D:) because: Not enough memory resources are available to complete this operation. (0x8007000E)try installing sdelete from the sysinternals page [0] and do
[0]: https://docs.microsoft.com/en-us/sysinternals/downloads/sdelete
sdelete -z C: (or a different drive letter if that's the case). that will zero out the space and perform a trim.[0]: https://docs.microsoft.com/en-us/sysinternals/downloads/sdelete
That doesn't look like a tool that would trim to reclaim free space for a thin provisioned disk, that looks like a secure way to erase free space to prevent recovery. While a nice tool that I wasn't aware of, I don't think that will accomplish the same thing as disk optimizations trim function. The disk I'm trying to trim fluctuates a bit over time and I like to reclaim that space if possible.
it can do both
Code:
SDelete v2.04 - Secure file delete
Copyright (C) 1999-2019 Mark Russinovich
Sysinternals - www.sysinternals.com
usage: sdelete [-p passes] [-r] [-s] [-q] <file or directory> [...]
sdelete [-p passes] [-z|-c [percent free]] <drive letter [...]>
sdelete [-p passes] [-z|-c] <physical disk number>
-c Clean free space. Specify an option amount of space
to leave free for use by a running system.
-f Force arguments containing only letters to be treated as a file/directory rather than a disk.
Not required if the argument contains other characters (path separators or file extensions for example)
-p Specifies number of overwrite passes (default is 1)
-r Remove Read-Only attribute
-s Recurse subdirectories
-z Zero free space (good for virtual disk optimization)
-nobanner Do not display the startup banner and copyright message.
Disks must not have any volumes in order to be cleaned.see the
-z option. it's also mentioned in our wiki page about shrinking qcow2 disks, which includes trimming the disk in preparation [0]please try and let me know if it helps
[0]: https://pve.proxmox.com/wiki/Shrink_Qcow2_Disk_Files#Windows_Guest_Preparation
Last edited:
oh, I see. Thanks for the tip. I'll give that a try, might set it up as a scheduled task if it works well.it can do both
Code:SDelete v2.04 - Secure file delete Copyright (C) 1999-2019 Mark Russinovich Sysinternals - www.sysinternals.com usage: sdelete [-p passes] [-r] [-s] [-q] <file or directory> [...] sdelete [-p passes] [-z|-c [percent free]] <drive letter [...]> sdelete [-p passes] [-z|-c] <physical disk number> -c Clean free space. Specify an option amount of space to leave free for use by a running system. -f Force arguments containing only letters to be treated as a file/directory rather than a disk. Not required if the argument contains other characters (path separators or file extensions for example) -p Specifies number of overwrite passes (default is 1) -r Remove Read-Only attribute -s Recurse subdirectories -z Zero free space (good for virtual disk optimization) -nobanner Do not display the startup banner and copyright message. Disks must not have any volumes in order to be cleaned.
see the-zoption. it's also mentioned in our wiki page about shrinking qcow2 disks, which includes trimming the disk in preparation [0]
please try and let me know if it helps
[0]: https://pve.proxmox.com/wiki/Shrink_Qcow2_Disk_Files#Windows_Guest_Preparation
talking about defragmenting, i was able to reproduce what you described in your first post. the disk defragmenter indeed uses a lot of memory when analyzing/optimizing the disks (but not as much as your case, for me it's around 8-10G). though this seems to be a windows specific problem and not related to proxmox in general. it could help with the memory usage during the defrag if you partition your disk into smaller parts. but at the moment i'm afraid the only "fix" would be adding more RAM to the VM and turning off other programs/services that use up memory while the defrag is going on.
Last edited:
I'm glad you were able to reproduce it that makes me feel a little better. I have a quite a bit of data on that disk (bit over half full) so that might also require more memory for defrag to process. It must be an interaction with the proxmox virtual disk driver and windows defrager, as I have physical windows 10 clients that do not experience this memory utilization when trimming a large 1TB ssd. In the end it isn't a huge issue I suppose.
Apologies if this has been posted elsewhere, I could not find it. Describes the issue in detail and how to fix: https://www.alldiscoveries.com/how-...frag-on-windows-server-virtualized-with-qemu/
Hi all, sorry to necropost, but this thread often hits the top of search engines when searching for solutions to windows' defrag eating all your resources.
The above link to "alldiscoveries.com" is correct however I have run into the following problems:
The instruction to set the discard_granularity value to 2MBytes is not in alignment with Hyper-V which (in my deployment tests at least) sets the value to 32MBytes. Microsoft advice seems to be that "slab consolidation" on drives with <8MByte slab sizes is not optimal. Setting to 32MBytes is therefore seemingly the better choice:
.. which for my deployment reduced optimize times on an 80GB disk down from 6 minutes to 20 seconds
The guidance to set the discard_granularity option for an entire storage controller has no effect on PVE 9 with virtio disks. This:
.. does not work, however:
.. setting per-device does work
I can't find any documentation on the global option.. love to hear if anyone else can shine a light on it
The above link to "alldiscoveries.com" is correct however I have run into the following problems:
The instruction to set the discard_granularity value to 2MBytes is not in alignment with Hyper-V which (in my deployment tests at least) sets the value to 32MBytes. Microsoft advice seems to be that "slab consolidation" on drives with <8MByte slab sizes is not optimal. Setting to 32MBytes is therefore seemingly the better choice:
Code:
discard_granularity=33554432The guidance to set the discard_granularity option for an entire storage controller has no effect on PVE 9 with virtio disks. This:
Code:
args: -global scsi-hd.discard_granularity=33554432
Code:
args: -set device.virtio0.discard_granularity=33554432I can't find any documentation on the global option.. love to hear if anyone else can shine a light on it
Hello,
Out of curiosity, do you have any source for this? I have seen in the past a couple of cases where increasing the size of the discard granularity had a positive impact on trimming times, but in those instances we recommended much smaller, conservative, increments.
Out of curiosity, do you have any source for this? I have seen in the past a couple of cases where increasing the size of the discard granularity had a positive impact on trimming times, but in those instances we recommended much smaller, conservative, increments.
Last edited:
The source seems to be https://support.microsoft.com/en-us...ned-luns-b6f87710-272e-430a-f194-9a74a697133f
According to the source, this should be fixed with patches dating to August 2014. So it might be a similar, but ultimately different issue (unless one uses a non-updated windows 2012 version).
It's not trimming times alone - it's that the defrag service in Windows acts like it has a memory leak if these slabs are too small, and the guest effectively crawls to a halt if you have a sizeable disk. I find this not to be an issue on WS2019 (v1809) but on WS2022 and above it is immediately obvious. Assume clients of similar vintage are the same. Windows automates a weekly optimize task once a week out of the box
In testing, an 80GB disk caused it to take about 5-6 minutes and eat a few GB of RAM and a bunch of IO - i assume moving individual 4k blocks around.. I stress I only found this when investigating.. it's not something you'll notice unless you're watching graphs. Another with a 1TB disk ate 32GB of allocated guest RAM before hitting its page file - that machine went into a loop - unable to complete the scheduled volume optimize task, crashing, being manually rebooted, and back around within a few hours.
Changing the slab size solved this problem. Experience suggests to me that reducing the slab count on a 1TB disk from over 268 million (at the default discard_granularity=4k) to 32,000 (by increasing discard_granularity=32MB) helps that avoid windows' defrag service from going off the rails
I followed up to verify what microsoft is doing with Hyper-V - it is obviously a completely different underlying technology but with an "out of the box" config Hyper-V host (Server 2019), the value passed through to a guest VM is a 32MB slab size. Conclusive evidence? no, but in lieu of much else...
I agree, it could be something unrelated - other threads on this topic have considered it relevant. Some suggest this was an out-of-band patch and didn't make it to later versions of Windows. This problem is definitely a long-standing one and not just for Proxmox.
The "solution" post from "alldiscoveries" says 2MB slab sizes and my findings say 32MB. Microsoft appear, maybe, at some point, possibly to have said no less than 8MB, therefore 32MB seems the more sensible option. My intent here was just to point that out