Windows guest - slow filesystem reads

[matthei]

Using Proxmox 7.2 with Windows 10 guest.
I noticed that compiling a codebase with about 12000 small files (215 MB total) takes a very long time. Same with for example Subversion-Commit which goes through all files and checks for changes.
After some experimenting, I coded a small program that goes through all files and just reads them. The first pass takes 67 seconds. Second pass takes 0.7 seconds (since the files are cached, probably in windows).

It's a Ryzen 4750, with a 2TB Samsung 980pro NVME.

I don't have a barebone windows machine to test this out directly, but I remember this was usually quite fast, a few seconds.

So there is probably a bottleneck somewhere... any idea what I could reconfigure to try to get these file reads up to speed?
For now I will be using this "Directory Warmer" program to keep reading the files every few minutes and keep it all cached. But I'd like to know if there's a more proper solution.
(I know that there's some performance overhead, but this seems too slow to be normal)

 

[Dunuin]

You are using virtio SCSI nd followed the Win10 best practices when instlling that win10? https://pve.proxmox.com/wiki/Windows_10_guest_best_practices

 

[matthei]

Yes, as far as I remember, but I better re-check. Thanks, I'll reply when I know more.

 

[matthei]

I checked the page and my configuration, but I don't see any issues between the two.
- Disks are created as SCSI, have Writeback cache (although I assume this only makes a difference for writes? The problem is have is with Reads).
- When installing windows, I did the "Load driver" and loaded the RedHat SCSI, and Network adapter and so on.
- QEMU guest agent is enabled, and i see that the IP is displayed in the Proxmox GUI so that means the guest part of it is probably working.

Well, I don't have the balooning device since it didn't work, so I think i uninstalled it.

 

[_gabriel]

lvmthin or zfs datastore ?

 

[matthei]

It's LVM-thin, and the disks are Raw format.

Thanks for both of your help so far.

 

[Matthias.]

Maybe just share the output of qm config VMID so we can see the whole configuration.
If you share that program of yours and how to create some small files, I can try to reproduce it

 

[matthei]

I finally found the time to prepare a test environment. Thanks a lot for following up on this. The DirectoryWarmer is a good solution as long as I'm working on one codebase, but for multiple node-js projects it's a pain. https://www.reddit.com/r/ProgrammerHumor/comments/6s0wov/heaviest_objects_in_the_universe/

Environment setup
230 MB codebase (obfuscated/sanitized). Basically 13k files of various sizes, mostly very small.
https://www.dropbox.com/s/gktglx191gxjtcl/2023-03-06 DummyCodebase-Sanitized.rar?dl=0

Directory warmer:
https://github.com/papler/DirectorWarmer/tree/0.1.0
Binaries (if you trust me enough, although it will run in a sandbox so it doesnt matter much i suppose)
https://github.com/papler/DirectorWarmer/releases/tag/0.1.0

Drop the binary in the main directory of the CodeBase and it will read all files of the directory that the .exe is currently in, and repeat that every 10 minutes. Running it first time will likely take a minute, second time around it's a second (it's probably reading from windows cache). If you leave it overnight, you will notice that eventually Windows un-caches the files and it again takes a while to read it.
As a comparison, Windows on bare metal reads these


Here's my 101.conf:

agent: 1
balloon: 0
bios: ovmf
boot: order=scsi0;net0
cores: 6
cpu: host,hidden=1,flags=+pcid
efidisk0: local-lvm:vm-101-disk-0,size=4M
hostpci0: 0000:10:00,pcie=1,x-vga=1
machine: pc-q35-6.0
memory: 49152
name: win10-primary
net0: virtio=---redacted----,bridge=vmbr0,firewall=1,tag=25
numa: 0
onboot: 1
ostype: win10
scsi0: local-lvm:vm-101-disk-1,cache=writeback,size=140G
scsi2: local-lvm:vm-101-disk-2,size=51208M
scsi4: local-lvm:vm-101-disk-3,cache=writeback,discard=on,size=100G
scsi9: local-lvm:vm-101-disk-4,backup=0,cache=writeback,discard=on,size=64G
scsihw: virtio-scsi-pci
smbios1: uuid=---redacted----
sockets: 1
usb0: host=3-1.1.1,usb3=1
usb1: host=3-1.1.2,usb3=1
usb2: host=3-1.1.3,usb3=1
usb3: host=3-1.1.4,usb3=1
usb4: host=3-1.2,usb3=1
usb5: host=3-1.3,usb3=1
usb6: host=3-1.4,usb3=1
vmgenid: ---redacted----

 

[matthei]

Actually, I just tested this on an older (circa 2018) machine with Windows on bare metal, and the read times come as double that on the Proxmox Windows Guest... so, it might not be an issue with Proxmox, it's just so many files.
But I do remember some operations (just going thru all the files and taking their names, last modification time) taking only a few seconds back then on baremetal, while now they take something like 15-30 as guest. I best do some more tests to isolate the exact operations and compare that to the baremetal windows machine.
(Well, the codebase also grew a bit since then, but only like 10%)

 

[Matthias.]

Maybe windows did cache/scan them without you knowing about it?

I can reproduce your findings on a windows VM.
I used a tiny python script (edited from https://stackoverflow.com/questions/2212643/python-recursive-folder-read/57223193#57223193) instead of your program, but with your dataset (thanks a lot!)

import os,timeit

def read():
    for currentpath, folders, files in os.walk('.'):
        for file in files:     
            a = open(os.path.join(currentpath, file)).readlines()


print(" first time: ", timeit.timeit(read, number=1))
print("second time: ", timeit.timeit(read, number=1))

input()

 first time:  40.53073719999975
second time:  4.097600800000691

(for some reasons, in later tests the first run took 80 seconds)

I'll try to run the same test on bare metal in a bit. But I suspect you're right and that's just windows being slow with small files.
Another thing you could try is disabling scanning for that folder. Got me a huge speedup, from 80s to 11 seconds https://support.microsoft.com/en-us...security-99e6004f-c54c-8509-773c-a4d776b77960
Of course, there are security implications, so I can only recommend it for running a test.
Because of this huge difference, I doubt virtualisation is the issue.

As a comparison, Windows on bare metal reads these

You seem to have lost some words here : )

 

[Matthias.]

On bare metal windows, I get very similar numbers, and the same speedup when disabling the scanner.

 

[matthei]

Thank you so much for pointing out the Defender Antivirus, I haven't thought of that.

That's got to be it... Now whether to disable it, is another question

You seem to have lost some words here : )

Yeah that's when I realized I need to do do a test on baremetal and forgot I stopped writing mid sentence.

We can consider this issue resolved, or well... not a proxmox issue.

 

[Matthias.]

Great! Please mark the thread as solved (edit it, there's a dropdown near the title) so others with the same problem can find it more easily.

 

[matthei]

I can't edit it, maybe the post is too old?

 

[Dunuin]

Jup, older than 1 month, so you can't edit it anymore.