Bad memory performance on Windows 10

[Dashwood]

Hi all

I set up a new server with a Proxmox VE installation:
- AMD Epyc 7301
- 64 GB DDR4 ECC memory
- 4x SSD in RAID5 with hardware RAID-Controller

I also created a Win10 KVM for code compiling:
- CPU=host and numa=1 (with all cores for testing)
- storage=virtio SCSI
- disk cache = I tested noCache, writeBack and unsafe
All VirtIO drivers, ballooning and guest agent are installed.

The problem now is that the compiling of the code takes more than twice the time as it takes on my few years old Intel i7 6 Gen. workstation with the same software installation and DDR3 memory. All cpu cores are unter full load while compiling.

So I took a system-benchmark on both machines and the VM was overall far better but the memory, which had less than half of the performance as on the workstation.

Next I compared the memory with sysbench on the Proxmox-server and the Workstation:

Sysbench command: sysbench --test=memory --num-threads=4 run

Result on the Proxmox-server (not in a VM):
Total operations: 19500912 (1949722.62 per second)
19043.86 MiB transferred (1904.03 MiB/sec)

Result on the Win10 VM in Windows subsystem for Linux:
Total operations: 8084654 (808256.14 per second)
7895.17 MiB transferred (789.31 MiB/sec)

Result on the Win10 VM booted in a Live Ubuntu:
Total operations: 17865358 (1786183.87 per second)
7895.17 MiB transferred (1744.32 MiB/sec)

Result on the Workstation on Win10 subsystem for Linux:
Total operations: 14111898 (1410967.57 per second)
13781.15 MiB transferred (1377.90 MiB/sec)

To me it looks like a driver problem, since the VM booted with Ubuntu gets nearly the full performance.
Does anyone have an idea what could cause this behaviour?

 

[wolfgang]

Hi,

why do you use a win10 with Linux subsystems?
Instead of compiling it in a Linux?

There are so many things involved in such an Environment and it is hard to debug such setup.

 

[Dashwood]

Hi,

Linux subsystem is only used for the sysbench benchmark to get comparable results from the Windows VM.
MSBuild is used to compile the actual code.

 

[BobhWasatch]

Doesn't wsl add a layer of virtualization? I do a lot of Linux development and tried out wsl (v1...IT hasn't updated us yet) to see if it was better than remote servers or regular virtualization. Basically running Linux in hyper-v is significantly faster for compiling code than doing the same task in wsl1. Even virtualbox is faster

 

[Dashwood]

Sorry if it´s not clear enough.
I only use WSL to compare sysbench (which is twice as fast on the old physical machine) and also tried a Windows benchmark software with the same result. Everything is faster on the VM except the RAM which gets only half of the benchmark score than the old physical machine.

The actual code is gets compiled with Msbuild and not WSL.

 

[wolfgang]

I think this is the wrong way to compare it.
It is generally complicated to measure this because Windows and Linux memory systems work entirely different.
Also, the memory allocation function has a massive impact on the results, and they are different on both platforms.

 

[Dashwood]

What would be an appropriate way to measure a Windows VM performance?
Somewhere has to be a bottleneck.

 

[rushandrush]

Hi, I agree with @wolfgang that the comparison could be in the wrong way.
I'm experiencing the same issue, and I believe there is a memory performance issue in Windows 10 VM.
I did similar things with sysbench as you, but since the result is not comparable, I also go with another approach for comparison:

I installed a Windows 10 Pro in a bare metal way, and run the AIDA64 Cache & Memory Benchmark tool, and then I installed a Proxmox in the same machine with a Windows 10 Pro as a VM on it and run the same benchmark tool again.

On the bare metal:



On the VM:



my vm config file:

args: -machine type=q35,kernel_irqchip=on
balloon: 0
bios: ovmf
boot: c
bootdisk: scsi0
cores: 12
cpu: host
efidisk0: local-lvm:vm-103-disk-1,size=4M
hostpci0: 83:00,pcie=1,x-vga=1
machine: q35
memory: 12288
name: SHA003
net0: virtio=xxx,bridge=vmbr0
numa: 1
numa0: cpus=0-5,hostnodes=0,memory=6144,policy=bind
numa1: cpus=6-11,hostnodes=1,memory=6144,policy=bind
ostype: win10
scsi0: local-lvm:vm-103-disk-0,backup=0,cache=writeback,discard=on,iothread=1,replicate=0,size=250G,ssd=1
scsihw: virtio-scsi-single
smbios1: uuid=xxx
sockets: 1

Hope someone could point out where I went wrong. Very much appreciate it.

 

[SamirD]

Try disabling numa and see what you get.

 

[rushandrush]

Try disabling numa and see what you get.

After setting numa to 0:

 

[rushandrush]

Hi @SamirD from the above screenshot, I believe NUMA settings are not the cause. What else do you suspect could be a cause?

 

[wolfgang]

Here this tool from intel will explain a bit[1]
I guess the main reason for the bad performance is the different use of the CPU (disable H/W prefetching).

1.) https://software.intel.com/content/www/us/en/develop/articles/intelr-memory-latency-checker.html

 

[SamirD]

Hi @SamirD from the above screenshot, I believe NUMA settings are not the cause. What else do you suspect could be a cause?

I agree that numa didn't explain it. Have you tried wolfgang's suggestion of disabling prefetching?

 

[rushandrush]

I agree that numa didn't explain it. Have you tried wolfgang's suggestion of disabling prefetching?

Hi @SamirD yes, I tried to disable hardware prefetching from BIOS and the performance slightly improves but still I haven't found a way to solve the major problem.

Here is the result of the VM after h/w prefetching is disabled.

 

[SamirD]

Quite puzzling. I'm sure it's just a setting somewhere, but what and where is the question and I'm not familiar with proxmox at all yet so I don't even know what to suggest. Hopefully someone else can chime in with some ideas.

 

[Anti-Ctrl]

So, i made an Account, just to say, that i have EXACTLY the same Problem. Memory Performance is really bad. I tried to debug this since 3 or 4 Weeks now and I´m all out of Ideas. In a Linux VM i get around 24GB/s with 1M BS which is around the maximum my Board/System can handle. I used the Phoronix Test Suite as a Measurement. In Windows 10 and Windows 7 i use AIDA64 and get around 1.5GB/s read and 900 MB/s write. I also ran Pheronix Test Suite on the Windows VM and got around 100 Times Slower Speeds, compared to a Linux VM.

Things i tried:
Disable/Enable NUMA
Disabled HEPT in the VM and BIOS.
Enabled Hugepages on the Host and inside the Windows VM
Checked Memory on the Host
Changed Performance Options in the BIOS
Pinned CPU Cores
Updated to the Latest Stable Virt-IO Drivers
Asked in Level1Techs Forum
surely there is something missing from this List.

Is there ANYTHING i can test further ?
BTW, CPU Interrupts are in the High 30ks with spikes to 80ks, if the Windows VM is running.

My System:
CPU: Dual Intel x5675 Xeon
MB: Asus Z8NAD6
RAM: 96GB 1066Mhz DDR3 CL7 from Samsung. (On supported List)
Graphics Card: R9 380 (Passed Through to Windows VM)
PSU: 550W Cheapo. Not crappy, but not well either. Good one is on its Way.

 

[Gektor]

I think, it's a bug of AIDA64 test algorithm in guest VM, because visually system run fast.

 

[Anti-Ctrl]

I think, it's a bug of AIDA64 test algorithm in guest VM, because visually system run fast.

It's not. Phoronix Test Suite shows the same bad Performance in a various of tests, at least for me.
Proxmox has just really bad Windows VM Performance and I, for myself, am going to switch to ESXi, which is a shame, because then i need another VM for ZFS Stuff.

 

[Gektor]

Hmmm.. i have test in my system (WS2019), and get very low speeds (almost 20x lower than on bare metal):


So, i think that it's AIDA64 bug, in other case, with that speeds. WS2019 boots takes very long time. In my case it's around 3-5 seconds....

 

[Gektor]

I have test in WinRAR:
Proxmox (6.3): ~7600 KB/s
Hyper-V (2019) ~ 8100 KB/s
Bare metal: ~8600 KB/s
Looks like 13% performance hit, not 20x as AIDA64 shows. So, i think, it's AIDA64 measurements bug in VM.