Proxmox Windows VM slowness

[mrbeanzg]

Hello everyone,

I have one problem that I can’t possibly solve.

I have a cluster of 4 x hp dl 360 gen 9 with 2xE5-2650 v3 connected to HP 3Par 7400 over 10gb ISCSI.

The windows vm are server 2019 with all latest drivers and virtio network and disk.The processor are set to host.

Our ERP is running on those Windows VM which is connected to MYSQL and SQL datbases and runs fine without any problem.

We also have another cluster on Hyper-V on olderBL360c Gen 7 servers, which are currently testing enviroment, where Windows server 2019 is also installed and fully patched.
The problem is that our ERP which is written in Delphi, runs much faster on Hyper-v older eniroment then on newer Proxmox one.



I have tried all possible solutions but nothing worked to get the same performance like on older servers. (driver version, bios settings)



Proxmox details:

Kernel: Linux 5.0.15-1-pve # 1 SMP PVE 5.0.15-1

PVE version: 6.0-4 / 2a719255



Any thought of being able to get performance as on old hardware is welcome.



Thanks everyone

 

[LnxBil]

What is performance for you? Could you please elaborate?
"slowness" is not very specific.

I'd compare raw numbers like:

• cpu benchmark
• disk throughput benchmark
• disk iops benchmark
• memory benchmark

 

[mrbeanzg]

Hi and thanks for reply.

i have run some benchmark and this is some rough score :

cinebench 20 :
hyper v old hardware - score 800
proxmox new hardware - score 1600
crystal disk :
hyper v old hardware - seq 400mb/s
proxmox new hardware - seq 8000mb/s
userbenchmark memory :
hyper v old hardware - Multi core 9,73GB/s
proxmox new hardware - Multi core 34.9GB/s

On all benchmarks, the new server is faster but not on our application.

If i run the application on old and new server vm, the old server outperform the new server. The load time on the application on old hyper v server is faster by 20 seconds than the new proxmox server.

How is this possible? some tweaks in proxmox host? i doubt that is hardware specific.

Thanks

 

[LnxBil]

Hmm, good to know that the numbers show that the new platform is faster, as it should be.

What does your application do? Does ist also has a network component? What about network speeds?

 

[mrbeanzg]

Hi,

Our app uses sql as backend db. i have try to locally install sql server but the same. sql server is on another vm that is connected via 10gb nic and all vm are access it.

I'm getting around 900mb/s transfer speed over network. Another hyper v vm are also connected to the same sql server and they are working fine.

i have tryed averything i could find on proxmox forum and around web but nothing helps.

As you said, the weird thing is that all benchmarks are report the faster vm but application is much slower.

Can i do anything else?

Best regards

 

[LnxBil]

Can i do anything else?

It's hard to debug if all benchmarks point to the conclusion that it should be fast. So I do not have anything to offer as we do not know what is slow at the moment. Unfortunately, a "slow feeling" cannot be quantified and therefore not tackled.

 

[RokaKen]

Can i do anything else?

Have you already followed the Windows 2019 Best Practices and subsequent performance tweaks as documented?

 

[NeytechAR]

Hi, I'm clinging to this thread because I have a similar issue regarding SQL Server VMs.
This is what I've done until now:
- Created 6 SQL Server VMs (running Windows Server 2016 Standard. VirtIO drivers installed. BIOS: UEFI)
- Each VM have 4 hdd:
HDD Sata0: Windows Server OS Partition (format: qcow2. Cache: Write Through)
HDD Sata1: SQL Server Data partition -for .mdf/.ndf files- (format: qcow2. Cache: Write Through)
HDD Sata2: SQL Server Log partition -for .ldf files- (format: qcow2. Cache: Write Through)
HDD Sata3: Temporary repository partition -in case a temporal db backup is needed for some developer need- (format: qcow2. Cache: Write Through)

So what I've noticed is that there are times when I'm running a restore or backup of some db and it takes way too long to finish and sometimes databases turns to 'Suspect' status. When that happens the only thing for me to do is delete that db and restore it from an old backup.
Doing some Googling I found a document describing best practices for Windows Server 2016 and SQL Server. Those recommendations was not to use qcow2 disc format but raw. And not using VirtIO Ballooning driver.
After reading that I wanted to try those recommendations but I stumbled with some questions:
-I've already installed all VirtIO drivers on those VMs including Ballooning driver. If I disable this option ([balloon=0]) is the same as if this driver was not installed?
- This one is most critical (I suppose). If try convert format from qcow2 to raw... is there any chance to loose everything or get some inconsistencies as result? I mean this is Windows if you know what I mean.
I was thinking on adding new discs with raw format, move .mdf/.ldf files all along and configure SQL Server to use these new discs. But my fear comes to the OS disc partition. Should I have to convert this disc from qcow2 to raw as well? And what are the risks?

I'm very thankful for your comments and assistance.

Edit: I disabled Ballooning ([balloon=0]), rebooted all these VMs and now seems to work kind of much smoother. I didn't change discs format to raw yet. Thank you

 

[mrbeanzg]

Hi all,

I forgot to reply to this thred.
I resolve the issue with host bios. i have set host bios to high performance and all start to work fast. I guess that proxmox dont react very well with os control settings.

Bes tregards

 

[apoc]

@mrbeanzg: also make sure you are not using the default CPU in your VM-config. Switching to the host-cpu-type makes things a lot faster and I mean A LOT!
if you have mixed environments though you should choose the lowest cpu family though. That increased performance on my end significantly.

 

[mrbeanzg]

Hi all,

I need to reopen this thred.
As mention the bios "max performance" have made VM faster, we still didn't get speed in our erp as on the old hardware.
We try following twaks :
1. playing with virtio drivers versions - nothing changed
2. turn off "balooning" as our servers have allot of free ram - nothing changed
3. set network to e1000e - nothing changed

we have noticed that the copyin from one windows vm to another have drops. copying is not uniform but jumps from 2mb to 100 mb when an iso file is copied.

Does anyone have similar problems and maybe a solution?

Thanks all and best regards

 

[Zananok]

Greetings
I have some issues as well
All my lxc and Linux VMs are fine, running at max speed (750+mbps)
However, my Win10 VM reports ~220 when using the E1000 and ~265mbps using VirtIO

The windows VM is using cpu: host (with access to half the cores) and half of all the ram in the system.


Regards,
Zananok

 

[gregzuro]

@mrbeanzg: also make sure you are not using the default CPU in your VM-config. Switching to the host-cpu-type makes things a lot faster and I mean A LOT!
if you have mixed environments though you should choose the lowest cpu family though. That increased performance on my end significantly.

Just to expand on this:

I've had terrible performance with a Windows 10 client (and somewhat terrible with Ubuntu 20.04) when using `host`.
Changing to the actual host type - in this case - `Sandy Bridge` - gave a *huge* performance increase.

I found that benchmarks (geekbench) were similar before and after the change, but the UI with the machines (windows loading, cursor, text entry) was improved vastly.

Hope this helps someone!

 

[apoc]

Interesting. Thanks for sharing. @gregzuro
Will have a look into that. Currently my VMs run on "host" profile - will switch to the real CPU (Opteron)

 

[Zananok]

Can confirm that unsetting cpu: host drastically improves the performance. I got noticeable gains from CPU and GPU performance.
The reason why I stick to setting CPU for a while, other than not being a very noticeable performance impact unless I was doing some heavy-duty processing, is the need to define the CPU with specific instruction sets to run Photoshop 2021. I have yet to look for a workaround but hope to find some solution soon.

Here are results for those interested in seeing the diff; no changes were done in any way, other than unsetting the CPU in the config.
With CPU flag to host: https://www.userbenchmark.com/UserRun/49811538
Without: https://www.userbenchmark.com/UserRun/49812170

Although there are strange reports of cores/threads given by the results, I give the VM 96 "pve cores" and 128 GB ram. It's a 100% passthrough of the GPU. The system is exotic cooled, so there is no chance for thermal throttling.

Edit: I think I lost some processing power for using win 11 in the VM, but I can't find my old result from when the server was air-cooled 1~2 years ago and did not have the CPU defined in the config.


Regards,
Zananok