Imported ESXi VM performance issues

[stevenwh]

Hi all, I'm trying to figure out what could be causing extremely poor performance on an imported ESXi Windows VM.

On ESXi (even on the exact same machine) the VM responds great. Very snappy responses and overall good performance. But after importing it into Proxmox, the performance is really struggling.

I've ruled out it being a disk IO issue by copying the cloning the virtual disk to a dedicated SSD (using dd to clone the disk directly to the SSD and using the SSD directly as a boot drive after adding it to the VM using command line qm set <id> -scsi0 <disk>

I've changed the Processor Type to host, tried with Numa both enabled and disabled. Set the CPU units as high as 2048. Tried core count from 4 to 8. And set CPU affinity for a 1 to 1 vcore / cpu core. The performance remained unchanged with all these steps. Simple idling on the desktop in this VM with minimal processes (I've disabled all start up processes) and the CPU is idling (in the VM task manager) at 60 to 80%. Looking at the processes specifically while doing this shows Task Manager using 50 to 60% on it's own....

There is a 1 to 2 second delay on any clicks to open menus (start menu or any of the desktop / task bar context menus). And this is even through remote desktop so not a console latency thing.

I removed VMWare tools and disabled a bunch of other things before importing the VM but doesn't seem to make a difference.

Like I said, this performance issue is not happening in ESXi on the exact same hardware. Right now while I've been testing trying to make the switch to Proxmox I've got the server configured where I can boot into Proxmox on one disk with it's own dedicated storage array, or I can boot into ESXi on a different disk and a different dedicated storage array. I also have a couple of individual SSDs for each that I can use for a direct dedicated disk to a VM to rule out IO performance bottle necks. When this VM is started in ESXi, everything responds almost as well as a native windows install would. Nearly instant menu openings and idle CPU usage around 5 to 10%.

I'm not sure what else I can try doing here and what might be the performance issue?

 

[bbgeek17]

Did you install the necessary VirtIO drivers? (disk, video, network)

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[stevenwh]

Yes all virt io drivers are installed as well as qemu guest agent and qemu guest agent is enabled in options.

 

[entilza]

Whats the cpu usage / etc in task manager inside the VM? ie whats using the % .. Could it be doing any updates?

 

[Dunuin]

qm set <id> -scsi0 <disk>

Then the VM is still using a virtual disk. Just the contents of that virtual disk are mapped to that physical disk. To rule out IO it would be better to PCI passthrough a NVMe SSD so the VM is working with the real physical hardware and skipping the virtualization. But yes, probably not a storage bottleneck anyway.

 

[Kingneutron]

Did you follow the Win10 guest guide?

https://pve.proxmox.com/wiki/Windows_10_guest_best_practices


Probably wouldn't hurt to run an sfc/scannow and a dism check, if all else fails I would try a "refresh" install of win10 from ISO

https://github.com/kneutron/ansitest/blob/master/winstuff/sfc-scannow-dism.cmd

 

[stevenwh]

Whats the cpu usage / etc in task manager inside the VM? ie whats using the % .. Could it be doing any updates?

Nothing really, task manager itself is showing using the majority of the cpu. The rest is spread across a few different service hosts (remote desktop, windows manager, etc) and 2 to 3% on antimalware (windows defender).

Did you follow the Win10 guest guide?

https://pve.proxmox.com/wiki/Windows_10_guest_best_practices


Probably wouldn't hurt to run an sfc/scannow and a dism check, if all else fails I would try a "refresh" install of win10 from ISO

https://github.com/kneutron/ansitest/blob/master/winstuff/sfc-scannow-dism.cmd

Yes everything in that guide has been done that I can do.




I remembered about another thing that I ran into a few weeks ago when I first started messing with Proxmox and I'm wondering if it might be the same problem. That being that VMWare by default shares non pass thru GPU resources to VMs automatically. Proxmox doesn't from what I can tell. So I wonder if the issue is because the VM on proxmox is eating away a lot of CPU for software rendering.

I'm trying to pass through a vGPU from my Tesla P40 to it that I have working in other Windows VMs that I installed on Proxmox for tests but for some reason I'm getting a code 43 in this vm. No idea why yet...

 

[stevenwh]

I remembered about another thing that I ran into a few weeks ago when I first started messing with Proxmox and I'm wondering if it might be the same problem. That being that VMWare by default shares non pass thru GPU resources to VMs automatically. Proxmox doesn't from what I can tell. So I wonder if the issue is because the VM on proxmox is eating away a lot of CPU for software rendering.

I'm trying to pass through a vGPU from my Tesla P40 to it that I have working in other Windows VMs that I installed on Proxmox for tests but for some reason I'm getting a code 43 in this vm. No idea why yet...

Yeah, I'm at a complete loss. Nothing I try to do gets rid of the Code 43 problem. I can fire up a brand new Windows 11 VM on Proxmox while this problematic VM is running and have the Tesla P40 vGPU work flawlessly in it. But it simply will not work on this VM. I really have no clue what could possibly be causing this much issue on this VM but it works perfectly fine under ESXi. Maybe just some really weird driver or software conflict, but no idea how to track it down. And the VM is running way to slow for me to keep messing with it aimlessly. Takes 5 to 10 minutes to do something that should only take 30 seconds at most because of the VM running so poorly. Really would like to figure it out cause it's so odd to not be able to find anything consuming resources yet it still be running this slow, but I've already wasted most of two days trying to mess with this, I could've reinstalled the VM and reconfigured all the settings in less time lol.... Hopefully importing other non windows VMs goes better.

 

[gfngfn256]

Maybe share the VMID.conf so users can get helping pointers

 

[stevenwh]

Maybe share the VMID.conf so users can get helping pointers

agent: 1
bios: ovmf
boot: order=scsi0;sata0
cores: 6
cpu: host
cpuunits: 2048
efidisk0: primary_raid10:vm-105-disk-0,size=4M
hostpci0: 0000:04:00.0,mdev=nvidia-51,pcie=1,x-vga=1
machine: pc-q35-8.1
memory: 32768
meta: creation-qemu=8.1.5,ctime=1714362520
name: tb-windows-clone
net0: virtio=BC:24:11:7E:C9:FE,bridge=vmbr0
numa: 1
ostype: win11
sata0: none,media=cdrom
scsi0: /dev/disk/by-id/wwn-0x61418770474bbc002dc1990b0a8023ba,size=931G
scsihw: virtio-scsi-single
smbios1: uuid=564d72eb-7965-275b-7545-4409ac97d94e
sockets: 1
unused0: primary_raid10:vm-105-disk-2
vga: none
vmgenid: ac7cfd5b-c0bb-4bd1-ab09-0943f6c24278

 

[gfngfn256]

Everything looking good in Device Manager?

 

[stevenwh]

Everything looking good in Device Manager?

Other than the P40 code 43ing for who knows what reason.. yeah. (just to be clear performance issues were there before tying to install the P40, was trying it to see if having non software rendering would make a difference.

 

[gfngfn256]

I guess you've also done a (successful) Windows Update.

 

[stevenwh]

I guess you've also done a (successful) Windows Update.

Yep, windows is fully updated

 

[gfngfn256]

Maybe I'm barking up the wrong tree, but what's that Remote Desktop Camera Device in Device manager for. Are you really trying to access a Web Cam through RDP? I tested on an RDP enabled Windows 11 & that doesn't show.

 

[stevenwh]

Maybe I'm barking up the wrong tree, but what's that Remote Desktop Camera Device in Device manager for. Are you really trying to access a Web Cam through RDP? I tested on an RDP enabled Windows 11 & that doesn't show.

Yes I use a web cam over RDP occasionally. It works mostly fine usually. Remote desktop client automatically passes it in if you have local resources enabled with it before connecting. It's not always installed on the machine and definitely not the cause of any performance issues as the performance issues persist if I connect via another method instead of remote desktop (at which point the camera does not exist).

 

[iwik]

What else is your proxmox host running? Only this one vm? Can you post screenshot of top?

I don't know what can cause this, but maybe some power management?

Few usefull commands (need to install stuff of course), which I have in my notes:

apt install i7z linux-cpupower
x86_energy_perf_policy performance
cpupower frequency-info governors
watch -n0 cpupower monitor -i2
turbostat --Summary --show Busy%,Bzy_MHz,PkgTmp,PkgWatt,GFXWatt,IRQ --interval 6

 

[gfngfn256]

Yes I use a web cam over RDP occasionally. It works mostly fine usually. Remote desktop client automatically passes it in if you have local resources enabled with it before connecting. It's not always installed on the machine and definitely not the cause of any performance issues as the performance issues persist if I connect via another method instead of remote desktop (at which point the camera does not exist).

Got you. But maybe "unplug" the webcam itself from the VM & try again.

 

[Dunuin]

Dis you disable "core isolation" in Windows for that VM?

 

[entilza]

Balloon off will help regardless but if its this sluggish must be something else, I wonder if you install a clean windows with nothing. Must be something with the gpu?