[SOLVED] Slow/laggy Windows+Linux VM performance

F

FGD-Garuda

Member
Jan 22, 2022
40
14
8
48
Hi,

I am having quite slow performance on both Windows and Linux VMs. It's even worse on Linux, which is weird. The performance is comparable than what it was on my Proxmox test machine (10yo 2-core Laptop!). Dragging windows is slow, hovering over a dock with icons is slow, opening file managers display slowly, etc. I spent close to 20 hours trying countless settings, driver installs, etc. and nothing improved nor got worse. It's either it runs slow or it doesn't boot at all.

The machine is not that old, high-level it's:
  • AMD Ryzen 9 4900H
    • Integrated AMD GPU Vega 8
  • 16Gigs RAM
  • WD Black SN750SE NVMe (supposed to be a gaming drive)

I have read a lot of web pages, here are a few I followed as much as it could apply to my setup:

https://dannyda.com/2020/10/25/how-...ite-performance-speed-on-non-high-end-system/
https://pve.proxmox.com/wiki/Performance_Tweaks
https://linuxhint.com/install_virtio_drivers_kvm_qemu_windows_vm/
https://core.dpdk.org/download/
https://davejansen.com/recommended-settings-windows-10-2016-2018-2019-vm-proxmox/
https://dannyda.com/2021/10/08/how-to-install-windows-11-on-proxmox-ve-pve-without-workarounds/
https://www.youtube.com/watch?v=6c-6xBkD2J4
https://pve.proxmox.com/wiki/Windows_10_guest_best_practices
https://www.spice-space.org/download/windows/spice-guest-tools/spice-guest-tools-latest.exe
https://pve.proxmox.com/wiki/Qemu/KVM_Virtual_Machines
https://forum.proxmox.com/threads/proxmox-performance.91683/#post-399764
https://forum.proxmox.com/threads/super-vm-very-slow.98648/
https://forum.proxmox.com/threads/cpu-host-enables-nested-virutalization…-but-is-way-slower-than-cpu-kvm64…-forum-and-wiki-are-contradictory.100110/#post-437939

These are some info from my host and guest (let's start with Windows10):

Code: 
# pveversion -v
proxmox-ve: 7.1-1 (running kernel: 5.15.12-1-pve)
pve-manager: 7.1-10 (running version: 7.1-10/6ddebafe)
pve-kernel-5.15: 7.1-8
pve-kernel-helper: 7.1-8
pve-kernel-5.13: 7.1-6
pve-kernel-5.15.12-1-pve: 5.15.12-3
pve-kernel-5.13.19-3-pve: 5.13.19-7
pve-kernel-5.13.19-2-pve: 5.13.19-4
ceph-fuse: 15.2.15-pve1
corosync: 3.1.5-pve2
criu: 3.15-1+pve-1
glusterfs-client: 9.2-1
ifupdown2: 3.1.0-1+pmx3
ksm-control-daemon: 1.4-1
libjs-extjs: 7.0.0-1
libknet1: 1.22-pve2
libproxmox-acme-perl: 1.4.1
libproxmox-backup-qemu0: 1.2.0-1
libpve-access-control: 7.1-6
libpve-apiclient-perl: 3.2-1
libpve-common-perl: 7.1-2
libpve-guest-common-perl: 4.0-3
libpve-http-server-perl: 4.1-1
libpve-storage-perl: 7.0-15
libspice-server1: 0.14.3-2.1
lvm2: 2.03.11-2.1
lxc-pve: 4.0.11-1
lxcfs: 4.0.11-pve1
novnc-pve: 1.3.0-1
proxmox-backup-client: 2.1.4-1
proxmox-backup-file-restore: 2.1.4-1
proxmox-mini-journalreader: 1.3-1
proxmox-widget-toolkit: 3.4-5
pve-cluster: 7.1-3
pve-container: 4.1-3
pve-docs: 7.1-2
pve-edk2-firmware: 3.20210831-2
pve-firewall: 4.2-5
pve-firmware: 3.3-4
pve-ha-manager: 3.3-3
pve-i18n: 2.6-2
pve-qemu-kvm: 6.1.0-3
pve-xtermjs: 4.12.0-1
qemu-server: 7.1-4
smartmontools: 7.2-1
spiceterm: 3.2-2
swtpm: 0.7.0~rc1+2
vncterm: 1.7-1
zfsutils-linux: 2.1.2-pve1

Code: 
# qm config 100
agent: 1
balloon: 0
bios: seabios
boot: order=virtio0
cores: 4
cpu: kvm64
efidisk0: VM_Images:vm-100-disk-1,efitype=4m,pre-enrolled-keys=1,size=4M
machine: pc-q35-6.1
memory: 5000
meta: creation-qemu=6.1.0,ctime=1643243513
name: Windows
net0: virtio=12:EF:B7:00:A0:B2,bridge=vmbr0,firewall=1
numa: 0
ostype: win11
scsihw: virtio-scsi-single
smbios1: uuid=584a678c-96da-46cf-b992-fb4f0ca64005
sockets: 1
spice_enhancements: videostreaming=all
tpmstate0: VM_Images:vm-100-disk-2,size=4M,version=v2.0
vga: qxl,memory=64
virtio0: VM_Images:vm-100-disk-0,cache=unsafe,discard=on,size=65G
vmgenid: c87cdcd1-f353-4f63-bc76-e6c2fc7bd397

Code: 
# numactl --hardware
available: 1 nodes (0)
node 0 cpus: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
node 0 size: 15410 MB
node 0 free: 12153 MB
node distances:
node   0
  0:  10

Code: 
# lscpu
Architecture:                    x86_64
CPU op-mode(s):                  32-bit, 64-bit
Byte Order:                      Little Endian
Address sizes:                   48 bits physical, 48 bits virtual
CPU(s):                          16
On-line CPU(s) list:             0-15
Thread(s) per core:              2
Core(s) per socket:              8
Socket(s):                       1
NUMA node(s):                    1
Vendor ID:                       AuthenticAMD
CPU family:                      23
Model:                           96
Model name:                      AMD Ryzen 9 4900H with Radeon Graphi
                                 cs
Stepping:                        1
Frequency boost:                 disabled
CPU MHz:                         2735.628
CPU max MHz:                     3300.0000
CPU min MHz:                     1400.0000
BogoMIPS:                        6587.55
Virtualization:                  AMD-V
L1d cache:                       256 KiB
L1i cache:                       256 KiB
L2 cache:                        4 MiB
L3 cache:                        8 MiB
NUMA node0 CPU(s):               0-15
Vulnerability Itlb multihit:     Not affected
Vulnerability L1tf:              Not affected
Vulnerability Mds:               Not affected
Vulnerability Meltdown:          Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass
                                  disabled via prctl and seccomp
Vulnerability Spectre v1:        Mitigation; usercopy/swapgs barriers
                                  and __user pointer sanitization
Vulnerability Spectre v2:        Mitigation; Full AMD retpoline, IBPB
                                  conditional, IBRS_FW, STIBP conditi
                                 onal, RSB filling
Vulnerability Srbds:             Not affected
Vulnerability Tsx async abort:   Not affected
Flags:                           fpu vme de pse tsc msr pae mce cx8 a
                                 pic sep mtrr pge mca cmov pat pse36
                                 clflush mmx fxsr sse sse2 ht syscall
                                  nx mmxext fxsr_opt pdpe1gb rdtscp l
                                 m constant_tsc rep_good nopl nonstop
                                 _tsc cpuid extd_apicid aperfmperf ra
                                 pl pni pclmulqdq monitor ssse3 fma c
                                 x16 sse4_1 sse4_2 movbe popcnt aes x
                                 save avx f16c rdrand lahf_lm cmp_leg
                                 acy svm extapic cr8_legacy abm sse4a
                                  misalignsse 3dnowprefetch osvw ibs
                                 skinit wdt tce topoext perfctr_core
                                 perfctr_nb bpext perfctr_llc mwaitx
                                 cpb cat_l3 cdp_l3 hw_pstate ssbd mba
                                  ibrs ibpb stibp vmmcall fsgsbase bm
                                 i1 avx2 smep bmi2 cqm rdt_a rdseed a
                                 dx smap clflushopt clwb sha_ni xsave
                                 opt xsavec xgetbv1 xsaves cqm_llc cq
                                 m_occup_llc cqm_mbm_total cqm_mbm_lo
                                 cal clzero irperf xsaveerptr rdpru w
                                 bnoinvd arat npt lbrv svm_lock nrip_
                                 save tsc_scale vmcb_clean flushbyasi
                                 d decodeassists pausefilter pfthresh
                                 old avic v_vmsave_vmload vgif v_spec
                                 _ctrl umip rdpid overflow_recov succ
                                 or smca

I have the QXL installed and all VirtIO drivers as I have asked the installer software to install all of them.
There is no Warning notification in the Device Manager.

I have tried a lot of settings as I said. CPU type, NUMA, Ballooning, Display, Machine, Controllers, Bus Device, Cache, RAM amount, CPU cores amount and even some CPU flags, but that went bad so I left all to default.
Also slow when using RDP to the VM, but it seems a little bit faster and I mean "a little".

I really don't know what else to check! :(
I bought this machine specifically to use Proxmox on it.
Since it's slow also with Linux VMs, something tells me it's either a major config setting my eyes are unable to see after 20 hours looking for, or the host machine hardware is not intended to work well with Proxmox-type hypervisors and I made a bad choice...

Last info: when I type say in the command line of a VM, it lags before the characters display. I wonder if it's grafx related. The processing seems quick, I was able to see the verbose update process in an Arch-based Linux distro and it was processing faster than it could display the lines.

Does anyone know what I could try to fix this?
Or at least what to do to troubleshoot. Sometimes it's better to identify the issue first, but sometimes it's better to try out various things until one works.

Thanks in advance!
 
Last edited:
You tried CPU type "host" instead of "kvm64"?
Graphical performance is always slow/laggy if you don't use PCI passthrough and passthrough a GPU because everything needs to be rendered in software by the CPU so everything that requires 3D acceleration (like some stuff of the Windows Desktop) or video decoding (playing a youtube video in browser) or video encoding (sending a video stream using RDP) might be very laggy/choppy. But linux CLI performance should be fine...thats slow too?
 
Last edited:
FGD-Garuda said:
....
Code: 
# qm config 100
agent: 1
balloon: 0
bios: seabios
boot: order=virtio0
cores: 4
cpu: kvm64
efidisk0: VM_Images:vm-100-disk-1,efitype=4m,pre-enrolled-keys=1,size=4M
machine: pc-q35-6.1
memory: 5000
meta: creation-qemu=6.1.0,ctime=1643243513
name: Windows
net0: virtio=12:EF:B7:00:A0:B2,bridge=vmbr0,firewall=1
numa: 0
ostype: win11
scsihw: virtio-scsi-single
smbios1: uuid=584a678c-96da-46cf-b992-fb4f0ca64005
sockets: 1
spice_enhancements: videostreaming=all
tpmstate0: VM_Images:vm-100-disk-2,size=4M,version=v2.0
vga: qxl,memory=64
virtio0: VM_Images:vm-100-disk-0,cache=unsafe,discard=on,size=65G
vmgenid: c87cdcd1-f353-4f63-bc76-e6c2fc7bd397
....
Click to expand...
Try to change:
cpu: host virtio0: VM_Images:vm-100-disk-0,cache=writeback,discard=on,size=65G vga: virtio,memory=512
 
Yes I have tried cpu host (seems a very little slower) as well as few cashing methods, none, writeback, writeback unsafe, direct and another one.

I went up to 128megs on video but I can easily give it 512 why not, I'll try that. I've read 16 should be enough but in this case extremes might provide clues on what's wrong with my setup...

Not sure if it's PCI passthrough related, cuz even typing anything in any app within the VM has a 1-2sec delay before the last character typed displays on screen. No harm on testing PCI passthrough enabled though, I need to find how to do that first. But ANY thing I can try is worth trying at this point. :)

Yes Linux is also slow and is even a little slower than Windows, which doesn't make sense to me. I will try another distro to make sure as the one I tried heavily uses blur and transparency.

I will also try to install Linux on the physical machine (instead of Proxmox), install Virt-Manager and install 1 Windows and 1 Linux guests to see how it performs. That will rule out the hardware issues, I believe.

I can do all that today, except PCI passthrough if I can't find a short tutorial quickly but it'll come on that one too. I won't spare anything until I find what's going on here. :)

Tnx guys!
 
You should try a linux VM without desktop environment and then do some stuff and look if stuff like unzipping and so on is fast or run some benchmarks from CLI. If you just use CLI using an SSH agent then you atleast now that the VM itself isn't slow but just everything graphical.
In general, if you don't really need a desktop environment then run the VMs headless without any GUI. That way you save 75% of the RAM and run into way less problems because only a fraction of packages needs to be installed.

You could also test SPICE as a remote desktop protocol. Will give you more features (like remote USB devices) and feels way snappier than VNC or RDP.
 
Last edited:
Ok I have results of tests.
  • I normally use SPICE and it's virutally exactly as slow as VNC and RDP, to the eyes
  • I now run
    • cpu = host
    • vga = 512megs
    • cache = writeback
      • The above don't change performance at all
  • I went in Proxmox Shell and timed the following:
    • tar -czvf tar.gz 2_6Gigs.iso
      • 1mins27sec
    • tar -xzvf tar.gz /tmp
      • 17sec
  • Then I booted with Level 3 directly to CLI inside a Linux VM and downloaded that SAME iso file from the same place on the web (it's the Linux distro ISO itself) and timed the following:
    • tar -czvf tar.gz 2_6Gigs.iso
      • 1mins30sec
    • tar -xzvf tar.gz /tmp
      • 20sec
So the VM without DE is 3.4% and 17.6% slower than the host, which I believe is fine?

The processing then seems to be ok, I believe...

Man, no matter the setting I change there is no improvement, it's as if something else global is so slow that any config change does not affect performance enough to make it slower than that global slowdown thing. :(

I will google on PCI passthrough but this seems mostly to be for NICs or controllers. SCSI is probably not under PCI and the video is not either.

EDIT: Even the mouse is laggy, when not dragging windows or doing anything other than moving the pointer around.
 
Last edited:
FGD-Garuda said:
I will google on PCI passthrough but this seems mostly to be for NICs or controllers. SCSI is probably not under PCI and the video is not either.
Click to expand...
Every GPU, dedicated or integrated, uses PCIe today, so you can use PCI passthrough if your mainboard supports it. Same for most other stuff like soundcards, NICs, HBA/raid controllers, USB controllers, ...
 
Ok nice, I have seen 2 tutorials on that.

https://pve.proxmox.com/wiki/Pci_passthrough
https://www.thomas-krenn.com/en/wiki/Enable_Proxmox_PCIe_Passthrough

Will try that out.
I do have an IOMMU setting in the BIOS. I tried it today and it broke SPICE. I reverted back the BIOS change and SPICE is still broken even on my other 2 VMs which I haven't used today. So I will probably need to tweak a few things or reinstall everything but I want to see if that may be the culprit, seems a pretty general impact to me, based on what you mentioned.
 
Spice and the NoVNC build into PVE wont work with PCI passthrough. But you can install the OS using NoVNC, setup something like RDP or Parsec, then passthrough the GPU and connect using that.
 
Ok I made it with PCI passthrough the iGPU to a Windows machine. I know that cuz the monitor connected to the host was on and as soon as I started the VM the monitor disconnected.

It is as laggy as in any other way. The lag behaves differently but in the end it lags about the same amount.

I will return this machine and get refunded. I strongly believe I have hardware and BIOS issues as I have discovered a few things that are not acceptable and other very very very very weird behaviors you do not want to have on a machine. I have spent one month, every day, working on setting up 2 HDs, one with Proxmox and one with a Linux distro. They all gave me trouble I have never seen with my other machines, so in the trash you go!

Let's blame the lag on that.

Tnx for the help, at least I have learned a lot about Proxmox so once I get a better machine the Proxmox build should be easy!
 
Well I found out the problem. It's 3D acceleration!
This seems, for some reason, to not exist in Proxmox.

I posted here to be notified if someone ever comes up with more info https://forum.proxmox.com/threads/kvm-3d-acceleration.25722/
I explained the test I performed to validate, in the end I cam reproduce extremely similar laggy performance in VirtualBox by simply unchecking 3D Acceleration. As soon as I turn it back on, it's as if the VM runs as fast as the host.

But this option does not seem to be there in Promox. :(
 
Thats why you passthrough the GPU. As soon as you passthrough the GPU to the VM the VM got full access to the physical GPU and so is able to make full use of the 3D acceleration of that card. So if it still lags with a passthroughed GPU, 3D acceleration shouldn't be a problem.
Did you also install the GPU drivers to the guest OS after passing through the GPU so the guests OS can actually make use of that GPU?
 
Last edited:
FGD-Garuda said:
Well I found out the problem. It's 3D acceleration!
This seems, for some reason, to not exist in Proxmox.

I posted here to be notified if someone ever comes up with more info https://forum.proxmox.com/threads/kvm-3d-acceleration.25722/
I explained the test I performed to validate, in the end I cam reproduce extremely similar laggy performance in VirtualBox by simply unchecking 3D Acceleration. As soon as I turn it back on, it's as if the VM runs as fast as the host.

But this option does not seem to be there in Promox. :(
Click to expand...
Have you tried to disable Spice in the VM config and check the performance?
 
Yes SPICE is disabled when I used PCI passthrough. I tried VirtIO and another one that I cannot remember, while using PCI passthrough.
I could only make PCI passthrough work with Windows so far, the RDP connection was dropping within a tenth of a second when I was connecting to the Linux guest. Would have to troubleshoot that.

I think I might be missing some kernel parms or other backend configuration to have full HW acceleration working with PCI passthrough, as the Windows guest was still laggy. It was SO simple when I enabled PCI passthrough and then issued the commands to validate it was activated and that in fact it was, I could not believe it was so simple to enable loll. And in fact that's true cuz it's still laggy in Windows.

However after looking into how PCI passhtrough works and its requirements (usage of remote connection, no SPICE, no VNC), I don't think that solution would work in a convenient way with my use cases, sadly.

For my Windows VMs yes I think if I can make HW accel work I should be ok.
But for my Linux VMs, using PCI passthrough opens a pendora's box with the following items:

  • I need very frequent access to GRUB menu (or any other boot menu, like systemd, etc.) as I heavily leverage BTRFS snapshots feature (talking about guest here). Booting snapshots is done from GRUB menu as it's a lot easier to see which one to boot and which kernel from within the snapshot to boot from. Using commands from within the VM like grub-reboot 4 is not convenient at all and does not allow to do it when I am inside a snapshot as it's the GRUB menu from the main subvolume that controls which entry boots. One fix would be to use the host snapshot feature from Proxmox. That's not the same though, I still need to do it within the guest as I build and use custom snapshot shell scripts that when ready I implement in my production machine. To test those, I need to use the snapshot feature from the guest.
  • When I need to edit a boot menu entry I do it directly in GRUB which is super fast, convenient and temporary.
  • I modify and build GRUB menus so I need to reboot in order to view how the theme will display.
  • Granted GRUB does not need HW accel, but it's not very interesting to always remove the PCI device, re-add it, remove again, re-add, etc... it adds a lot of extra steps that frankly should not be required.
  • Those VMs are for test purposes, I change stuff all the time and I break stuff all the time. I need to see the GRUB menu and the boot process before network and RDP connection is established, as I need to see what's going on in order to troubleshoot. This is also the only way I know of so far to see exactly how the bootsplash screen behaves, that I also modify from time to time. What I found on google tells me on home machines RDP requires the guest RDP service to be activated and of course a network connection running. With PCI passthrough I cannot connect to the VMs until the login screen and that is what I experienced here.
  • The last item in that box is I often change guest window size on the fly for various convenient reasons. RDP softwares seem to have a preset list of screen resolutions to choose from. Maybe there exist some that allow custom size, but then that's one more thing to fix in the pendora's box in order to have full HW accel.
I don't know if all Type 1 HyperVisors work this way, I will certainly learn more on that.

In the end it's obvious my problem was HW accel, as it was worse on the Linux distro I was testing and it's one that uses 3D accel quite a lot. For example without 3D the windows loose their wobbly effect. I enable 3D and the effect is back. Along with blistering speed inside the guest.


I will rethink all this, but huge tnx to you guys at least I know what to hit to fix the laggy guests. :) When you don't know what the problem is, it makes fixing a lot more complicated! I'm very happy I got your replies, I wouldn't have thought about testing what you asked otherwise. :)
 
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom
Back