[SOLVED] Poor Performance in Games With Multiple VMs

[Itay1778]

Hi, I've been trying for a few months to make this work properly and I'm so close but now I have the biggest problem of all
Poor performance!
So, what I'm trying to do is one Windows VM for games
And one more VM for everyday use that MacOS is on and I connect via Parsec to the Windows VM.
Installing the two VMs and passing all the right and necessary hardware properly... took a few tries and planners (mostly on MacOS) but I succeeded and the two VMs each work great.
But here comes the problem
Game performance is not good, I do my tests on FH3 mainly because I know what its performance is on the hardware itself.
What I have tried to do so far to understand why the performance is not good:
I started from scratch I just created at first just the Windows VM and installed Windows and the drivers of the VM and the game and that's it. At first I played directly from the video card (I connected a screen to it) and everything worked great 60 FPS locked. So after that I connected via Parsec to my MacBook (using of course a network cable) and the performance was great, quality and everything.
I created a MacOS VM, set up initial settings and then installed Parsec and tried to connect to Windows VM and then.... very poor performance 30-45 FPS. At first I thought maybe my CPU was slow for two VMs at that level. I looked and the CPU was on 85 - 95 percent most of the time, but not 100 so I do not really think it's the CPU because if it was CPU it is on 100 percent all this time no?
The MacOS VM was about 20-30 percent at the time of connecting to Parsec and the Windows VM was 50-75 percent.
My processor has 4 cores. I gave the 4 cores to two VMs at the same time because when I'm not playing the Windows VM will not do anything... so I did not want a bad performance on MacOS. I said maybe because of that so I changed for MacOS from 4 for 2 cores but the performance did not change.... what is more, the use of the overall CPU didn't reach 90 percent at all. Was most of the time at 80-85 percent.

I do not understand why it is like this... what is the problem? If it was something in the CPU then it's about 100 percent all the time no? I think and hope this is some setting in Proxmox.

Hope I explained everything properly, and everything is clear.
Thanks in advance

My specs:
MB: GA-Z270X-Ultra-Gaming
CPU: i5 7600K (overclock to 4.7GHz)
RAM: 16GB
GPU for Windows: GTX 1060 6GB
GPU for MacOS: GT 710 2GB

Windows VM Config

agent: 1
bios: ovmf
boot: 
cores: 4
cpu: host
efidisk0: local-lvm:vm-101-disk-1,size=4M
hostpci0: 0000:01:00,pcie=1,x-vga=1
hostpci1: 0000:00:14,pcie=1
hostpci2: 0000:00:1f,pcie=1
machine: pc-q35-5.2
memory: 8192
name: Windows-10
net0: virtio=D2:40:77:CA:94:CC,bridge=vmbr0,firewall=1
numa: 0
ostype: win10
sata0: local-lvm:vm-101-disk-0,size=128G,ssd=1
scsihw: virtio-scsi-pci
smbios1: uuid=3dae9714-0c83-4f29-bcc3-1e4b13d3535a
sockets: 1
vga: none
vmgenid: c81e95f2-9eda-49dc-a998-f510258e0d98

MacOS VM Config

agent: 1
args: -device isa-applesmc,osk="The_OSK_I_removed_for_posting_here"  -cpu host,kvm=on,vendor=GenuineIntel,+kvm_pv_unhalt,+kvm_pv_eoi,+hypervisor,+invtsc
balloon: 0
bios: ovmf
boot: 
cores: 2
cpu: Penryn
efidisk0: local-lvm:vm-100-disk-1,size=4M
hostpci1: 0000:02:00,pcie=1,x-vga=1
hostpci2: 0000:0b:00,pcie=1
machine: q35
memory: 6144
name: MacOS
net0: virtio=16:30:9D:C7:92:9E,bridge=vmbr0,firewall=1
numa: 0
ostype: other
scsihw: virtio-scsi-pci
smbios1: uuid=9ff0a391-3a4e-4946-a27f-df1cd163775d
sockets: 1
usb0: host=3-1
vga: none
virtio0: local-lvm:vm-100-disk-0,size=70G
vmgenid: f06ba0d5-3447-40a3-9a1d-21d03b5a837f

Please ignore all the disks in the configs - this is just for testing installation and in the real usage each VM has its own SSD.

 

[leesteken]

Your CPU only has 4 cores (and 4 threads) and you are giving all four to your gaming VM. Also, 14GB of your 16GB is given to VMs that have all memory locked into RAM because of PCI passthrough. All this leaves very little for the host (and other VMs). You might not be able to use your CPU's full 100% because of this.
Your gaming VM will probably run more smoothly when you give it 2 cores less and less memory. Please try this without running other VM(s), just to check.
Your are also not using para-virtualization. Installing the VirtIO-drivers will help with faster disk I/O and lower latency.

 

[Itay1778]

Thanks for the reply.

14GB of your 16GB is given to VMs that have all memory locked into RAM because of PCI passthrough. All this leaves very little for the host (and other VMs). You might not be able to use your CPU's full 100% because of this.

I'm not really planning on adding any more VMs or CT until I increase the RAM, that's also why I gave it my all for these two VMs. In Windows VM I have the balloon service active and working I see it reports correctly to Proxmox the amount of RAM used.
But I really noticed that when I look at Node Summary it still shows that the VM uses all its RAM, is it because of the PCI passthrough? If so I would love a short explanation of why it's like this (learning is always good)
And if I understood you correctly, because there is not much RAM left for the Host then the CPU fails to reach 100%?

Your gaming VM will probably run more smoothly when you give it 2 cores less and less memory. Please try this without running other VM(s), just to check.

Ok I will try to give the Windows VM only 2 cores and see, but just to mention again all my problems start when I run the second VM. If it's just Windows alone without any other VM only then it works properly

Your are also not using para-virtualization. Installing the VirtIO-drivers will help with faster disk I/O and lower latency.

I installed the virio drivers on the windows VM. These if you mean something else?

 

[leesteken]

Thanks for the reply.

I'm not really planning on adding any more VMs or CT until I increase the RAM, that's also why I gave it my all for these two VMs. In Windows VM I have the balloon service active and working I see it reports correctly to Proxmox the amount of RAM used.
But I really noticed that when I look at Node Summary it still shows that the VM uses all its RAM, is it because of the PCI passthrough? If so I would love a short explanation of why it's like this (learning is always good)

The correct report shows that the balloon driver is working, but it cannot really give back memory to the host or other VMs. This is because you are passing real devices that can initiate Direct Memory Access (DMA) to any part of the VM memory at any time. If the VM memory is not pinned into actual memory, such a DMA could overwrite host or other VM memory.

And if I understood you correctly, because there is not much RAM left for the Host then the CPU fails to reach 100%?

Not exactly, it is because each VM wants 4 cores at the same time, which means that is has to wait for the host or other VMs, even if they do only little work for which 1 core would be enough. Therefore, some of the time some cores are idle but cannot be used by another VM, which prevents you from using all of your cores at a 100% all the time. There is scheduling overhead which becomes more noticable when you push the whole over the core limit. The high memory usage might cause heavy swap I/O and/or reduce the memory available for file or storage caching, which also causes more I/O and this also takes CPU time.

Ok I will try to give the Windows VM only 2 cores and see, but just to mention again all my problems start when I run the second VM. If it's just Windows alone without any other VM only then it works properly

If you have many cores and many lightly-used VMs with few cores (like in an enterprise setup), you can overcommit because the scheduler had more cores to smooth little peaks in usage.
But in your case 2 VMs (8 cores, and the host needs some) are competing over very limited resources, causing higher latency and stuttering. Maybe two cores is not optimal for your games, but you'll need at least 1 core for Proxmox (which gathers statistics for the web GUI). Maybe your Mac VM could also do with only 1 core? But still they will interfere with your gaming VM if that one wants all 4 cores. Maybe 3 for gaming, 1 for everything else would be a good compromize? You will have to try and see.

I installed the virio drivers on the windows VM. These if you mean something else?

Your VM config shows sata0 and not virtio0 or better yet scsi0, so you are not using the VirtIO drivers for your disk. I don't know if WIndows need reinstalling or can switch without problems, sorry. There are other post in this forum about Windows VirtIO drivers.

 

[Itay1778]

The correct report shows that the balloon driver is working, but it cannot really give back memory to the host or other VMs. This is because you are passing real devices that can initiate Direct Memory Access (DMA) to any part of the VM memory at any time. If the VM memory is not pinned into actual memory, such a DMA could overwrite host or other VM memory.

Not exactly, it is because each VM wants 4 cores at the same time, which means that is has to wait for the host or other VMs, even if they do only little work for which 1 core would be enough. Therefore, some of the time some cores are idle but cannot be used by another VM, which prevents you from using all of your cores at a 100% all the time. There is scheduling overhead which becomes more noticable when you push the whole over the core limit. The high memory usage might cause heavy swap I/O and/or reduce the memory available for file or storage caching, which also causes more I/O and this also takes CPU time.

Ok, thanks for all the explanations.

But in your case 2 VMs (8 cores, and the host needs some) are competing over very limited resources, causing higher latency and stuttering. Maybe two cores is not optimal for your games, but you'll need at least 1 core for Proxmox (which gathers statistics for the web GUI). Maybe your Mac VM could also do with only 1 core? But still they will interfere with your gaming VM if that one wants all 4 cores. Maybe 3 for gaming, 1 for everything else would be a good compromize? You will have to try and see.

So the correct settings is basically to give each VMs 2 cores? Or is that not good either, because you said that proxmox needs at least a core for it, so basically give one VM 2 cores and the other one core right?

Your VM config shows sata0 and not virtio0 or better yet scsi0, so you are not using the VirtIO drivers for your disk. I don't know if WIndows need reinstalling or can switch without problems, sorry. There are other post in this forum about Windows VirtIO drivers.

I thought thats what you say, Windows uses Sata because I first tried to install Windows on virtio and also on scsi and installed both drivers in Windows but after a few restarts to Windows, I do not know what really happened there but it seems that Windows on its own decided to delete the drivers, so I went for the thing that will surely work that it sata. That it works relatively well. And you say scsi is better than virtio according to some posts I read in several forums that virtio is better than scsi.... now you really confused me

 

[leesteken]

My (personal opinion) is that using VirtIO SCSI (not other emulated SCSI choices) performs as good as VirtIO and has more features (such as SSD emulation). I even use it for virtual CD/DVD. Feel free to try other options and see if it makes a differences, but ide en sata are emulated and slower than both virtio and non-emulated scsi.
Each VM 2 cores could work if your Mac VM is not busy while you use the gaming VM, and the other way around, because one lightly-used VM and Proxmox can probably share. Please just try and see if it makes a difference and what works for you. If you need more performance for a single VM, I suggest upgrading your hardware or run Proxmox on a system which you don't use for gaming (as it is server oriented and optimized anyway).

 

[Itay1778]

My (personal opinion) is that using VirtIO SCSI (not other emulated SCSI choices) performs as good as VirtIO and has more features (such as SSD emulation). I even use it for virtual CD/DVD. Feel free to try other options and see if it makes a differences, but ide en sata are emulated and slower than both virtio and non-emulated scsi.
Each VM 2 cores could work if your Mac VM is not busy while you use the gaming VM, and the other way around, because one lightly-used VM and Proxmox can probably share. Please just try and see if it makes a difference and what works for you. If you need more performance for a single VM, I suggest upgrading your hardware or run Proxmox on a system which you don't use for gaming (as it is server oriented and optimized anyway).

Ok, so after I fix this whole performance and core issue, I'll try to reinstall Windows with SCSI.
Yes, the MacOS VM will not do too much when I use the Windows VM.
I'll now check it out, and see if it matters at all.
Thanks for all the help.

 

[Itay1778]

upgrading your hardware

Ok, first of all thank you so much for all the help and for making me understand what the problem is.
So it seems I can not currently do what I want as I want, because I have to upgrade the hardware first to the processor with more cores.
Right now the AMD Ryzen 5 5600X looks really good, put 6 cores for each VM (because it has 12 threads) or 5 if I want another VM and CT there.
But I have another Proxmox for stuff like that. This Proxmox is really only for games in Windows, and MacOS for everyday use

 

[leesteken]

Please be aware that those extra threads (like Intel's Hyperthreading) won't make the system twice as performant. It is more like between 1.2x and 1.4x instead of 2x. It does make the system more responsive because the scheduler has more to work with. However, this only works if you keep your cores per VM small. You could give the gaming VM half of the threads and keep the rest for host and Mac, but do not give the Mac VM all the other 6. Just 2 or maybe 3.
I know this sounds wasteful, not using all threads for VMs, and this way you will never see 100% system usage. But that is a good thing if you want to game on a virtualization system, because you want some room to keep the latency low and the system responsive.

PS: Regarding new hardware: while I prefer Ryzen myself, maybe you could just upgrade to an i7-7700 without replacing the rest of your system?

 

[Itay1778]

Please be aware that those extra threads (like Intel's Hyperthreading) won't make the system twice as performant. It is more like between 1.2x and 1.4x instead of 2x. It does make the system more responsive because the scheduler has more to work with. However, this only works if you keep your cores per VM small. You could give the gaming VM half of the threads and keep the rest for host and Mac, but do not give the Mac VM all the other 6. Just 2 or maybe 3.
I know this sounds wasteful, not using all threads for VMs, and this way you will never see 100% system usage. But that is a good thing if you want to game on a virtualization system, because you want some room to keep the latency low and the system responsive.

Ok, so give 6 cores to gaming VM but instead of giving only 3 cores to MacOS VM because it will be for everyday use I do not want anything less than 4 cores, so I thought maybe 6 cores for gaming VM and MacOS 4 cores and Proxmox still has 2 cores. I understand that I also consider threads as "core" but still do not cross the maximum of the CPU as I did now, so there should be no problems right? And Proxmox can get by with2, no?
And we're talking about faster cores than I had now so that should also help keep latency low and the system responsive

PS: Regarding new hardware: while I prefer Ryzen myself, maybe you could just upgrade to an i7-7700 without replacing the rest of your system?

I thought about it, but because I want at least 4 cores per VM then there will be nothing left for the Host itself, because the i7 7700k has only 8 threads... the question is whether it will be good or not?
And also regarding Ryzen I probably will not go with it because it does not have the iGPU and because I pass all the GPUs to VMs so I am afraid I will have a problem with it, and that Proxmox will make problems with it remaining headless.
So I will probably buy intel. But I will wait for them to have a new generation because it should be soon. And right now I will be in Dual-boot between Windows and MacOS

 

[leesteken]

If you are willing to buy another system: why not run one system for Windows, one system for Mac and don't use Proxmox at all (just share the screen, keyboard and mouse)? Then you can use each system for a 100% for a particular purpose and you won't have the virtualization overhead.

 

[Itay1778]

If you are willing to buy another system: why not run one system for Windows, one system for Mac and don't use Proxmox at all (just share the screen, keyboard and mouse)? Then you can use each system for a 100% for a particular purpose and you won't have the virtualization overhead.

That would be great but... other than just replacing the motherboard and processor and maybe adding RAM. I'll have to buy a case, power supply, CPU cooling and etc. And here I am just replacing only the motherboard and processor, and in the future new GPUs.
And I also planned to use this hardware, (after I upgrade) to my storage server (TrueNAS). I still do not have the budget to perform this upgrade now, and when I have the budget I will have to choose either it (replace processor, etc) or upgrade my TrueNAS Server and this way I do a few things together.
True it will be the best thing system for each of them but unfortunately it is not possible, mainly for reasons of budgets.

And as for the replacement to the i7 7700K, and set it up as I mentioned, do you think it will not be good?

 

[leesteken]

I cannot give any guarantees (and certainly no refunds!) if your new hardware does not perform well enough or does not support (enough) PCI passthrough, sorry.
I would guess that 8 threads will work good enough with 3 cores for each VM. Maybe ask (or wait for) other people to give an opinion on this?
Have you tested if using less cores made the gaming VM more responsive/work better?

 

[Itay1778]

I cannot give any guarantees (and certainly no refunds!) if your new hardware does not perform well enough or does not support (enough) PCI passthrough, sorry.
I would guess that 8 threads will work good enough with 3 cores for each VM. Maybe ask (or wait for) other people to give an opinion on this?
Have you tested if using less cores made the gaming VM more responsive/work better?

Before I buy the new board I will research it regarding PCI passthrough and virtualization
Yes I tried to put in gaming VM fewer cores, I tried 2 cores and it didn't look nice at all... I went from 130 FPS to 25 FPS...

I probably will not upgrade to i7 7700k, just will save for a new platform already.