Using both integrated and discrete GPU

[vdjur]

Hi,

I am considering buying new PC for Proxmox virtualization. It would have Ryzen CPU with integrated GPU and dedicated Nvidia 3060 GPU.

My plan would be for Proxmox server to use integrated GPU - that would be physically connected via HDMI cable to monitor 1. This would be used as management console.

I would then use PCI-E passthrough for dedicated Nvidia GPU and use it from single Windows 11 VM instance. This would be connected with DisplayPort cable to monitor 2 and monitor 3 (Windows 11 would have two monitors).

Would such configuration be possible (without much headache), so I could simultaneously use both GPUs with display output to all three monitors?
(integrated GPU would display Proxmox server console and dedicated GPU would output directly on two monitors for Windows VM)

Thank you very much for your help!

 

[leesteken]

That will work, IF you get a GPU that works well with passthrough (i.e., that it resets properly when passed to VFIO). Also, get a CPU with integrated graphics that works with Linux. And make sure to get a motherboard that supports PCIe passthrough (with the CPU). Search this forum (and/or the internet) for CPU+motherboard+GPU combinations that are known to work (well) with passthrough.

 

[vdjur]

Thank you very much!

 

[iamanewuser]

My situation is very similar to yours, use integrated GPU for PVE itself and use Nvidia 1060 GPU passthrough for Windows 10 VM, they work well.

 

[vdjur]

Thank you for you feedback! Do you have Intel or AMD CPU?

I am still waiting for ZEN 4 to be released but once I finalize my setup I will publish my experience here
In the meantime if anyone has additional suggestions - they are welcome!

 

[iamanewuser]

Intel CPU I7-10700

 

[vdjur]

Hi,

I got the computer one week ago and here is my initial experience with Proxmox (TLDR: it is 99% what I wanted)

Relevant build components:
-ASUS ROG STRIX X670E-E GAMING WIFI
-AMD Ryzen 9 7950X
-Kingston FURY Beast RGB 64GB (2x 32GB) DDR5-5200 KF552C40BBAK2-64
-Samsung 990 PRO, NVME
-Nvidia RTX 3060 Gigabyte OC LHR 12GB, OC-12GD 2.0

Proxmox version: newest stable release

---

First I installed Win 11 PRO with default wizard settings. After benchmarking disk performance I got only ~2 GB/s read instead of expected 7 GB/s. Root cause for this was IDE disk which was used by Proxmox wizard by default (which is OK to simplify Windows installation process without requiring additional drivers).

I then configured VirtIO SCSI controller by loading driver from virtio ISO image (pro tip: it is easier to do this during initial Windows install). After this I got ~7.4 GB/s read!

Then I configured Proxmox to use host processor and not default kvm64. Reason for this was because I wanted to use Docker Desktop and Windows subsystem for Linux in my Windows VM This required Hyper-V feature in Windows and enabled CPU virtualization. After switching to host CPU this worked great!!!

Next I wanted to use Nvidia RTX 3060 by using GPU passthrough. I followed instructions on internet but nothing worked... Nvidia driver in device manager always returned error
I almost lost hope on this but then I converted my CPU type from "host" back to default "kvm64". GPU was then magically detected, I got display output from VM on two monitors !!! I quickly tested random Steam game and it worked. Great success!

Issue with this configuration is that I will not be able to use Hyper-V because CPU type is not "host". This means I will have to use different Linux VM for Docker (not what I planned but OK).

And finally, I wanted to use USB speaker and other USB peripherals. I added USB speaker via add USB device in Proxmox. Sound worked but ... There was some noise/crackling sound
I resolved this by adding PCI passthrough for USB controller on motherboard. Now all devices attached to this controller are automatically detected and managed by Windows VM. Sound seems OK now

This is more or less it I managed to setup majority of things and hopefully nothing will break

 

[vdjur]

Hi,

I want to share another update, this time about "NVMe disk wear" issue. Maybe it will be helpful for someone

Currently I have two Windows 11 VMs and two Debian VMs. Today I noticed in Proxmox that my Samsung 990 PRO has 2% wear which seems really too much for such a short period. With this tempo it would wear out in less than one year...

I don't have ZFS on this drive, just VMs so I started shutting down VMs to see which one was causing problems. I was periodically measuring total bytes written with following command:

smartctl -A /dev/nvme0 | /bin/awk '$0~/Written/{ print strftime("%Y-%m-%d %H:%M:%S"), $3,$4,$5$6}'

It outputs something like this: 2022-12-30 09:30:25 Written: 2,875,596 [1.47TB]

My conclusion was that this problem was mainly visible for Windows VMs. I turned off paging file in Windows but it didn't help. In the end it seems to me that problem was caused by enabled "Ballooning Device" feature in Proxmox VM memory section (my Windows VMs had all the required drivers installed). This feature was turned on for all my VMs and after I turned it off for Windows VMs it seems to be much better now (it is still turned on for Debian VMs because it seems to be working OK there).

I will continue to monitor wear level but hopefully this was the problem.

 

[vdjur]

Unfortunately it seems write amplification from VMs is too high, I am currently at 2 TB writes without doing much ... I think I will go back to native Windows installation :]

 

[vdjur]

Abnormal disk wear of Samsung 990 PRO is probably due to Samsung drive issues that was reported by lots of users:

https://www.neowin.net/news/samsung...e-and-replace-990-pro-ssd-rapid-health-drops/
https://www.reddit.com/r/hardware/comments/10jkwwh/samsung_990_pro_ssd_with_rapid_health_drops/

980 PRO may also be affected, especially 2 TB version (Samsung issued firmware fix).

I would recommend to all users to temporarily wait with purchase of Samsung 980 / 990 PRO drives until Samsung resolves this issue.

 

[majorgear]

Hi,

I got the computer one week ago and here is my initial experience with Proxmox (TLDR: it is 99% what I wanted)

Relevant build components:
-ASUS ROG STRIX X670E-E GAMING WIFI
-AMD Ryzen 9 7950X
-Kingston FURY Beast RGB 64GB (2x 32GB) DDR5-5200 KF552C40BBAK2-64
-Samsung 990 PRO, NVME
-Nvidia RTX 3060 Gigabyte OC LHR 12GB, OC-12GD 2.0

Proxmox version: newest stable release

---

First I installed Win 11 PRO with default wizard settings. After benchmarking disk performance I got only ~2 GB/s read instead of expected 7 GB/s. Root cause for this was IDE disk which was used by Proxmox wizard by default (which is OK to simplify Windows installation process without requiring additional drivers).

I then configured VirtIO SCSI controller by loading driver from virtio ISO image (pro tip: it is easier to do this during initial Windows install). After this I got ~7.4 GB/s read!

Then I configured Proxmox to use host processor and not default kvm64. Reason for this was because I wanted to use Docker Desktop and Windows subsystem for Linux in my Windows VM This required Hyper-V feature in Windows and enabled CPU virtualization. After switching to host CPU this worked great!!!

Next I wanted to use Nvidia RTX 3060 by using GPU passthrough. I followed instructions on internet but nothing worked... Nvidia driver in device manager always returned error
I almost lost hope on this but then I converted my CPU type from "host" back to default "kvm64". GPU was then magically detected, I got display output from VM on two monitors !!! I quickly tested random Steam game and it worked. Great success!

Issue with this configuration is that I will not be able to use Hyper-V because CPU type is not "host". This means I will have to use different Linux VM for Docker (not what I planned but OK).

And finally, I wanted to use USB speaker and other USB peripherals. I added USB speaker via add USB device in Proxmox. Sound worked but ... There was some noise/crackling sound
I resolved this by adding PCI passthrough for USB controller on motherboard. Now all devices attached to this controller are automatically detected and managed by Windows VM. Sound seems OK now

This is more or less it I managed to setup majority of things and hopefully nothing will break

Did you need to configure the integrated GPU as the default in BIOS for PVE to only use that one for the OS/Server? And then the discrete GPU was available for use by VM's ?