3 GPUs Proxmox workstation

[acidcrash]

Hi everyone, I'm writing here for the first time, so hopefully this is the right section where to ask.

I'd like to build a Proxmox workstation to host mainly 3 VMs:

• MacOS (my main operating system, I use it for browsing, email, coding, etc)
• Linux (also for work, same as above)
• Windows (for gaming)

The hardware I was planning to use is the following:

• CPU INTEL CORE I9 13900K 3,00 GHZ RAPTOR LAKE SOCKET 1700 BOX
• ASUS PROART Z690 CREATOR WIFI LGA 1700
• 2x RAM CRUCIAL DDR5 4800MHZ 64 GB (2X32GB)
• 2x SSD M.2 CRUCIAL 2TB P5 PLUS
• SAPPHIRE NITRO+ RADEON RX 6800 16GB GAMING GDDR6 (Windows)
• SAPPHIRE RADEON RX 6600 GAMING 8 GB GDDR6 (MacOS)
• ASUS RADEON RX 6400 PHOENIX 4GB GDDR6 (Linux)
• POWER SUPPLY - CORSAIR HX1000 HIGH PERFORMANCE

I've three main concerns which come from me being a bit rusty with custom pc builds (I use a Macbook for years now).

1. First one is regarding PCIE lanes. This motherboard I've chosen has enough PCI lanes for the hw I'd like to use? It has enough PCIE slots, but I'm not sure I can put all this hw together.
2. Do I really need the linux GPU or I could use the motherboard embedded one shared between Linux vm and Proxmox?
3. Do you see any issue with passing-through all these GPUs?

Apart from these two specific questions, do you have any useful suggestion?

 

[Dunuin]

First one is regarding PCIE lanes. This motherboard I've chosen has enough PCI lanes for the hw I'd like to use? It has enough PCIE slots, but I'm not sure I can put all this hw together.

You usually can only use PCI passthrough with PCIe slots that are directly connected to the CPU, not the ones connected to the chipset. That mainboard only got 3 PCIe slots and only two of them are connected to the CPU (and that only in 8x mode when using both slots). So I think that won't be possible to passthough all 3 GPUs. You usually want a xeon/threadripper/epyc platform with way more PCIe lanes for those things.

1. Do I really need the linux GPU or I could use the motherboard embedded one shared between Linux vm and Proxmox?

You can't share GPUs. If you passthrough that GPU then PVE got no GPU anymore. And often you won't be able to passthrough a GPU if it was initialized by the UEFI or PVE before. So that depends...you will have to test it with your hardware configuration.

 

[acidcrash]

You usually can only use PCI passthrough with PCIe slots that are directly connected to the CPU, not the ones connected to the chipset. That mainboard only got 3 PCIe slots and only two of them are connected to the CPU (and that only in 8x mode when using both slots). So I think that won't be possible to passthough all 3 GPUs. You usually want a xeon/threadripper/epyc platform with way more PCIe lanes for those things.

Oh that's useful, I thought also PCIe connected to chipset would allow passthrough. Anyway in my case I could probably do 8x8x4? From my understanding AMD GPUs also support PCIe 4x and since I wouldn't really need much GPU power to run linux for my needs, that maybe could work? Then I'd have the embedded GPU for Proxmox.

I know a threadripper would be the best choice...but they are damn expensive

You can't share GPUs. If you passthrough that GPU then PVE got no GPU anymore. And often you won't be able to passthrough a GPU if it was initialized by the UEFI or PVE before. So that depends...you will have to test it with your hardware configuration.

Maybe this could be useful in my case? https://wiki.archlinux.org/title/Intel_GVT-g