Hardware for GPU Passthrough

[ZoThyx]

Hi everyone,
I have already tried to do GPU Passthrough, but without success, here is my configuration:
- Ryzen 5 5600G
- RX 6750 XT
- CoolerMaster 800+ Gold
- 2x8Go RAM
- Omen motherboard from a pre-build pc
The motherboard is very low end, so I can't unlock anything in bios. There is almost no option that I can enable/disable to be able to remove code 43 when installing drivers on a Windows VM.
So I'm looking for a compatible motherboard where I could enable/disable the following options that could potentially make GPU Passthrough work :
- Above 4G Decoding
- Resizable BAR (Smart/Clever Memory Access or SMA or CMA)

 

[Dunuin]

I would also upgrade RAM. 16GB isn't much. Especially when using PCI passthrough as all the RAM available to a VM will be used 100% of the time. So give that VM 12GB RAM (and even those 12GB are probably not that much if you do stuff that requires such a GPU) and there will be no RAM left for running any other VMs/LXCs. And when not running multiple VMs/LXCs its usually a better idea to use bare metal dualboot instead of virtualization if you only want to switch between different OSs.

 

[ZoThyx]

I would also upgrade RAM. 16GB isn't much. Especially when using PCI passthrough as all the RAM available to a VM will be used 100% of the time. So give that VM 12GB RAM (and even those 12GB are probably not that much if you do stuff that requires such a GPU) and there will be no RAM left for running any other VMs/LXCs. And when not running multiple VMs/LXCs its usually a better idea to use bare metal dualboot instead of virtualization if you only want to switch between different OSs.

The RAM is in my Amazon cart. How much did you recommend ? And the motherboard is probably an ATX or M-ATX !

 

[leesteken]

- Ryzen 5 5600G
So I'm looking for a compatible motherboard

The most passthrough friendly (almost all on-board devices and PCIe and M.2 slots are in separate IOMMU groups) for AM4 are X570 boards. ASRock Rack has nice a mATX server board. X370/X470 a two PCIe slots (but one when the CPU has integrated graphics) and B550 only only for passthrough (and always one M.2 slot).

The RAM is in my Amazon cart. How much did you recommend ? And the motherboard is probably an ATX or M-ATX !

I would advise unbuffered ECC DIMMs and buy a motherboad that supports it (most AM4 motherboard unofficially but not all, like MSI).

 

[Dunuin]

The RAM is in my Amazon cart. How much did you recommend ?

I personally would buy as much as the Mainboard does support. Or at least the least amount of modules that is required for multi channel with the biggest possible capacity that a module supports. So in case you got a dual channel motherboard with 4 RAM slots and the motherboard supports up to 128GB RAM, I would buy 2x 32GB modules to be able to make use of dual channel for better performance as well as having the option to add another 2x 32GB modules later in case you realize you are running out of RAM.
RAM is usually the thing that runs out first as you can't properly overprovision it, unlike storage or CPU.

And as leesteken already mentioned, if its possible to run ECC RAM, buy ECC RAM, in case you care about your data or stability.
I even bought ECC UDIMMs for my x570 gaming PC, as the previous Gaming PC without ECC RAM corrupted hundreds of GBs of files when the RAM was silently failing.
RAM got pretty cheap the last year. I bought my 16GB DDR4 3200MHz ECC UDIMMs for 92€ each. Now you get the same modules but with double the RAM (so 32GB DIMMs) for 70€ only.

 

[ZoThyx]

I take into account all your answers.
Can one of you give me the reference of a motherboard regardless of the brand as long as the passthrough works and that supports ddr4 ecc, as well as ddr4 ecc memory sticks.
I really don't want to be wrong when buying this since I also intend to add 2 NVME of 2 TB each for storage...

 

[leesteken]

Can one of you give me the reference of a motherboard regardless of the brand as long as the passthrough works and that supports ddr4 ecc, as well as ddr4 ecc memory sticks.

You could have searched the forum: https://forum.proxmox.com/threads/amd-motherboard-help.135569/post-599986

I really don't want to be wrong when buying this since I also intend to add 2 NVME of 2 TB each for storage...

Make sure to buy enterprise SSDs with PLP (as also advised many times on the forum), especially for ZFS.

EDIT: As I said AsRock Rack also has a nice server board, but I went for Gigabyte because they allow you to select the boot GPU from all x16 slots. Do note that I don't take responsibility for any problems you encounter for following my advise. I'm just a stranger on the internet, make sure to do your own research!

 

[ZoThyx]

Make sure to buy enterprise SSDs with PLP (as also advised many times on the forum), especially for ZFS.

I’m looking on 2 SN700 which are WD Red

 

[Dunuin]

I’m looking on 2 SN700 which are WD Red

Not bad SSDs, but still only pro-sumer and not enterprise grade. So not recommended for a ZFS Mirror because of the missing power-loss protection. Proper SSDs would be a Samsung PM9A3, Samsung PM983, Micron 7300 Pro, Micron 7450 Pro or Solidigm P4511 if you want a 2TB M.2 software raid1. There are only those 2 models that got M.2 + 2TB + PLP.

 

[ZoThyx]

On the French Amazon there is just one AsRock and it’s very expensive. But there is an GIGABYTE B550 AORUS Elite V2 Micro ATX or an MSI MPG B550 Gaming Plus. I probably don’t take ECC memory but I can do without.

 

[ZoThyx]

Not bad SSDs, but still only pro-sumer and not enterprise grade. So not recommended for a ZFS Mirror because of the missing power-loss protection. Proper SSDs would be a Samsung PM9A3, Samsung PM983, Micron 7300 Pro, Micron 7450 Pro or Solidigm P4511 if you want a 2TB M.2 software raid1. There are only those 2 models that got M.2 + 2TB + PLP.

I look at it right now.

 

[leesteken]

On the French Amazon there is just one AsRock and it’s very expensive. But there is an GIGABYTE B550 AORUS Elite V2 Micro ATX or an MSI MPG B550 Gaming Plus. I probably don’t take ECC memory but I can do without.

I was worried for a moment but at least you cannot hold us accountable if you ignore our advice. ;-)

 

[ZoThyx]

Under no circumstances can I hold you responsible, I'm just trying to make sure I don't spend for nothing. But I don't quite understand what role ecc memory plays in a passthrough GPU. In addition, is it really a passthrough if I use the graphics part of the CPU for proxmox and dedicate the entire graphics card to a VM?

 

[leesteken]

Under no circumstances can I hold you responsible,

You say that now, but people tend to not respond well when we point out they wasted money on QLC or SMR drives for example. Or how issues could have been avoided.

I'm just trying to make sure I don't spend for nothing.

I do try to suggest things that I think improves your setup the long run, based on my experience (on this forum and otherwise). Good stuff is indeed expensive.

But I don't quite understand what role ecc memory plays in a passthrough GPU.

It's not related but it does prevent issues in the long run. @Dunuin just pointed out a (not uncommon) problem that occurred when not using ECC.

In addition, is it really a passthrough if I use the graphics part of the CPU for proxmox and dedicate the entire graphics card to a VM?

Yes, passthrough is all about the passing of physical hardware to a virtual machine, being it a PCIe device, GPU, USB, disk, etc. Please read up on PCI(e) passthrough, for example by reading the manual: https://pve.proxmox.com/pve-docs/pve-admin-guide.html#qm_pci_passthrough.

Do with the advice what you will. Obviously there are constraints like time and money and availabilit. You only know your use case for running virtualization and have to work with the choices you will have made.

 

[ZoThyx]

It's not related but it does prevent issues in the long run. @Dunuin just pointed out a (not uncommon) problem that occurred when not using ECC.

But the 5600G does not support ECC memory... So I just can buy a b550 an keep my RAM without ECC ?!

 

[ZoThyx]

Not bad SSDs, but still only pro-sumer and not enterprise grade. So not recommended for a ZFS Mirror because of the missing power-loss protection. Proper SSDs would be a Samsung PM9A3, Samsung PM983, Micron 7300 Pro, Micron 7450 Pro or Solidigm P4511 if you want a 2TB M.2 software raid1. There are only those 2 models that got M.2 + 2TB + PLP.

I looked at the price of the NVMes you mentioned, and it's like 2 to 3 times more expensive than others.

 

[leesteken]

But the 5600G does not support ECC memory... So I just can buy a b550 an keep my RAM without ECC ?!

Yes, you can go without ECC (but I'm quite sure all Ryzen support ECC UDIMMs). B550 should supports one PCIe slot (and one M.2) for passthrough, which is also the max for your CPU. Hopefully it also supports passthrough of one USB controller. Don't expect much more from it though. Maybe search for your motherboard and VFIO or IOMMU to find out other peoples experiences on the internet with passthrough before buying.

I looked at the price of the NVMes you mentioned, and it's like 2 to 3 times more expensive than others.

You will probably be fine not having PLP and using LVM. Get your feet wet with virtualization and passthrough and upgrade when you run into bottlenecks later on.

 

[ZoThyx]

You say that now, but people tend to not respond well when we point out they wasted money on QLC or SMR drives for example. Or how issues could have been avoided.

I do try to suggest things that I think improves your setup the long run, based on my experience (on this forum and otherwise). Good stuff is indeed expensive.

It's not related but it does prevent issues in the long run. @Dunuin just pointed out a (not uncommon) problem that occurred when not using ECC.

Yes, passthrough is all about the passing of physical hardware to a virtual machine, being it a PCIe device, GPU, USB, disk, etc. Please read up on PCI(e) passthrough, for example by reading the manual: https://pve.proxmox.com/pve-docs/pve-admin-guide.html#qm_pci_passthrough.

Do with the advice what you will. Obviously there are constraints like time and money and availabilit. You only know your use case for running virtualization and have to work with the choices you will have made.

I understand very well everything you tell me. But the goal is not to have a giga setup, I just want a vm that allows you to play games like GTA or RDR2. And storage is only for files, I don't have thousands of them and are all backed up on an external hard drive. So I don't think I need a whole setup worthy of a company for this?

Thank you for your answer, I will try to find people who have passed their GPU Passthrough to inform me about their motherboard. Unless you have an idea in mind about a model that might be suitable. In any case, I would inquire before ordering

 

[leesteken]

play games like GTA or RDR2

I don't know about those games, but you might want to make sure their copyright protection and/or multiplayer anti-cheat does not prevent you from running in a VM. There are some threads here about people having problems playing (mostly competitive multiplayer) games because some software detects that it's inside a VM.

 

[ZoThyx]

I don't know about those games, but you might want to make sure their copyright protection and/or multiplayer anti-cheat does not prevent you from running in a VM. There are some threads here about people having problems playing (mostly competitive multiplayer) games because some software detects that it's inside a VM.

It’s not for playing on multiplayer, it to do campaign/history. Before the 6750 was in my gaming pc and playing very fine on 2k. So I think it will be fine to play on 1080p inside a VM