How to pass through 4+ GPUs?

[240vac]

I'm trying to pass through 3 K80s to a VM, which appear as 6 devices. I've just started using Proxmox, and the GUI option to add additional PCI devices after the 4th is grayed out. I found some blog posts online which modified some configuration files to make room for additional slots, but the option is still grayed out. Seems like there's got to be some GUI config somewhere that I may need to modify? Can anyone point me in the right direction?

Thanks!

 

[Chriswiss]

Hello,

You will find a lot of information on the official documentation.

https://pve.proxmox.com/wiki/Pci_passthrough

CG

 

[240vac]

OK, the documentation doesn't address the GUI portion but I did manage to get the devices passed through. However, GPU P2P doesn't seem to be enabled.

 

[dcsapak]

GPU P2P

what do you mean with that?

 

[240vac]

So my MXNet scripts call it GPU P2P, but it's specifically referenced in the documentation for GPUDirect
https://developer.nvidia.com/gpudirect

It essentially allows GPUs on the same PCIE bus to directly access memory from each other. There are a few reasons why it might not work between K80s, but since each single K80 is actually 2 devices connected internally with a PCIE bridge they should work perfectly.

 

[dcsapak]

can you post your vm config ?

 

[240vac]

bootdisk: scsi0
cores: 24
cpu: host
hostpci0: 3d:00,pcie=1
hostpci1: 3e:00,pcie=1
hostpci2: b1:00,pcie=1
hostpci3: b2:00,pcie=1
hostpci4: da:00,pcie=1
hostpci5: db:00,pcie=1
ide2: none,media=cdrom
machine: q35
memory: 18000
name: Ubuntu
net0: virtio=0E:07:F6:9F:E2:12,bridge=vmbr0,firewall=1
numa: 0
ostype: l26
scsi0: local-lvm:vm-100-disk-0,size=200G
scsihw: virtio-scsi-pci
smbios1: uuid=2db64ca3-6d7e-42ef-a753-aa0cbd9bbd45
sockets: 2
vga: virtio
vmgenid: 88d0eb13-7edf-4c2c-8a96-0423d0fb7ca

 

[dcsapak]

hostpci4: da:00,pcie=1
hostpci5: db:00,pcie=1

our code only supports 4 hostpci devices (0-3) so i guess you patched our source for this to work?

i guesst the 3d/3e, b1/b2 etc belong together?

you could try to put multple devices in one hostpci device (they get then passed through as functions)

e.g.:

hostpci0: 3d:00.0;3e:00.0,pcie=1
hostpci1: b1:00.0;b2:00.0,pcie=1