How to pass through 4+ GPUs?

2

240vac

New Member
Apr 29, 2019
4
0
1
26
I'm trying to pass through 3 K80s to a VM, which appear as 6 devices. I've just started using Proxmox, and the GUI option to add additional PCI devices after the 4th is grayed out. I found some blog posts online which modified some configuration files to make room for additional slots, but the option is still grayed out. Seems like there's got to be some GUI config somewhere that I may need to modify? Can anyone point me in the right direction?

Thanks!
 
OK, the documentation doesn't address the GUI portion but I did manage to get the devices passed through. However, GPU P2P doesn't seem to be enabled.
 
So my MXNet scripts call it GPU P2P, but it's specifically referenced in the documentation for GPUDirect
https://developer.nvidia.com/gpudirect

It essentially allows GPUs on the same PCIE bus to directly access memory from each other. There are a few reasons why it might not work between K80s, but since each single K80 is actually 2 devices connected internally with a PCIE bridge they should work perfectly.
 
can you post your vm config ?
 
bootdisk: scsi0
cores: 24
cpu: host
hostpci0: 3d:00,pcie=1
hostpci1: 3e:00,pcie=1
hostpci2: b1:00,pcie=1
hostpci3: b2:00,pcie=1
hostpci4: da:00,pcie=1
hostpci5: db:00,pcie=1
ide2: none,media=cdrom
machine: q35
memory: 18000
name: Ubuntu
net0: virtio=0E:07:F6:9F:E2:12,bridge=vmbr0,firewall=1
numa: 0
ostype: l26
scsi0: local-lvm:vm-100-disk-0,size=200G
scsihw: virtio-scsi-pci
smbios1: uuid=2db64ca3-6d7e-42ef-a753-aa0cbd9bbd45
sockets: 2
vga: virtio
vmgenid: 88d0eb13-7edf-4c2c-8a96-0423d0fb7ca
 
240vac said:
hostpci4: da:00,pcie=1
hostpci5: db:00,pcie=1
Click to expand...
our code only supports 4 hostpci devices (0-3) so i guess you patched our source for this to work?

i guesst the 3d/3e, b1/b2 etc belong together?

you could try to put multple devices in one hostpci device (they get then passed through as functions)

e.g.:

hostpci0: 3d:00.0;3e:00.0,pcie=1
hostpci1: b1:00.0;b2:00.0,pcie=1
 
You must log in or register to reply here.
Top Bottom