How to pass through 4+ GPUs?


New Member
Apr 29, 2019
I'm trying to pass through 3 K80s to a VM, which appear as 6 devices. I've just started using Proxmox, and the GUI option to add additional PCI devices after the 4th is grayed out. I found some blog posts online which modified some configuration files to make room for additional slots, but the option is still grayed out. Seems like there's got to be some GUI config somewhere that I may need to modify? Can anyone point me in the right direction?

OK, the documentation doesn't address the GUI portion but I did manage to get the devices passed through. However, GPU P2P doesn't seem to be enabled.
So my MXNet scripts call it GPU P2P, but it's specifically referenced in the documentation for GPUDirect

It essentially allows GPUs on the same PCIE bus to directly access memory from each other. There are a few reasons why it might not work between K80s, but since each single K80 is actually 2 devices connected internally with a PCIE bridge they should work perfectly.
can you post your vm config ?
bootdisk: scsi0
cores: 24
cpu: host
hostpci0: 3d:00,pcie=1
hostpci1: 3e:00,pcie=1
hostpci2: b1:00,pcie=1
hostpci3: b2:00,pcie=1
hostpci4: da:00,pcie=1
hostpci5: db:00,pcie=1
ide2: none,media=cdrom
machine: q35
memory: 18000
name: Ubuntu
net0: virtio=0E:07:F6:9F:E2:12,bridge=vmbr0,firewall=1
numa: 0
ostype: l26
scsi0: local-lvm:vm-100-disk-0,size=200G
scsihw: virtio-scsi-pci
smbios1: uuid=2db64ca3-6d7e-42ef-a753-aa0cbd9bbd45
sockets: 2
vga: virtio
vmgenid: 88d0eb13-7edf-4c2c-8a96-0423d0fb7ca
hostpci4: da:00,pcie=1
hostpci5: db:00,pcie=1
our code only supports 4 hostpci devices (0-3) so i guess you patched our source for this to work?

i guesst the 3d/3e, b1/b2 etc belong together?

you could try to put multple devices in one hostpci device (they get then passed through as functions)


hostpci0: 3d:00.0;3e:00.0,pcie=1
hostpci1: b1:00.0;b2:00.0,pcie=1


The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!