Support for topology of multiple vGPUs?

[nielsh]

Hello,

We're looking into adding vGPUs to Proxmox for running AI Inference using vLLM. We did some tests with single-GPU servers and this works well.
However, we'd also like to be able to forward multiple vGPUs to a single VM. It is unknown to us how this works memory-wise.

We have tested 2 vGPUs using a single physical GPU, and it seems everything goes through the CPU:

        GPU0    GPU1    CPU Affinity    NUMA Affinity   GPU NUMA ID
GPU0     X      PHB     0-7     0               N/A
GPU1    PHB      X      0-7     0               N/A

Legend:

  X    = Self
  SYS  = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
  NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
  PHB  = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
  PXB  = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)
  PIX  = Connection traversing at most a single PCIe bridge
  NV#  = Connection traversing a bonded set of # NVLinks

This is extremely slow and makes it unfeasable to run LLMs large than the VRAM of a single GPU.
Does anyone know if I have multiple GPUs and attach them as vGPUs, that they can share memory bandwidth more efficiently? (i.e. the NODE, PXB or PIX methods)?

We'd like to run multiple RTX 6000 PRO Blackwell GPUs. They do not support NVLINK so we wouldn't be able to use that technology.

Thanks!