Full GPU passthrough of 4x SXM5 H100 (Dell XE8640) to single VM not working

[mansoor9559]

Hi all, we are trying to setup one VM with 4 H100 (SXM5, on a Dell XE8640) GPUs as full pass through. But this is not happening.
Installing nvidia driver on Proxmox host gives us 4 GPUs but VM is unable to do the same.
We have tried all config for the VM which include BAR space and all sorts of other config.
Appreciate if you can provide any clue. Thanks

VM config

```

arch: x86_64
args: -fw_cfg opt/ovmf/X-PciMmio64Mb,string=2097152 -global q35-pcihost.pci-hole64-size=4096G
bios: ovmf
boot: order=scsi0;net0
cores: 8
cpu: host
efidisk0: pool1:vm-100-disk-0,efitype=4m,ms-cert=2023k,pre-enrolled-keys=1,size=1M
hostpci0: 0000:4e:00,pcie=1
hostpci1: 0000:5f:00,pcie=1
hostpci2: 0000:cb:00,pcie=1
hostpci3: 0000:db:00,pcie=1
machine: q35
memory: 65536
meta: creation-qemu=11.0.3,ctime=1787286120
name: test-01
net0: virtio=XX:XX:XX:XX:XX:XX,bridge=vmbr0,firewall=1
numa0: cpus=0-7,hostnodes=0,memory=32768,policy=bind
numa1: cpus=8-15,hostnodes=1,memory=32768,policy=bind
ostype: l26
scsi0: local-lvm:vm-100-disk-0,iothread=1,size=512G
scsihw: virtio-scsi-single
smbios1: uuid=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
sockets: 2
vmgenid: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
```

 

[driley]

hi,
did you follow the GPU Passthrough Guide?

I would avoid installing the NVIDIA driver directly on the host. When the host claims the GPUs, the VM cannot properly initialize them. You can remove the drivers by running:

apt remove nvidia-* --purge
apt remove nouveau* --purge
apt autoremove

Next, you need to blacklist these drivers so the host doesn't grab them on the next boot. Run these commands:

echo "blacklist nouveau" >> /etc/modprobe.d/blacklist.conf
echo "blacklist nvidia*" >> /etc/modprobe.d/blacklist.conf
update-initramfs -u -k all

After blacklisting you will need to reboot.

What also helps is creating a "PCI Device Mapping" you can do this by navigating to 'Datacenter' -> 'Resource Mappings' and add a PCI Devices here and select a H100 and use the "Pass through all functions as one device" option. Keep in mind that this won't resolve the issue.

What logs do you see on your host when starting the VM? You can follow the journalctl output by running:

journalctl -f

Does the VM boot? do you see any errors on screen? If it boots do you see any errors in the journalctl of the guest?
Which OS are you running in the VM and which drivers did you install?