NVME passthrough

onepamopa

Member
Dec 1, 2019
81
11
13
36
I want to pass-through an NVME drive, the ID is 1bb1:5013 / 48:00.0

Its in it's own IOMMU group:
# find /sys/kernel/iommu_groups/ -type l |grep "53"
/sys/kernel/iommu_groups/53/devices/0000:48:00.0

Do I have to add 1bb1:5013 to pci-stub.ids= or vfio-pci.ids= in grub's config?

Also, How to add that device as the "primary" disk for a new VM (i.e. to be the boot drive & the OS drive)

The idea is to have as less virtualization as possible, the NVME should be directly managed by the OS installed on it.
 

avw

Renowned Member
May 31, 2020
1,025
177
63
It's a PCI(e) device, so if you want no emulation or overhead you can use PCI passthrough, but I don't know how to configure a VM to use it to boot.
 

onepamopa

Member
Dec 1, 2019
81
11
13
36
I figured things out:

1. add 1bb1:5013 to vfio-pci.ids in /etc/default/grub
2. add 1bb1:5013 to options vfio-pci ids= in /etc/modprobe.d/vfio.conf
3. When creating the VM, in UEFI de-select creation of EFI disk.
4. Add the NVME device passthrough first (hostpci0).

5. Add other passthrough devices (such as GPU, USB controller, etc) after it (had trouble booting when the nvme device was hostpci1/2 etc), this appeared in dmesg:
[ 2442.450870] pcieport 0000:40:01.4: DPC: containment event, status:0x1f05 source:0x4800
[ 2442.450871] pcieport 0000:40:01.4: DPC: ERR_FATAL detected
[ 2442.693466] pcieport 0000:40:01.4: AER: Device recovery successful

6. Adjust boot order - cd/dvd first, hostpci0 second.
 
Last edited:
  • Like
Reactions: avw

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get your own in 60 seconds.

Buy now!