NVME passthrough

[onepamopa]

I want to pass-through an NVME drive, the ID is 1bb1:5013 / 48:00.0

Its in it's own IOMMU group:
# find /sys/kernel/iommu_groups/ -type l |grep "53"
/sys/kernel/iommu_groups/53/devices/0000:48:00.0

Do I have to add 1bb1:5013 to pci-stub.ids= or vfio-pci.ids= in grub's config?

Also, How to add that device as the "primary" disk for a new VM (i.e. to be the boot drive & the OS drive)

The idea is to have as less virtualization as possible, the NVME should be directly managed by the OS installed on it.

 

[onepamopa]

Anyone on this? How to use the NVME "raw" w/o any emulation?

 

[leesteken]

It's a PCI(e) device, so if you want no emulation or overhead you can use PCI passthrough, but I don't know how to configure a VM to use it to boot.

 

[onepamopa]

I figured things out:

1. add 1bb1:5013 to vfio-pci.ids in /etc/default/grub
2. add 1bb1:5013 to options vfio-pci ids= in /etc/modprobe.d/vfio.conf
3. When creating the VM, in UEFI de-select creation of EFI disk.
4. Add the NVME device passthrough first (hostpci0).

5. Add other passthrough devices (such as GPU, USB controller, etc) after it (had trouble booting when the nvme device was hostpci1/2 etc), this appeared in dmesg:
[ 2442.450870] pcieport 0000:40:01.4: DPC: containment event, status:0x1f05 source:0x4800
[ 2442.450871] pcieport 0000:40:01.4: DPC: ERR_FATAL detected
[ 2442.693466] pcieport 0000:40:01.4: AER: Device recovery successful

6. Adjust boot order - cd/dvd first, hostpci0 second.

 

[solonion]

I'm following this guide to pass nvme to my vm, and I want to add some tips.
For systemd-boot (zfs), the first step should be add vfio-pci.ids=1d97:1602 to /etc/kernel/cmdline and do proxmox-boot-tool refresh
And for the second step, softdep nvme pre: vfio-pci also need to be added in my server or else nvme driver will be used instead of vfio-pci

 

[onepamopa]

I'm following this guide to pass nvme to my vm, and I want to add some tips.
For systemd-boot (zfs), the first step should be add vfio-pci.ids=1d97:1602 to /etc/kernel/cmdline and do proxmox-boot-tool refresh
And for the second step, softdep nvme pre: vfio-pci also need to be added in my server or else nvme driver will be used instead of vfio-pci

Where did you add softdep nvme pre: vfio-pci ?
Upon checking lspci, I see the NVME I've passed through shows as:

Kernel driver in use: vfio-pci
Kernel modules: nvme

Also, I have 2 NVMe's (one is for lvm storage, the other one - passed through) softdep nvme pre: vfio-pci won't break that?