PMC-Sierra Device can't PCI Passthrough

N

niMaDer

New Member
Apr 8, 2023
4
0
1
Bash: 
# pveversion
pve-manager/8.1.4/ec5affc9e41f1d79 (running kernel: 6.5.11-8-pve)

I have Apex Storage X21 card, with 16 * nvme m.2
But i can't PCI Passthrough this card to VM.

Bash: 
# cat /etc/default/grub
...
GRUB_CMDLINE_LINUX_DEFAULT="quiet amd_iommu=on iommu=pt pcie_acs_override=downstream"
...

Bash: 
# lspci
...
81:00.0 PCI bridge: PMC-Sierra Inc. Device 4000
81:00.1 Memory controller: PMC-Sierra Inc. Device 4000
82:00.0 PCI bridge: PMC-Sierra Inc. Device 4000
82:01.0 PCI bridge: PMC-Sierra Inc. Device 4000
82:02.0 PCI bridge: PMC-Sierra Inc. Device 4000
82:03.0 PCI bridge: PMC-Sierra Inc. Device 4000
82:04.0 PCI bridge: PMC-Sierra Inc. Device 4000
82:05.0 PCI bridge: PMC-Sierra Inc. Device 4000
82:06.0 PCI bridge: PMC-Sierra Inc. Device 4000
82:07.0 PCI bridge: PMC-Sierra Inc. Device 4000
82:08.0 PCI bridge: PMC-Sierra Inc. Device 4000
82:09.0 PCI bridge: PMC-Sierra Inc. Device 4000
82:0a.0 PCI bridge: PMC-Sierra Inc. Device 4000
82:0b.0 PCI bridge: PMC-Sierra Inc. Device 4000
82:0c.0 PCI bridge: PMC-Sierra Inc. Device 4000
82:0d.0 PCI bridge: PMC-Sierra Inc. Device 4000
82:0e.0 PCI bridge: PMC-Sierra Inc. Device 4000
82:0f.0 PCI bridge: PMC-Sierra Inc. Device 4000
82:10.0 PCI bridge: PMC-Sierra Inc. Device 4000
82:11.0 PCI bridge: PMC-Sierra Inc. Device 4000
82:12.0 PCI bridge: PMC-Sierra Inc. Device 4000
82:13.0 PCI bridge: PMC-Sierra Inc. Device 4000
82:14.0 PCI bridge: PMC-Sierra Inc. Device 4000
83:00.0 Non-Volatile memory controller: Realtek Semiconductor Co., Ltd. RTS5763DL NVMe SSD Controller (rev 01)
85:00.0 Non-Volatile memory controller: Realtek Semiconductor Co., Ltd. RTS5763DL NVMe SSD Controller (rev 01)
87:00.0 Non-Volatile memory controller: Realtek Semiconductor Co., Ltd. RTS5763DL NVMe SSD Controller (rev 01)
89:00.0 Non-Volatile memory controller: Realtek Semiconductor Co., Ltd. RTS5763DL NVMe SSD Controller (rev 01)
8b:00.0 Non-Volatile memory controller: Realtek Semiconductor Co., Ltd. RTS5763DL NVMe SSD Controller (rev 01)
8c:00.0 Non-Volatile memory controller: Realtek Semiconductor Co., Ltd. RTS5763DL NVMe SSD Controller (rev 01)
8d:00.0 Non-Volatile memory controller: Realtek Semiconductor Co., Ltd. RTS5763DL NVMe SSD Controller (rev 01)
8e:00.0 Non-Volatile memory controller: Realtek Semiconductor Co., Ltd. RTS5763DL NVMe SSD Controller (rev 01)
8f:00.0 Non-Volatile memory controller: Realtek Semiconductor Co., Ltd. RTS5763DL NVMe SSD Controller (rev 01)
90:00.0 Non-Volatile memory controller: Realtek Semiconductor Co., Ltd. RTS5763DL NVMe SSD Controller (rev 01)
91:00.0 Non-Volatile memory controller: Realtek Semiconductor Co., Ltd. RTS5763DL NVMe SSD Controller (rev 01)
92:00.0 Non-Volatile memory controller: Realtek Semiconductor Co., Ltd. RTS5763DL NVMe SSD Controller (rev 01)
93:00.0 Non-Volatile memory controller: Realtek Semiconductor Co., Ltd. RTS5763DL NVMe SSD Controller (rev 01)
94:00.0 Non-Volatile memory controller: Realtek Semiconductor Co., Ltd. RTS5763DL NVMe SSD Controller (rev 01)
95:00.0 Non-Volatile memory controller: Realtek Semiconductor Co., Ltd. RTS5763DL NVMe SSD Controller (rev 01)
96:00.0 Non-Volatile memory controller: Realtek Semiconductor Co., Ltd. RTS5763DL NVMe SSD Controller (rev 01)
...

Bash: 
# lspci -v -s 81:00.0
81:00.0 PCI bridge: PMC-Sierra Inc. Device 4000 (prog-if 00 [Normal decode])
        Subsystem: PMC-Sierra Inc. Device beef
        Physical Slot: 23
        Flags: bus master, fast devsel, latency 0, IOMMU group 37
        Bus: primary=81, secondary=82, subordinate=97, sec-latency=0
        I/O behind bridge: [disabled] [32-bit]
        Memory behind bridge: c9e00000-cadfffff [size=16M] [32-bit]
        Prefetchable memory behind bridge: [disabled] [64-bit]
        Capabilities: [40] Express Upstream Port, MSI 00
        Capabilities: [7c] MSI: Enable- Count=1/8 Maskable- 64bit+
        Capabilities: [8c] Power Management version 3
        Capabilities: [94] Subsystem: PMC-Sierra Inc. Device beef
        Capabilities: [100] Advanced Error Reporting
        Capabilities: [148] Power Budgeting <?>
        Capabilities: [158] Multicast
        Capabilities: [188] Secondary PCI Express
        Capabilities: [1b4] Device Serial Number 50-0e-00-4a-00-00-00-01
        Capabilities: [1c0] Latency Tolerance Reporting
        Capabilities: [1c8] Access Control Services
        Capabilities: [1f0] Data Link Feature <?>
        Capabilities: [1fc] Physical Layer 16.0 GT/s <?>
        Capabilities: [23c] Lane Margining at the Receiver <?>
        Capabilities: [7f8] Vendor Specific Information: ID=ffff Rev=1 Len=808 <?>
        Kernel driver in use: pcieport

# lspci -v -s 81:00.1
81:00.1 Memory controller: PMC-Sierra Inc. Device 4000
        Subsystem: PMC-Sierra Inc. Device 4000
        Physical Slot: 23
        Flags: fast devsel, IOMMU group 38
        Memory at 47dffc00000 (64-bit, prefetchable) [disabled] [size=4M]
        Capabilities: [40] MSI: Enable- Count=1/4 Maskable- 64bit+
        Capabilities: [50] MSI-X: Enable- Count=4 Masked-
        Capabilities: [5c] Power Management version 3
        Capabilities: [64] Express Endpoint, MSI 00
        Capabilities: [100] Advanced Error Reporting
        Capabilities: [148] Device Serial Number 50-0e-00-4a-00-00-00-01
        Capabilities: [154] Access Control Services
        Kernel driver in use: vfio-pci
        Kernel modules: switchtec

Bash: 
# cat /etc/modprobe.d/vfio.conf
options vfio-pci ids=11f8:4000,11f8:beef

# dmesg | grep -i vfio
[  127.634803] VFIO - User Level meta-driver version: 0.3
[  127.652410] vfio_pci: add [11f8:4000[ffffffff:ffffffff]] class 0x000000/00000000
[  127.652440] vfio_pci: add [11f8:beef[ffffffff:ffffffff]] class 0x000000/00000000

Bash: 
# qm show 100 --pretty
/usr/bin/kvm \
  -id 100 \
  -name 'TrueNAS,debug-threads=on' \
  -no-shutdown \
  -chardev 'socket,id=qmp,path=/var/run/qemu-server/100.qmp,server=on,wait=off' \
  -mon 'chardev=qmp,mode=control' \
  -chardev 'socket,id=qmp-event,path=/var/run/qmeventd.sock,reconnect=5' \
  -mon 'chardev=qmp-event,mode=control' \
  -pidfile /var/run/qemu-server/100.pid \
  -daemonize \
  -smbios 'type=1,uuid=06ff3a96-c98b-4cd6-9a23-e6bb0a75d6af' \
  -drive 'if=pflash,unit=0,format=raw,readonly=on,file=/usr/share/pve-edk2-firmware//OVMF_CODE_4M.secboot.fd' \
  -drive 'if=pflash,unit=1,id=drive-efidisk0,format=raw,file=/dev/zvol/rpool/data/vm-100-disk-2,size=540672' \
  -smp '16,sockets=1,cores=16,maxcpus=16' \
  -nodefaults \
  -boot 'menu=on,strict=on,reboot-timeout=1000,splash=/usr/share/qemu-server/bootsplash.jpg' \
  -vnc 'unix:/var/run/qemu-server/100.vnc,password=on' \
  -cpu host,+kvm_pv_eoi,+kvm_pv_unhalt \
  -m 102400 \
  -object 'iothread,id=iothread-virtioscsi0' \
  -object 'iothread,id=iothread-virtioscsi1' \
  -readconfig /usr/share/qemu-server/pve-q35-4.0.cfg \
  -device 'vmgenid,guid=4c987853-1ba5-43d5-9286-41223343dda7' \
  -device 'usb-tablet,id=tablet,bus=ehci.0,port=1' \
  -device 'vfio-pci,host=0000:81:00.0,id=hostpci0,bus=ich9-pcie-port-1,addr=0x0' \
  -device 'vfio-pci,host=0000:81:00.1,id=hostpci1,bus=ich9-pcie-port-2,addr=0x0' \
  -device 'VGA,id=vga,bus=pcie.0,addr=0x1' \
  -chardev 'socket,path=/var/run/qemu-server/100.qga,server=on,wait=off,id=qga0' \
  -device 'virtio-serial,id=qga0,bus=pci.0,addr=0x8' \
  -device 'virtserialport,chardev=qga0,name=org.qemu.guest_agent.0' \
  -device 'virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x3,free-page-reporting=on' \
  -iscsi 'initiator-name=iqn.1993-08.org.debian:01:ded7f741dbbc' \
  -device 'virtio-scsi-pci,id=virtioscsi0,bus=pci.3,addr=0x1,iothread=iothread-virtioscsi0' \
  -drive 'file=/dev/zvol/rpool/data/vm-10-disk-0,if=none,id=drive-scsi0,cache=writeback,discard=on,format=raw,aio=io_uring,detect-zeroes=unmap' \
  -device 'scsi-hd,bus=virtioscsi0.0,channel=0,scsi-id=0,lun=0,drive=drive-scsi0,id=scsi0,rotation_rate=1,bootindex=100,serial=0001' \
  -device 'virtio-scsi-pci,id=virtioscsi1,bus=pci.3,addr=0x2,iothread=iothread-virtioscsi1' \
  -drive 'file=/dev/zvol/rpool/data/vm-100-disk-1,if=none,id=drive-scsi1,cache=writeback,discard=on,format=raw,aio=io_uring,detect-zeroes=unmap' \
  -device 'scsi-hd,bus=virtioscsi1.0,channel=0,scsi-id=0,lun=1,drive=drive-scsi1,id=scsi1,rotation_rate=1,serial=0002' \
  -netdev 'type=tap,id=net0,ifname=tap100i0,script=/var/lib/qemu-server/pve-bridge,downscript=/var/lib/qemu-server/pve-bridgedown,vhost=on' \
  -device 'virtio-net-pci,mac=62:F5:56:82:0E:AD,netdev=net0,bus=pci.0,addr=0x12,id=net0,rx_queue_size=1024,tx_queue_size=256' \
  -machine 'type=q35+pve0'


When start vm, got this error:
Code: 
kvm: -device vfio-pci,host=0000:81:00.0,id=hostpci0,bus=ich9-pcie-port-1,addr=0x0: vfio 0000:81:00.0: failed to open /dev/vfio/37: No such file or directory
TASK ERROR: start failed: QEMU exited with code 1


Please help me
 
Last edited:
You must log in or register to reply here.
Top Bottom
Back