HBA passthrough to FreeNAS fails

Y

yantramanav

New Member
May 7, 2020
6
1
3
42
Berlin
Hi,

I'm trying to setup FreeNAS inside proxmox (v6.1-11), and I can't seem to pass the PCIe HBA to the freeNAS. If I attach the PCI device to the FreeNAS VM, the proxmox system hangs and the screen is filled with Error: Buffer I/O error on dev dm-7

I'm running AMD Ryzen 3 1200 on Asus 320M-K mother board. The HBA PCIe card is QNINE PCIe SATA card which runs the Marvell 88SE9215 chip.

Here's my various configurations:
Grub:

GRUB_CMDLINE_LINUX_DEFAULT="quiet amd_iommu=on iommu=pt"

Code: 
root@bakasur ~# cat /etc/modprobe.d/pve-blacklist.conf

# This file contains a list of modules which are not supported by Proxmox VE
# nidiafb see bugreport https://bugzilla.proxmox.com/show_bug.cgi?id=701
blacklist nvidiafb
# lspci -knn
Code: 
00:00.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Root Complex [1022:1450]
    Subsystem: ASUSTeK Computer Inc. Family 17h (Models 00h-0fh) Root Complex [1043:8747]
00:00.2 IOMMU [0806]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) I/O Memory Management Unit [1022:1451]
    Subsystem: ASUSTeK Computer Inc. Family 17h (Models 00h-0fh) I/O Memory Management Unit [1043:8747]
00:01.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-1fh) PCIe Dummy Host Bridge [1022:1452]
00:01.3 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) PCIe GPP Bridge [1022:1453]
    Kernel driver in use: pcieport
00:02.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-1fh) PCIe Dummy Host Bridge [1022:1452]
00:03.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-1fh) PCIe Dummy Host Bridge [1022:1452]
00:03.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) PCIe GPP Bridge [1022:1453]
    Kernel driver in use: pcieport
00:04.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-1fh) PCIe Dummy Host Bridge [1022:1452]
00:07.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-1fh) PCIe Dummy Host Bridge [1022:1452]
00:07.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Internal PCIe GPP Bridge 0 to Bus B [1022:1454]
    Kernel driver in use: pcieport
00:08.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-1fh) PCIe Dummy Host Bridge [1022:1452]
00:08.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Internal PCIe GPP Bridge 0 to Bus B [1022:1454]
    Kernel driver in use: pcieport
00:14.0 SMBus [0c05]: Advanced Micro Devices, Inc. [AMD] FCH SMBus Controller [1022:790b] (rev 59)
    Subsystem: ASUSTeK Computer Inc. FCH SMBus Controller [1043:8747]
    Kernel driver in use: piix4_smbus
    Kernel modules: i2c_piix4, sp5100_tco
00:14.3 ISA bridge [0601]: Advanced Micro Devices, Inc. [AMD] FCH LPC Bridge [1022:790e] (rev 51)
    Subsystem: ASUSTeK Computer Inc. FCH LPC Bridge [1043:8747]
00:18.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 0 [1022:1460]
00:18.1 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 1 [1022:1461]
00:18.2 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 2 [1022:1462]
00:18.3 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 3 [1022:1463]
    Kernel driver in use: k10temp
    Kernel modules: k10temp
00:18.4 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 4 [1022:1464]
00:18.5 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 5 [1022:1465]
00:18.6 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 6 [1022:1466]
00:18.7 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 7 [1022:1467]
01:00.0 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Device [1022:43bc] (rev 02)
    Subsystem: ASMedia Technology Inc. Device [1b21:1142]
    Kernel driver in use: xhci_hcd
01:00.1 SATA controller [0106]: Advanced Micro Devices, Inc. [AMD] Device [1022:43b8] (rev 02)
    Subsystem: ASMedia Technology Inc. Device [1b21:1062]
    Kernel driver in use: ahci
    Kernel modules: ahci
01:00.2 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:43b3] (rev 02)
    Kernel driver in use: pcieport
02:04.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 300 Series Chipset PCIe Port [1022:43b4] (rev 02)
    Kernel driver in use: pcieport
02:05.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 300 Series Chipset PCIe Port [1022:43b4] (rev 02)
    Kernel driver in use: pcieport
02:06.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 300 Series Chipset PCIe Port [1022:43b4] (rev 02)
    Kernel driver in use: pcieport
02:07.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 300 Series Chipset PCIe Port [1022:43b4] (rev 02)
    Kernel driver in use: pcieport
03:00.0 SATA controller [0106]: Marvell Technology Group Ltd. Device [1b4b:9215] (rev 11)
    Subsystem: Marvell Technology Group Ltd. Device [1b4b:9215]
    Kernel driver in use: ahci
    Kernel modules: ahci
05:00.0 Ethernet controller [0200]: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller [10ec:8168] (rev 15)
    Subsystem: ASUSTeK Computer Inc. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller [1043:8677]
    Kernel driver in use: r8169
    Kernel modules: r8169
07:00.0 VGA compatible controller [0300]: NVIDIA Corporation GK208 [GeForce GT 710B] [10de:128b] (rev a1)
    Subsystem: ASUSTeK Computer Inc. GK208B [GeForce GT 710] [1043:86cb]
    Kernel driver in use: nvidia
    Kernel modules: nvidiafb, nouveau, nvidia_drm, nvidia
07:00.1 Audio device [0403]: NVIDIA Corporation GK208 HDMI/DP Audio Controller [10de:0e0f] (rev a1)
    Subsystem: ASUSTeK Computer Inc. GK208 HDMI/DP Audio Controller [1043:86cb]
    Kernel driver in use: snd_hda_intel
    Kernel modules: snd_hda_intel
08:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Zeppelin/Raven/Raven2 PCIe Dummy Function [1022:145a]
    Subsystem: Advanced Micro Devices, Inc. [AMD] Zeppelin/Raven/Raven2 PCIe Dummy Function [1022:145a]
08:00.2 Encryption controller [1080]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Platform Security Processor [1022:1456]
    Subsystem: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Platform Security Processor [1022:1456]
    Kernel driver in use: ccp
    Kernel modules: ccp
08:00.3 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) USB 3.0 Host Controller [1022:145c]
    Subsystem: ASUSTeK Computer Inc. Family 17h (Models 00h-0fh) USB 3.0 Host Controller [1043:8747]
    Kernel driver in use: xhci_hcd
09:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Zeppelin/Renoir PCIe Dummy Function [1022:1455]
    Subsystem: Advanced Micro Devices, Inc. [AMD] Zeppelin/Renoir PCIe Dummy Function [1022:1455]
09:00.2 SATA controller [0106]: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] [1022:7901] (rev 51)
    Subsystem: ASUSTeK Computer Inc. FCH SATA Controller [AHCI mode] [1043:8747]
    Kernel driver in use: ahci
    Kernel modules: ahci
09:00.3 Audio device [0403]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) HD Audio Controller [1022:1457]
    Subsystem: ASUSTeK Computer Inc. Family 17h (Models 00h-0fh) HD Audio Controller [1043:86c7]
    Kernel driver in use: snd_hda_intel
    Kernel modules: snd_hda_intel

VFIO information
Code: 
root@bakasur ~# lspci -n -s 03:00
03:00.0 0106: 1b4b:9215 (rev 11)

root@bakasur ~# cat /etc/modprobe.d/vfio.conf 
options vfio-pci ids=1b4b:9215

Finally the guest VM (FreeNAS) config

Code: 
bios: ovmf
bootdisk: scsi0
cores: 4
efidisk0: local-lvm:vm-101-disk-1,size=4M
machine: q35
memory: 8048
name: freeNAS
net0: virtio=62:6A:EB:5C:16:9C,bridge=vmbr0,firewall=1
numa: 0
ostype: l26
scsi0: local-lvm:vm-101-disk-0,size=20G
scsihw: virtio-scsi-pci
smbios1: uuid=974a63d8-8c62-4558-b83a-77267bfde80f
sockets: 1
vmgenid: 3d3eecac-7435-4427-ad04-c6e6789658f4

If I add hostpci0: 03:00.0,rombar=0 or similar, the proxmox server hangs.

How can I blacklist the Marvell driver on proxmox and share it with the freeNAS?
The Marvell SATA controller is not in it's own IOMMU but I tried switching the PCIe slots, and it stays grouped with other devices.
 
yantramanav said:
I'm sorry, I didn't understand this, could you please explain?
Click to expand...

If you passthrough a device to a VM and there is already a driver loaded for the device, the device is unregistered from the driver, meaning that the device is not longer attached to the driver and as the device is free from a driver it is deregistred from the PCI bus and attached to the bus in your VM. This is transparent for you and therefore a passthrough-VM takes longer to start and a fully-virtualized VM. If you don't like the approach, look at the Wiki article I posted, there is also driver and device blacklisting explained.

yantramanav said:
I can see that the proxmox is using the PCIe card correctly, and it detects the HDDs attached to it. WHich tweaks from the wiki should I try in particular?
Click to expand...

All of them :-D
Getting it to work can be very tricky. The main part is the IOMMU group. You normally cannot divide a group, so you have glitches etc. If you cannot separate it further, your platform simply cannot do it as you like. That's the sad truth.

Is your hardware/bios equipped with a setting for IOMMU virtualization? If so, it is enabled?
 
Thanks for you help, much appreciated.

I just managed to group the Marvell card under it's own IOMMU group by chaning the grub line to:
GRUB_CMDLINE_LINUX_DEFAULT="quiet amd_iommu=on iommu=pt pcie_acs_override=downstream,multifunction
I can see that the SATA controller is in it's own group.

Code: 
...
IOMMU Group 18:
    02:07.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 300 Series Chipset PCIe Port [1022:43b4] (rev 02)
IOMMU Group 19:
    03:00.0 SATA controller [0106]: Marvell Technology Group Ltd. Device [1b4b:9215] (rev 11)
IOMMU Group 2:
    00:02.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-1fh) PCIe Dummy Host Bridge [1022:1452]
...

I can now attach the PCIe SATA controller to the FreeNAS VM, and it starts fine and doesn't crash the whole proxmox install. However the FreeNAS can't see the SATA disks. This is the output of the lspci from within the Guest VM.

Code: 
root@freenas[~]# lspci
00:00.0 Host bridge: Intel Corporation 82G33/G31/P35/P31 Express DRAM Controller
00:01.0 VGA compatible controller: Device 1234:1111 (rev 02)
00:1a.0 USB controller: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #4 (rev 03)
00:1a.1 USB controller: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #5 (rev 03)
00:1a.2 USB controller: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #6 (rev 03)
00:1a.7 USB controller: Intel Corporation 82801I (ICH9 Family) USB2 EHCI Controller #2 (rev 03)
00:1b.0 Audio device: Intel Corporation 82801I (ICH9 Family) HD Audio Controller (rev 03)
00:1c.0 PCI bridge: Red Hat, Inc. QEMU PCIe Root port
00:1c.1 PCI bridge: Red Hat, Inc. QEMU PCIe Root port
00:1c.2 PCI bridge: Red Hat, Inc. QEMU PCIe Root port
00:1c.3 PCI bridge: Red Hat, Inc. QEMU PCIe Root port
00:1d.0 USB controller: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #1 (rev 03)
00:1d.1 USB controller: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #2 (rev 03)
00:1d.2 USB controller: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #3 (rev 03)
00:1d.7 USB controller: Intel Corporation 82801I (ICH9 Family) USB2 EHCI Controller #1 (rev 03)
00:1e.0 PCI bridge: Intel Corporation 82801 PCI Bridge (rev 92)
00:1f.0 ISA bridge: Intel Corporation 82801IB (ICH9) LPC Interface Controller (rev 02)
00:1f.2 SATA controller: Intel Corporation 82801IR/IO/IH (ICH9R/DO/DH) 6 port SATA Controller [AHCI mode] (rev 02)
00:1f.3 SMBus: Intel Corporation 82801I (ICH9 Family) SMBus Controller (rev 02)
05:01.0 PCI bridge: Red Hat, Inc. QEMU PCI-PCI bridge
05:02.0 PCI bridge: Red Hat, Inc. QEMU PCI-PCI bridge
05:03.0 PCI bridge: Red Hat, Inc. QEMU PCI-PCI bridge
05:04.0 PCI bridge: Red Hat, Inc. QEMU PCI-PCI bridge
06:03.0 Unclassified device [00ff]: Red Hat, Inc. Virtio memory balloon
06:05.0 SCSI storage controller: Red Hat, Inc. Virtio SCSI
06:12.0 Ethernet controller: Red Hat, Inc. Virtio network device

What could be wrong here?
 
yantramanav said:
I can now attach the PCIe SATA controller to the FreeNAS VM, and it starts fine and doesn't crash the whole proxmox install.
Click to expand...

perfect!

yantramanav said:
However the FreeNAS can't see the SATA disks.
Click to expand...

Next would be to see if it's a FreeBSD problem or a general one. Best is to try to boot a live Linux iso and see if you see the devices there. If they're present, you have a Problem with FreeBSD, not with the passthrough itself.
 
Thanks for the tip.
I spun up an Ubuntu 20.04 server image and attached the PCIe device to it. I can see that Linux has identified the Marvell device.

Code: 
bhootraja@sarpalok:~$ lspci -knn
00:00.0 Host bridge [0600]: Intel Corporation 82G33/G31/P35/P31 Express DRAM Controller [8086:29c0]
    Subsystem: Red Hat, Inc. QEMU Virtual Machine [1af4:1100]
... <truncated>
00:1e.0 PCI bridge [0604]: Intel Corporation 82801 PCI Bridge [8086:244e] (rev 92)
00:1f.0 ISA bridge [0601]: Intel Corporation 82801IB (ICH9) LPC Interface Controller [8086:2918] (rev 02)
    Subsystem: Red Hat, Inc. QEMU Virtual Machine [1af4:1100]
    Kernel driver in use: lpc_ich
    Kernel modules: lpc_ich
00:1f.2 SATA controller [0106]: Intel Corporation 82801IR/IO/IH (ICH9R/DO/DH) 6 port SATA Controller [AHCI mode] [8086:2922] (rev 02)
    Subsystem: Red Hat, Inc. QEMU Virtual Machine [1af4:1100]
    Kernel driver in use: ahci
    Kernel modules: ahci
... <truncated>
01:00.0 SATA controller [0106]: Marvell Technology Group Ltd. Device [1b4b:9215] (rev 11)
    Subsystem: Marvell Technology Group Ltd. Device [1b4b:9215]
    Kernel driver in use: ahci
    Kernel modules: ahci
05:01.0 PCI bridge [0604]: Red Hat, Inc. QEMU PCI-PCI bridge [1b36:0001]
    Kernel driver in use: shpchp
... <truncated>
06:05.0 SCSI storage controller [0100]: Red Hat, Inc. Virtio SCSI [1af4:1004]
    Subsystem: Red Hat, Inc. Virtio SCSI [1af4:0008]
    Kernel driver in use: virtio-pci

So the problem seems to be on FreeBSD side. How do I make sure that FreeBSD detects the PCIe device?
 
yantramanav said:
How do I make sure that FreeBSD detects the PCIe device?
Click to expand...

That's a good question. Is the chipset supported in FreeNAS or FreeBSD? Next would be to start a FreeBSD Live System and it from there if it's a FreeBSD or a FreeNAS problem and ask in a FreeBSD/FreeNAS specific forum. This one is very Linux centric, and my FreeBSD years were in another decade.
 
Looks like it's a FreeBSD issue, I've started a FreeBSD system and the card is not visible there. Should I check on the FreeBSD forums or FreeNAS forums?

I checked on the FreeNAS forums and they told that FreeBSD has support for Marvell 88SE9215 chip. I tried finding the recommended LSI chips but none of them could fit the PCI Express 2.0 x1 slot present in the motherboard. Could you please recommend me an LSI card that fits into that slot?


Code: 
root@mukta:~ # pciconf -lv
<truncated>
none0@pci0:0:28:0:    class=0x060400 card=0x00001b36 chip=0x000c1b36 rev=0x00 hdr=0x01
    vendor     = 'Red Hat, Inc.'
    device     = 'QEMU PCIe Root port'
    class      = bridge
    subclass   = PCI-PCI
none1@pci0:0:28:1:    class=0x060400 card=0x00001b36 chip=0x000c1b36 rev=0x00 hdr=0x01
    vendor     = 'Red Hat, Inc.'
    device     = 'QEMU PCIe Root port'
    class      = bridge
    subclass   = PCI-PCI
none2@pci0:0:28:2:    class=0x060400 card=0x00001b36 chip=0x000c1b36 rev=0x00 hdr=0x01
    vendor     = 'Red Hat, Inc.'
    device     = 'QEMU PCIe Root port'
    class      = bridge
    subclass   = PCI-PCI
none3@pci0:0:28:3:    class=0x060400 card=0x00001b36 chip=0x000c1b36 rev=0x00 hdr=0x01
    vendor     = 'Red Hat, Inc.'
    device     = 'QEMU PCIe Root port'
    class      = bridge
    subclass   = PCI-PCI
<truncated>
pcib1@pci0:0:30:0:    class=0x060401 card=0x00000000 chip=0x244e8086 rev=0x92 hdr=0x01
    vendor     = 'Intel Corporation'
    device     = '82801 PCI Bridge'
    class      = bridge
    subclass   = PCI-PCI
isab0@pci0:0:31:0:    class=0x060100 card=0x11001af4 chip=0x29188086 rev=0x02 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = '82801IB (ICH9) LPC Interface Controller'
    class      = bridge
    subclass   = PCI-ISA
ahci0@pci0:0:31:2:    class=0x010601 card=0x11001af4 chip=0x29228086 rev=0x02 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = '82801IR/IO/IH (ICH9R/DO/DH) 6 port SATA Controller [AHCI mode]'
    class      = mass storage
    subclass   = SATA
none4@pci0:0:31:3:    class=0x0c0500 card=0x11001af4 chip=0x29308086 rev=0x02 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = '82801I (ICH9 Family) SMBus Controller'
    class      = serial bus
    subclass   = SMBus
pcib2@pci0:5:1:0:    class=0x060400 card=0x00000000 chip=0x00011b36 rev=0x00 hdr=0x01
    vendor     = 'Red Hat, Inc.'
    device     = 'QEMU PCI-PCI bridge'
    class      = bridge
    subclass   = PCI-PCI
pcib3@pci0:5:2:0:    class=0x060400 card=0x00000000 chip=0x00011b36 rev=0x00 hdr=0x01
    vendor     = 'Red Hat, Inc.'
    device     = 'QEMU PCI-PCI bridge'
    class      = bridge
    subclass   = PCI-PCI
pcib4@pci0:5:3:0:    class=0x060400 card=0x00000000 chip=0x00011b36 rev=0x00 hdr=0x01
    vendor     = 'Red Hat, Inc.'
    device     = 'QEMU PCI-PCI bridge'
    class      = bridge
    subclass   = PCI-PCI
pcib5@pci0:5:4:0:    class=0x060400 card=0x00000000 chip=0x00011b36 rev=0x00 hdr=0x01
    vendor     = 'Red Hat, Inc.'
    device     = 'QEMU PCI-PCI bridge'
    class      = bridge
    subclass   = PCI-PCI
virtio_pci1@pci0:6:5:0:    class=0x010000 card=0x00081af4 chip=0x10041af4 rev=0x00 hdr=0x00
    vendor     = 'Red Hat, Inc.'
    device     = 'Virtio SCSI'
    class      = mass storage
    subclass   = SCSI

root@mukta:~ # uname -a
FreeBSD mukta 12.1-RELEASE FreeBSD 12.1-RELEASE r354233 GENERIC  amd64
 
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom