PCI Sata controller: no devices in Proxmox

[notarobot]

Hello everyone,
New server is up and running and is working like a charm except one major issue.
I had to add a few extra SATA ports so I bought this SATA PCI controller (based on Marvell 9215).
At boot I see the card and the drive attached to it but one Proxmox is started no traces of the disk

If I do lspci -v:

02:00.0 SATA controller: Marvell Technology Group Ltd. Device 9215 (rev 11) (prog-if 01 [AHCI 1.0])
        Subsystem: Marvell Technology Group Ltd. Device 9215
        Physical Slot: 2
        Flags: fast devsel, IRQ 16
        I/O ports at d050 [size=8]
        I/O ports at d040 [size=4]
        I/O ports at d030 [size=8]
        I/O ports at d020 [size=4]
        I/O ports at d000 [size=32]
        Memory at df340000 (32-bit, non-prefetchable) [size=2K]
        Expansion ROM at df300000 [disabled] [size=256K]
        Capabilities: [40] Power Management version 3
        Capabilities: [50] MSI: Enable- Count=1/1 Maskable- 64bit-
        Capabilities: [70] Express Legacy Endpoint, MSI 00
        Capabilities: [e0] SATA HBA v0.0
        Kernel driver in use: vfio-pci
        Kernel modules: ahci

No traces of the disks in /dev/ and they are working everywhere else and detected in BIOS. I'm thinking of a kernel issue ?

If anybody can help that would be nice !

 

[dcsapak]

Kernel driver in use: vfio-pci

it seem that the 'vfio-pci' driver takes the card (maybe you configured something for passthrough?) so the kernel will not query it for disks

 

[notarobot]

I also thought that was weird. I haven't configured the card for passthrough so that's even weirder...

 

[notarobot]

I double checked: At boot I have a screen that says I have this card plugged in with a 500GB SSD plugged in. That makes me think that the problem is at Proxmox level.

 

[notarobot]

I may have found something: the SATA PCI Card is on the same IOMMU group that a network card. This network card is being passthrough in one of my VM so maybe that's the problem. How can I put this card in its own IOMMU group ?

 

[dcsapak]

I may have found something: the SATA PCI Card is on the same IOMMU group that a network card. This network card is being passthrough in one of my VM so maybe that's the problem. How can I put this card in its own IOMMU group ?

you cannot really, this is depending on the mainboard bios and pcie configuration.
you can try to switch pcie slots (if possible)

can you post your vfio module configs (if any) and the config of the vm with the passed through network card and the output of lspci ?
did you blacklist any drivers ?

(side note: we have included the acs override patch (https://wiki.archlinux.org/index.ph...passing_the_IOMMU_groups_(ACS_override_patch)) in our kernel, but make sure you know the possible side effects)

 

[notarobot]

I've "solved" this issue this morning, my VM needed a network card and a SATA controller so instead of using my motherboard controller I gave it the PCI card. Problem solved !
Thanks a lot for the help anyway !

 

[freesbee]

...maybe I only was unlucky: I bought this very similar SATA controller by QNINE, which is apparently based on Marvell 88SE9215.
Unfortunately lspci does not show me anything new.
I am using this hypervisor just for home experiments, and I would need to expand the SATA ports as the mainboard only has 4.
My grub file has the following line:

GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on pcie_acs_override=downstream,multifunction video=efifb:eek:ff"

and my /etc/modules has:

vfio
vfio_iommu_type1
vfio_pci
vfio_virqfd

I am only at the beginning with my ProxMox experience, so I am a little bit in trouble to understand if the controller cannot be detected or if I am simply doing anything wrong.
Can anybody give me a hint into the right direction?
Thank you a lot in advance

 

[leesteken]

...maybe I only was unlucky: I bought this very similar SATA controller by QNINE, which is apparently based on Marvell 88SE9215.
Unfortunately lspci does not show me anything new.
I am using this hypervisor just for home experiments, and I would need to expand the SATA ports as the mainboard only has 4.
My grub file has the following line:

GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on pcie_acs_override=downstream,multifunction video=efifb:eek:ff"

and my /etc/modules has:

vfio
vfio_iommu_type1
vfio_pci
vfio_virqfd

I am only at the beginning with my ProxMox experience, so I am a little bit in trouble to understand if the controller cannot be detected or if I am simply doing anything wrong.
Can anybody give me a hint into the right direction?
Thank you a lot in advance

I don't think your problem has anything to do with this two year old problem.
If the SATA controller does not show up at all, I assume the card is broken. Or it does show up but you overlooked it because the chip is the same or very similar to another SATA controller on your motherboard? Connect a drive to the card and see if you can find the drive. You can then find the card using /dev/disk/by-path/.

 

[freesbee]

So I have followed your suggestion, remounted everything, including 2 new disks that are connected to this new controller.
The command lshw -C disk only shows me the 4 HDDs that are currently connected to the mainboard (currently ordered in a RAIDZ10 array), but ignores the 2 new disks connected to the new controller.
The mainboard is an ASUS Z170-P with latest BIOS, and I have tried to connect the controller to both PCI-E 3.0 x1 ports.
The command lspci shows the same output as without the controller:

root@virtuoso:~# lspci
00:00.0 Host bridge: Intel Corporation Skylake Host Bridge/DRAM Registers (rev 07)
00:02.0 VGA compatible controller: Intel Corporation HD Graphics 530 (rev 06)
00:14.0 USB controller: Intel Corporation Sunrise Point-H USB 3.0 xHCI Controller (rev 31)
00:16.0 Communication controller: Intel Corporation Sunrise Point-H CSME HECI #1 (rev 31)
00:17.0 SATA controller: Intel Corporation Sunrise Point-H SATA controller [AHCI mode] (rev 31)
00:1b.0 PCI bridge: Intel Corporation Sunrise Point-H PCI Root Port #17 (rev f1)
00:1b.3 PCI bridge: Intel Corporation Sunrise Point-H PCI Root Port #20 (rev f1)
00:1c.0 PCI bridge: Intel Corporation Sunrise Point-H PCI Express Root Port #1 (rev f1)
00:1c.2 PCI bridge: Intel Corporation Sunrise Point-H PCI Express Root Port #3 (rev f1)
00:1c.3 PCI bridge: Intel Corporation Sunrise Point-H PCI Express Root Port #4 (rev f1)
00:1d.0 PCI bridge: Intel Corporation Sunrise Point-H PCI Express Root Port #9 (rev f1)
00:1f.0 ISA bridge: Intel Corporation Sunrise Point-H LPC Controller (rev 31)
00:1f.2 Memory controller: Intel Corporation Sunrise Point-H PMC (rev 31)
00:1f.3 Audio device: Intel Corporation Sunrise Point-H HD Audio (rev 31)
00:1f.4 SMBus: Intel Corporation Sunrise Point-H SMBus (rev 31)
04:00.0 PCI bridge: ASMedia Technology Inc. ASM1083/1085 PCIe to PCI Bridge (rev 04)
06:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 15)
07:00.0 Non-Volatile memory controller: Samsung Electronics Co Ltd NVMe SSD Controller SM981/PM981
root@virtuoso:~#

I could mount the controller to another mainboard to verify if it is functional, but this would be only a waste of time as I have bought this controller only for expanding my home ProxMox hypervisor, and that is my need at the moment.
According to your experience is there anything else that I could do, or just send back the controller as it is not detected by Linux? (at least on my build)

 

[leesteken]

Are those slots active? Sometimes a slot is disabled if a M.2-slot is used that shares the PCIe-lanes, for example. Do you have another add-in card to check?
The user manual (1.5.2 Configuring an expansion card) states that you need to assign an IRQ to the card, and it also shows that some IRQs are shared between slots and on-board devices. I have never need to do so, but maybe this motherboard requires some manual action in its BIOS when adding devices?
Maybe it is some weird incompatibility between this particular controller and that motherboard?

 

[freesbee]

Well, if one PCIe lane is reserved for the M.2 drive, this does not explain why also the other slot does not work, as I have left this mainboard configuration quite "standard". I suspect that the time already invested in the project does not justify additional efforts, so now I am tendetiously oriented to return this controller and search for alternatives...
But before that I will verify IRQs as you suggested.

 

[leesteken]

I did not find anything in the manual about certain slots being shared with other devices or slots, so that is probably not it. Maybe the PCIe slots are just deactivated (possibly from not having an IRQ). Maybe ASUS support can help? I'm sorry, I don't have experience with this particular issue or your hardware and I am just trying to be give helpful guesses.

 

[freesbee]

So I have verified if the serial ports were active, and they were active [IO=3Fh8; IRQ=4] as default settings.
I have tried all other options, but still lspci does not find anything new.
I would get to the conclusion that Marvell 88SE9215 cannot be detected by Linux (or maybe just Proxmox 6) and return the card.
For my rig expansion I will try to find some LSI controller with HBA firmware, as I have the information that they work properly with ZFS.
Always better than replacing a mainboard...
Thank you very much for your help. I will report here the solution that I will find... assuming that I will find one

 

[freesbee]

...well, as I promised here my feedback, maybe will be helpful also for others.
I think that I could find a good solution: I increased my budget for this project by ~12 EUR (yes, 12, not 120, just twelve EUR) and I have purchased a
"Fujitsu 9211-8i 6Gbps HBA LSI FW: P20" from some kind Chinese eSeller. The controller arrived in 2 weeks, with the traditional Shenzen precision that we all know....
I plugged it into my rig, and proxmox has seen it immediately, as lspci has given:

root@myrig:~# lspci
(...)
01:00.0 RAID bus controller: LSI Logic / Symbios Logic SAS2008 PCI-Express Fusion-MPT SAS-2 [Falcon] (rev 03)
(...)

Then I shutdown the machine again, and plugged all my 4 drives arranged in RAIDZ10 to the new controller.
It took some 30 additional seconds to boot, but afterwards I had my existing storage connected to the new controller perfectly working.
Now I have still 4 ports free on the controller (the kind Chinese has sent me 2 of these cables, exactly as shown in the eBay announcement) and 4 more free ports on the mainboard. Therefore I will be soon able to start my experiments.
This is real "value for the money" !!!
Again I would like to thank avw for the support he has offered.