[SOLVED] Passthrough SATA onboard controller not appear on "Mapped Device" in GUI

[cryonie]

Hello,

I'm trying to passthrough my onboard SATA controller.
I've read many guides (including proxmox wiki) and made many tests but i never managed to make the PCI appear in the "Mapped Devices" field when I try to add a PCI Device on proxmox GUI.

What i've done :
Add "intel_iommu=on iommu=pt" to line GRUB_CMDLINE_LINUX_DEFAULT= in file /etc/default/grub

I've put those lines in /etc/modules

vfio
vfio_iommu_type1
vfio_pci
vfio_virqfd

Then updates grub by "update-grub" command.

When i rebooted everything seemed good.
"dmesg | grep -e DMAR -e IOMMU" shows me this :

[    0.008386] ACPI: DMAR 0x00000000B780D000 0000A8 (v01 INTEL  CFL      00000002      01000013)
[    0.008419] ACPI: Reserving DMAR table memory at [mem 0xb780d000-0xb780d0a7]
[    0.050675] DMAR: IOMMU enabled
[    0.135756] DMAR: Host address width 39
[    0.135757] DMAR: DRHD base: 0x000000fed90000 flags: 0x0
[    0.135764] DMAR: dmar0: reg_base_addr fed90000 ver 1:0 cap 1c0000c40660462 ecap 19e2ff0505e
[    0.135766] DMAR: DRHD base: 0x000000fed91000 flags: 0x1
[    0.135769] DMAR: dmar1: reg_base_addr fed91000 ver 1:0 cap d2008c40660462 ecap f050da
[    0.135771] DMAR: RMRR base: 0x000000b749d000 end: 0x000000b74bcfff
[    0.135773] DMAR: RMRR base: 0x000000bc000000 end: 0x000000cc7fffff
[    0.135775] DMAR-IR: IOAPIC id 2 under DRHD base  0xfed91000 IOMMU 1
[    0.135776] DMAR-IR: HPET id 0 under DRHD base 0xfed91000
[    0.135777] DMAR-IR: Queued invalidation will be enabled to support x2apic and Intr-remapping.
[    0.138960] DMAR-IR: Enabled IRQ remapping in x2apic mode
[    1.662703] DMAR: No ATSR found
[    1.662704] DMAR: No SATC found
[    1.662705] DMAR: IOMMU feature fl1gp_support inconsistent
[    1.662707] DMAR: IOMMU feature pgsel_inv inconsistent
[    1.662708] DMAR: IOMMU feature nwfs inconsistent
[    1.662709] DMAR: IOMMU feature pasid inconsistent
[    1.662710] DMAR: IOMMU feature eafs inconsistent
[    1.662710] DMAR: IOMMU feature prs inconsistent
[    1.662711] DMAR: IOMMU feature nest inconsistent
[    1.662712] DMAR: IOMMU feature mts inconsistent
[    1.662713] DMAR: IOMMU feature sc_support inconsistent
[    1.662713] DMAR: IOMMU feature dev_iotlb_support inconsistent
[    1.662715] DMAR: dmar0: Using Queued invalidation
[    1.662717] DMAR: dmar1: Using Queued invalidation
[    1.663141] DMAR: Intel(R) Virtualization Technology for Directed I/O

So i have "[ 0.050493] DMAR: IOMMU enabled" that didn't appear before the grub modification

"dmesg | grep 'remapping'" shiws me this :

[    0.135523] DMAR-IR: Queued invalidation will be enabled to support x2apic and Intr-remapping.
[    0.138704] DMAR-IR: Enabled IRQ remapping in x2apic mode

So as guide says "Enabled IRQ remapping in x2apic mode" is what we are lookign for

So for now everything is fine.
"pvesh get /nodes/{node}/hardware/pci --pci-class-blacklist """ shows me this :

┌──────────┬────────┬──────────────┬────────────┬────────┬───────────────────────────────────────────────────┬──────┬──────────────────┬─────────────────────
│ class    │ device │ id           │ iommugroup │ vendor │ device_name                                       │ mdev │ subsystem_device │ subsystem_device_nam
╞══════════╪════════╪══════════════╪════════════╪════════╪═══════════════════════════════════════════════════╪══════╪══════════════════╪═════════════════════
│ 0x010601 │ 0xa352 │ 0000:00:17.0 │          4 │ 0x8086 │ Cannon Lake PCH SATA AHCI Controller              │      │ 0x8592           │                     
├──────────┼────────┼──────────────┼────────────┼────────┼───────────────────────────────────────────────────┼──────┼──────────────────┼─────────────────────
│ 0x010802 │ 0x0001 │ 0000:01:00.0 │          8 │ 0x1e0f │ NVMe SSD Controller BG4                           │      │ 0x0001           │                     
├──────────┼────────┼──────────────┼────────────┼────────┼───────────────────────────────────────────────────┼──────┼──────────────────┼─────────────────────
│ 0x010802 │ 0xa808 │ 0000:02:00.0 │          9 │ 0x144d │ NVMe SSD Controller SM981/PM981/PM983             │      │ 0xa801           │ SSD 970 EVO         
├──────────┼────────┼──────────────┼────────────┼────────┼───────────────────────────────────────────────────┼──────┼──────────────────┼─────────────────────
│ 0x020000 │ 0x15bb │ 0000:00:1f.6 │          7 │ 0x8086 │ Ethernet Connection (7) I219-LM                   │      │ 0x8591           │                     
├──────────┼────────┼──────────────┼────────────┼────────┼───────────────────────────────────────────────────┼──────┼──────────────────┼─────────────────────
│ 0x030000 │ 0x3e92 │ 0000:00:02.0 │          0 │ 0x8086 │ CoffeeLake-S GT2 [UHD Graphics 630]               │      │ 0x8592           │                     
├──────────┼────────┼──────────────┼────────────┼────────┼───────────────────────────────────────────────────┼──────┼──────────────────┼─────────────────────
│ 0x050000 │ 0xa36f │ 0000:00:14.2 │          3 │ 0x8086 │ Cannon Lake PCH Shared SRAM                       │      │ 0x8592           │                     
├──────────┼────────┼──────────────┼────────────┼────────┼───────────────────────────────────────────────────┼──────┼──────────────────┼─────────────────────
│ 0x060000 │ 0x3ec2 │ 0000:00:00.0 │          1 │ 0x8086 │ 8th Gen Core Processor Host Bridge/DRAM Registers │      │ 0x8592           │                     
├──────────┼────────┼──────────────┼────────────┼────────┼───────────────────────────────────────────────────┼──────┼──────────────────┼─────────────────────
│ 0x060100 │ 0xa306 │ 0000:00:1f.0 │          7 │ 0x8086 │ Q370 Chipset LPC/eSPI Controller                  │      │ 0x8592           │                     
├──────────┼────────┼──────────────┼────────────┼────────┼───────────────────────────────────────────────────┼──────┼──────────────────┼─────────────────────
│ 0x060400 │ 0xa340 │ 0000:00:1b.0 │          5 │ 0x8086 │ Cannon Lake PCH PCI Express Root Port #17         │      │ 0x8592           │                     
├──────────┼────────┼──────────────┼────────────┼────────┼───────────────────────────────────────────────────┼──────┼──────────────────┼─────────────────────
│ 0x060400 │ 0xa32c │ 0000:00:1b.4 │          6 │ 0x8086 │ Cannon Lake PCH PCI Express Root Port #21         │      │ 0x8592           │                     
├──────────┼────────┼──────────────┼────────────┼────────┼───────────────────────────────────────────────────┼──────┼──────────────────┼─────────────────────
│ 0x0c0330 │ 0xa36d │ 0000:00:14.0 │          3 │ 0x8086 │ Cannon Lake PCH USB 3.1 xHCI Host Controller      │      │ 0x8592           │                     
├──────────┼────────┼──────────────┼────────────┼────────┼───────────────────────────────────────────────────┼──────┼──────────────────┼─────────────────────
│ 0x0c0500 │ 0xa323 │ 0000:00:1f.4 │          7 │ 0x8086 │ Cannon Lake PCH SMBus Controller                  │      │ 0x8592           │                     
├──────────┼────────┼──────────────┼────────────┼────────┼───────────────────────────────────────────────────┼──────┼──────────────────┼─────────────────────
│ 0x0c8000 │ 0xa324 │ 0000:00:1f.5 │          7 │ 0x8086 │ Cannon Lake PCH SPI Controller                    │      │ 0x8592           │                     
├──────────┼────────┼──────────────┼────────────┼────────┼───────────────────────────────────────────────────┼──────┼──────────────────┼─────────────────────
│ 0x118000 │ 0xa379 │ 0000:00:12.0 │          2 │ 0x8086 │ Cannon Lake PCH Thermal Controller                │      │ 0x8592           │                     
└──────────┴────────┴──────────────┴────────────┴────────┴───────────────────────────────────────────────────┴──────┴──────────────────┴─────────────────────

So my SATA controller is alone on his group and this is also cool.

So as console gives me, everything should be good but when i look at proxmox GUI :
I see nothing in the "Mapped Device".

Even if I see the different devices in "Raw Device"


So now the questions
1) Were all those steps necessary and did I miss sont steps ?
2) What is the difference between "Raw Device" and "Mapped Device" ?
And if ti is necessary, 3) how could i make the SATA controller appaer in the "Mapped Device" section ?

Thank you for your help !

 

[leesteken]

So now the questions
1) Were all those steps necessary and did I miss sont steps ?

vfio_vifqfd no longer exists in PVE 8.

2) What is the difference between "Raw Device" and "Mapped Device" ?

Use mapped devices to be able to move VMs with PCI(e) passthrough between Proxnox nodes in a cluster. Each node does need to have the "same" hardware available for passthrough but the PCI ID might be different. That's why mapping is necessary (for migrations of VMs with passthrough in a cluster).

And if ti is necessary,

No, you can just use PCI(e) passthrough (of a "raw" device) on a single node: https://pve.proxmox.com/pve-docs/pve-admin-guide.html#qm_pci_passthrough

3) how could i make the SATA controller appaer in the "Mapped Device" section ?

By mapping the resource: https://pve.proxmox.com/pve-docs/pve-admin-guide.html#resource_mapping

 

[cryonie]

Thank you for the reply !

1) Ok i'll remove it
2) Ok, difference noted,
3) Well ok didn't see that but I won't do it because my proxmox is on it's own

Thanks again, now it's time to plug things on the SATA and see if it's working

 

[cryonie]

Ok it seems to be working (when i'm in my VM i see the disk on my SATA controller) but something is strange.

I boot my proxmox (VM not booted) I see my disk on SATA controllerin my proxmox.
Once I launch the VM, the disk disapear from proxmox and will not reappear even if the VM is shutdown.

Is this a normal behaviour ?
The SATA drives should not be always "hidden" from proxmox ?
And most of all, is it a problem ? Because i like this way. If my VM doesn't boot I could access the disks just by mounting them. I like it like that but i'm really not sure it's normal

 

[leesteken]

Ok it seems to be working (when i'm in my VM i see the disk on my SATA controller) but something is strange.

I boot my proxmox (VM not booted) I see my disk on SATA controllerin my proxmox.
Once I launch the VM, the disk disapear from proxmox and will not reappear even if the VM is shutdown.

Is this a normal behaviour ?
The SATA drives should not be always "hidden" from proxmox ?

It is normal. Unless you early-bind the device to vfio-pci (and make sure vfio-pci loads before the normal driver for the device with a softdep), it will be accessible by the host.