Networking fails after starting VM with SATA PCI-passthrough enabled

M

mercury131

Active Member
Aug 7, 2018
24
5
43
34
Hello everyone!

I try to passthrough my SATA PCI controller to my VM.

But when I start vm with SATA PCI controller passthrough - proxmox networking fails.

I got working VM with GPU-passthrough enabled, It's works fine.

I try to add sata controller to /etc/modprobe.d/vfio.conf , but after reboot system doesnt use vfio-pci driver for my controller
Code: 
root@proxmox-lab1:~# lspci -knn| grep -A2 03:05.0
03:05.0 RAID bus controller [0104]: Silicon Image, Inc. SiI 3114 [SATALink/SATARaid] Serial ATA Controller [1095:3114] (rev 02)
        Subsystem: Silicon Image, Inc. SiI 3114 [SATALink/SATARaid] Serial ATA Controller [1095:7114]
        Kernel driver in use: sata_sil

I try to assign vfio driver via this commands:
Code: 
echo 0000:03:05.0 > /sys/bus/pci/drivers/sata_sil/unbind
echo 0000:03:05.0 > /sys/bus/pci/drivers/vfio-pci/bind
It doesn't work..

I aslo check IOMMU groups :
Code: 
/sys/kernel/iommu_groups/0/devices/0000:00:00.0
/sys/kernel/iommu_groups/1/devices/0000:00:04.0
/sys/kernel/iommu_groups/2/devices/0000:00:09.0
/sys/kernel/iommu_groups/3/devices/0000:00:11.0
/sys/kernel/iommu_groups/4/devices/0000:00:12.0
/sys/kernel/iommu_groups/4/devices/0000:00:12.2
/sys/kernel/iommu_groups/5/devices/0000:00:13.0
/sys/kernel/iommu_groups/5/devices/0000:00:13.2
/sys/kernel/iommu_groups/6/devices/0000:00:14.0
/sys/kernel/iommu_groups/7/devices/0000:00:14.1
/sys/kernel/iommu_groups/8/devices/0000:00:14.2
/sys/kernel/iommu_groups/9/devices/0000:00:14.3
/sys/kernel/iommu_groups/10/devices/0000:00:14.4
/sys/kernel/iommu_groups/10/devices/0000:03:05.0
/sys/kernel/iommu_groups/11/devices/0000:00:14.5
/sys/kernel/iommu_groups/12/devices/0000:00:15.0
/sys/kernel/iommu_groups/12/devices/0000:00:15.3
/sys/kernel/iommu_groups/12/devices/0000:04:00.0
/sys/kernel/iommu_groups/12/devices/0000:05:00.0
/sys/kernel/iommu_groups/13/devices/0000:00:16.0
/sys/kernel/iommu_groups/13/devices/0000:00:16.2
/sys/kernel/iommu_groups/14/devices/0000:01:00.0
/sys/kernel/iommu_groups/14/devices/0000:01:00.1
/sys/kernel/iommu_groups/15/devices/0000:02:00.0

My ethernet controller and sata controller in different IOMMU groups:
Code: 
03:05.0 RAID bus controller [0104]: Silicon Image, Inc. SiI 3114 [SATALink/SATARaid] Serial ATA Controller [1095:3114] (rev 02)
        Subsystem: Silicon Image, Inc. SiI 3114 [SATALink/SATARaid] Serial ATA Controller [1095:7114]
        Kernel driver in use: sata_sil
        Kernel modules: sata_sil
04:00.0 SATA controller [0106]: ASMedia Technology Inc. ASM1062 Serial ATA Controller [1b21:0612] (rev 02)
        Subsystem: ASMedia Technology Inc. ASM1062 Serial ATA Controller [1b21:1060]
        Kernel driver in use: ahci
        Kernel modules: ahci
05:00.0 Ethernet controller [0200]: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller [10ec:8168] (rev 06)
        Subsystem: ASRock Incorporation Motherboard (one of many) [1849:8168]
        Kernel driver in use: r8169
        Kernel modules: r8169

I try to passthrough device 03:05.0

My VM config:
Code: 
args: -cpu 'host,-hypervisor,+kvm_pv_unhalt,+kvm_pv_eoi,hv_spinlocks=0x1fff,hv_vapic,hv_time,hv_reset,hv_vpindex,hv_runtime,hv_relaxed,kvm=off,hv_vendor_id=intel'
balloon: 0
boot: dcn
bootdisk: sata0
cores: 8
cpu: host,flags=+pcid;+spec-ctrl
ide2: none,media=cdrom
machine: q35
memory: 4096
name: sata
net0: e1000=AA:28:FD:3B:7A:AD,bridge=vmbr0
numa: 0
ostype: win10
sata0: Gold:vm-105-disk-1,size=32G
scsihw: megasas
smbios1: uuid=2622e266-5364-4dae-908e-3760e3d806e4,manufacturer=Asrock,product=970mpro3,version=9703,serial=3K1N212,sku=3K1N212,family=9703
sockets: 1
vga: std
hostpci0: 03:05

My
/etc/default/grub
Code: 
GRUB_DEFAULT=0
GRUB_TIMEOUT=5
GRUB_DISTRIBUTOR="Proxmox Virtual Environment"
GRUB_CMDLINE_LINUX_DEFAULT="quiet amd_iommu=on intremap=no_x2apic_optout pci.stub.ids=1b21:0612"
GRUB_CMDLINE_LINUX="root=ZFS=rpool/ROOT/pve-1 boot=zfs"

# Disable os-prober, it might add menu entries for each guest
GRUB_DISABLE_OS_PROBER=true

# Uncomment to enable BadRAM filtering, modify to suit your needs
# This works with Linux (no patch required) and with any kernel that obtains
# the memory map information from GRUB (GNU Mach, kernel of FreeBSD ...)
#GRUB_BADRAM="0x01234567,0xfefefefe,0x89abcdef,0xefefefef"

# Uncomment to disable graphical terminal (grub-pc only)
#GRUB_TERMINAL=console

# The resolution used on graphical terminal
# note that you can use only modes which your graphic card supports via VBE
# you can see them in real GRUB with the command `vbeinfo'
#GRUB_GFXMODE=640x480

# Uncomment if you don't want GRUB to pass "root=UUID=xxx" parameter to Linux
#GRUB_DISABLE_LINUX_UUID=true

# Disable generation of recovery mode menu entries
GRUB_DISABLE_RECOVERY="true"

# Uncomment to get a beep at grub start
#GRUB_INIT_TUNE="480 440 1"
my /etc/modules
Code: 
vfio
vfio_iommu_type1
vfio_pci
vfio_virqfd

my
/etc/modprobe.d/vfio.conf
Code: 
options vfio-pci ids=1002:67ff,1002:aae0,1095:3114 disable_vga=1

My Sata controller info for vfio
Code: 
lspci -n -s 03:05.0
03:05.0 0104: 1095:3114 (rev 02)

Working VM config (GPU passthrough):
Code: 
bios: ovmf
boot: dcn
bootdisk: sata0
cores: 8
cpu: host,hidden=1
efidisk0: Gold:vm-108-disk-2,size=128K
hostpci0: 01:00.0;01:00.1,pcie=1,x-vga=on
ide2: local:iso/virtio-win-0.1.141.iso,media=cdrom,size=309208K
machine: q35
memory: 8192
name: steam
net0: virtio=32:0A:9B:30:E0:5E,bridge=vmbr0
numa: 0
onboot: 1
ostype: win10
sata0: Gold:vm-108-disk-3,size=45G
scsihw: virtio-scsi-pci
smbios1: uuid=16fac416-c8b1-48f8-992e-07a079a9ad48
sockets: 1
vga: std
virtio1: Silver:vm-108-disk-1,cache=writeback,size=500G

Code: 
pveversion
pve-manager/5.2-1/0fcd7879 (running kernel: 4.15.17-1-pve)

My hardware:
Code: 
dmidecode -t 2
# dmidecode 3.0
Getting SMBIOS data from sysfs.
SMBIOS 2.7 present.

Handle 0x0002, DMI type 2, 15 bytes
Base Board Information
        Manufacturer: ASRock
        Product Name: 970M Pro3
        Version:
        Serial Number: M80-B1005000766
        Asset Tag:
        Features:
                Board is a hosting board
                Board is replaceable
        Location In Chassis:
        Chassis Handle: 0x0003
        Type: Motherboard
        Contained Object Handles: 0

Code: 
dmidecode -t 4
# dmidecode 3.0
Getting SMBIOS data from sysfs.
SMBIOS 2.7 present.

Handle 0x0004, DMI type 4, 42 bytes
Processor Information
        Socket Designation: CPUSocket
        Type: Central Processor
        Family: FX
        Manufacturer: AMD
        ID: 20 0F 60 00 FF FB 8B 17
        Signature: Family 21, Model 2, Stepping 0
        Flags:
                FPU (Floating-point unit on-chip)
                VME (Virtual mode extension)
                DE (Debugging extension)
                PSE (Page size extension)
                TSC (Time stamp counter)
                MSR (Model specific registers)
                PAE (Physical address extension)
                MCE (Machine check exception)
                CX8 (CMPXCHG8 instruction supported)
                APIC (On-chip APIC hardware supported)
                SEP (Fast system call)
                MTRR (Memory type range registers)
                PGE (Page global enable)
                MCA (Machine check architecture)
                CMOV (Conditional move instruction supported)
                PAT (Page attribute table)
                PSE-36 (36-bit page size extension)
                CLFSH (CLFLUSH instruction supported)
                MMX (MMX technology supported)
                FXSR (FXSAVE and FXSTOR instructions supported)
                SSE (Streaming SIMD extensions)
                SSE2 (Streaming SIMD extensions 2)
                HTT (Multi-threading)
        Version: AMD FX(tm)-8320 Eight-Core Processor
        Voltage: 1.4 V
        External Clock: 200 MHz
        Max Speed: 3500 MHz
        Current Speed: 3500 MHz
        Status: Populated, Enabled
        Upgrade: Socket AM3
        L1 Cache Handle: 0x0005
        L2 Cache Handle: 0x0006
        L3 Cache Handle: 0x0007
        Serial Number: To Be Filled By O.E.M.
        Asset Tag: To Be Filled By O.E.M.
        Part Number: To Be Filled By O.E.M.
        Core Count: 8
        Core Enabled: 8
        Thread Count: 8
        Characteristics:
                64-bit capable

What I do Wrong?
How to correct passthrough SATA controller?
I know, I can just passthrough harddrive via VIRTIO , but I need Hardware sata controller inside VM for some soft installed in guest OS.
 
Last edited:
dcsapak said:
what is the other device inside the iommu group '10' ?
Click to expand...
All devices:
Code: 
root@proxmox-lab1:~# lspci
00:00.0 Host bridge: Advanced Micro Devices, Inc. [AMD/ATI] RD9x0/RX980 Host Bridge (rev 02)
00:00.2 IOMMU: Advanced Micro Devices, Inc. [AMD/ATI] RD890S/RD990 I/O Memory Management Unit (IOMMU)
00:04.0 PCI bridge: Advanced Micro Devices, Inc. [AMD/ATI] RD890/RD9x0/RX980 PCI to PCI bridge (PCI Express GPP Port 0)
00:09.0 PCI bridge: Advanced Micro Devices, Inc. [AMD/ATI] RD890/RD9x0/RX980 PCI to PCI bridge (PCI Express GPP Port 4)
00:11.0 SATA controller: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 SATA Controller [AHCI mode] (rev 40)
00:12.0 USB controller: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 USB OHCI0 Controller
00:12.2 USB controller: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 USB EHCI Controller
00:13.0 USB controller: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 USB OHCI0 Controller
00:13.2 USB controller: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 USB EHCI Controller
00:14.0 SMBus: Advanced Micro Devices, Inc. [AMD/ATI] SBx00 SMBus Controller (rev 42)
00:14.2 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] SBx00 Azalia (Intel HDA) (rev 40)
00:14.3 ISA bridge: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 LPC host controller (rev 40)
00:14.4 PCI bridge: Advanced Micro Devices, Inc. [AMD/ATI] SBx00 PCI to PCI Bridge (rev 40)
00:14.5 USB controller: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 USB OHCI2 Controller
00:15.0 PCI bridge: Advanced Micro Devices, Inc. [AMD/ATI] SB700/SB800/SB900 PCI to PCI bridge (PCIE port 0)
00:15.3 PCI bridge: Advanced Micro Devices, Inc. [AMD/ATI] SB900 PCI to PCI bridge (PCIE port 3)
00:16.0 USB controller: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 USB OHCI0 Controller
00:16.2 USB controller: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 USB EHCI Controller
00:18.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 15h Processor Function 0
00:18.1 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 15h Processor Function 1
00:18.2 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 15h Processor Function 2
00:18.3 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 15h Processor Function 3
00:18.4 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 15h Processor Function 4
00:18.5 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 15h Processor Function 5
01:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Baffin [Polaris11] (rev cf)
01:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Device aae0
02:00.0 USB controller: Etron Technology, Inc. EJ188/EJ198 USB 3.0 Host Controller
03:05.0 RAID bus controller: Silicon Image, Inc. SiI 3114 [SATALink/SATARaid] Serial ATA Controller (rev 02)
04:00.0 SATA controller: ASMedia Technology Inc. ASM1062 Serial ATA Controller (rev 02)
05:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 06)

Another device in group 10
Code: 
root@proxmox-lab1:~# lspci | grep 00:14.4
00:14.4 PCI bridge: Advanced Micro Devices, Inc. [AMD/ATI] SBx00 PCI to PCI Bridge (rev 40)
 
If I remove device 00:14.4 via commands
Code: 
cd /sys/devices/pci0000\:00/0000:00:14.4/
echo -n 1 > remove
I lost device 03:05.0 RAID bus controller: Silicon Image, Inc. SiI 3114 [SATALink/SATARaid] Serial ATA Controller (rev 02)
from lspci and cannot use it in VM.
8feb22c8fa9af74565694d39380dd9dc.png
 
mercury131 said:
I try to add sata controller to /etc/modprobe.d/vfio.conf , but after reboot system doesnt use vfio-pci driver for my controller
Click to expand...
you could try to rename the file to 00-vfio.conf ( i believe it loads it in order ) and post the content of it ?

also did you update your initramfs after ? ( with update-initramfs -u -k all )
 
dcsapak said:
you could try to rename the file to 00-vfio.conf ( i believe it loads it in order ) and post the content of it ?

also did you update your initramfs after ? ( with update-initramfs -u -k all )
Click to expand...

I rename file vfio.conf to 0-vfio.conf and then run command:
Code: 
update-initramfs -u -k all
After reboot I see changes, now device 03:05.0 use v-fio driver:
Code: 
root@proxmox-lab1:~# lspci -knn| grep -A2 03:05.0
03:05.0 RAID bus controller [0104]: Silicon Image, Inc. SiI 3114 [SATALink/SATARaid] Serial ATA Controller [1095:3114] (rev 02)
        Subsystem: Silicon Image, Inc. SiI 3114 [SATALink/SATARaid] Serial ATA Controller [1095:7114]
        Kernel driver in use: vfio-pci
It's good.

But if I run VM with pci passthrough (with this device 03:05.0) - proxmox networking fails again.

I think Its may be because second device in iommu group is PCI to PCI Bridge.
May that be root case of problem?
I don't understand how to change iommu group for device :
Code: 
00:14.4 PCI bridge: Advanced Micro Devices, Inc. [AMD/ATI] SBx00 PCI to PCI Bridge (rev 40)
 
mercury131 said:
I think Its may be because second device in iommu group is PCI to PCI Bridge.
Click to expand...
this may very well be

mercury131 said:
I don't understand how to change iommu group for device :
Click to expand...
you cannot really, this is determined by your motherboard, the only thing you can try is to put it in a different pcie slot
 
Me too, I've been struggling with PCI SATA cards based on the Silicon Image 3114 chip (an old one, and a brand new one). Too unstable, I think --- I've abandoned them :-(
 
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom
Back