Hi,
I will passthrough my LSI sas 9207-8i in IT Mode P20 to my VM.
When I dismark the PCI-Express tag it works fine and I see and can use the HBA in my VM, but it is very slow.
So I changed the Machine to q35 and mark the PCI-Express tag.
After that I can't see the HBA in my VM. At the boot the HBA Bios is displayed.
I think have tried everything fron the documentation https://pve.proxmox.com/wiki/PCI(e)_Passthrough.
I hope somebody have an idea an can help me to fix my problem.
regards
Lars
Interrupt Remapping is enabled:
Device is in an separate IOMMU group:
lspci -nnk => Kernel driver in use: vfio-pci
lspci
dmesg
VM.conf
PVE Version
I will passthrough my LSI sas 9207-8i in IT Mode P20 to my VM.
When I dismark the PCI-Express tag it works fine and I see and can use the HBA in my VM, but it is very slow.
So I changed the Machine to q35 and mark the PCI-Express tag.
After that I can't see the HBA in my VM. At the boot the HBA Bios is displayed.
I think have tried everything fron the documentation https://pve.proxmox.com/wiki/PCI(e)_Passthrough.
I hope somebody have an idea an can help me to fix my problem.
regards
Lars
Interrupt Remapping is enabled:
Code:
root@pve:~# dmesg | grep -e DMAR -e IOMMU -e AMD-Vi
[ 0.586939] pci 0000:00:00.2: AMD-Vi: IOMMU performance counters supported
[ 0.590294] pci 0000:00:00.2: AMD-Vi: Found IOMMU cap 0x40
[ 0.590296] pci 0000:00:00.2: AMD-Vi: Extended features (0x58f77ef22294ade):
[ 0.590299] AMD-Vi: Interrupt remapping enabled
[ 0.590300] AMD-Vi: Virtual APIC enabled
[ 0.590301] AMD-Vi: X2APIC enabled
[ 0.590456] AMD-Vi: Lazy IO/TLB flushing enabled
[ 0.591769] perf/amd_iommu: Detected AMD IOMMU #0 (2 banks, 4 counters/bank).Device is in an separate IOMMU group:
Code:
root@pve:~# find /sys/kernel/iommu_groups/ -type l | grep 2b
/sys/kernel/iommu_groups/13/devices/0000:2b:00.0lspci -nnk => Kernel driver in use: vfio-pci
Code:
2b:00.0 Serial Attached SCSI controller [0107]: LSI Logic / Symbios Logic SAS2008 PCI-Express Fusion-MPT SAS-2 [Falcon] [1000:0072] (rev 03)
Subsystem: LSI Logic / Symbios Logic SAS2008 PCI-Express Fusion-MPT SAS-2 [Falcon] [1000:3020]
Kernel driver in use: vfio-pci
Kernel modules: mpt3saslspci
Code:
root@pve:~# lspci | grep LSI
2b:00.0 Serial Attached SCSI controller: LSI Logic / Symbios Logic SAS2008 PCI-Express Fusion-MPT SAS-2 [Falcon] (rev 03)dmesg
Code:
root@pve:~# dmesg | grep 2b
[ 0.000000] efi: ACPI 2.0=0xec89d000 ACPI=0xec89d000 SMBIOS=0xed2b3000 SMBIOS 3.0=0xed2b2000 MEMATTR=0xe7ed0018 ESRT=0xe7f30618
[ 0.195639] pci 0000:2b:00.0: [1000:0072] type 00 class 0x010700
[ 0.195656] pci 0000:2b:00.0: reg 0x10: [io 0xf000-0xf0ff]
[ 0.195667] pci 0000:2b:00.0: reg 0x14: [mem 0xf7fc0000-0xf7fc3fff 64bit]
[ 0.195677] pci 0000:2b:00.0: reg 0x1c: [mem 0xf7f80000-0xf7fbffff 64bit]
[ 0.195690] pci 0000:2b:00.0: reg 0x30: [mem 0xf7f00000-0xf7f7ffff pref]
[ 0.195743] pci 0000:2b:00.0: supports D1 D2
[ 0.195768] pci 0000:2b:00.0: reg 0x174: [mem 0x00000000-0x00003fff 64bit]
[ 0.195770] pci 0000:2b:00.0: VF(n) BAR0 space: [mem 0x00000000-0x0003ffff 64bit] (contains BAR0 for 16 VFs)
[ 0.195780] pci 0000:2b:00.0: reg 0x17c: [mem 0x00000000-0x0003ffff 64bit]
[ 0.195781] pci 0000:2b:00.0: VF(n) BAR2 space: [mem 0x00000000-0x003fffff 64bit] (contains BAR2 for 16 VFs)
[ 0.195879] pci 0000:00:03.1: PCI bridge to [bus 2b]
[ 0.217938] pci 0000:2b:00.0: BAR 9: no space for [mem size 0x00400000 64bit]
[ 0.217940] pci 0000:2b:00.0: BAR 9: failed to assign [mem size 0x00400000 64bit]
[ 0.217942] pci 0000:2b:00.0: BAR 7: no space for [mem size 0x00040000 64bit]
[ 0.217943] pci 0000:2b:00.0: BAR 7: failed to assign [mem size 0x00040000 64bit]
[ 0.217945] pci 0000:00:03.1: PCI bridge to [bus 2b]
[ 0.217997] pci 0000:00:03.1: PCI bridge to [bus 2b]
[ 0.218138] pci 0000:2b:00.0: BAR 6: assigned [mem 0xf0000000-0xf007ffff pref]
[ 0.218140] pci 0000:2b:00.0: BAR 3: assigned [mem 0xf0080000-0xf00bffff 64bit]
[ 0.218148] pci 0000:2b:00.0: BAR 9: assigned [mem 0xf00c0000-0xf04bffff 64bit]
[ 0.218152] pci 0000:2b:00.0: BAR 1: assigned [mem 0xf04c0000-0xf04c3fff 64bit]
[ 0.218160] pci 0000:2b:00.0: BAR 7: assigned [mem 0xf04c4000-0xf0503fff 64bit]
[ 0.218163] pci 0000:00:03.1: PCI bridge to [bus 2b]
[ 0.218247] pci_bus 0000:2b: resource 0 [io 0xf000-0xffff]
[ 0.218248] pci_bus 0000:2b: resource 1 [mem 0xf0000000-0xf05fffff]
[ 0.589852] pci 0000:2b:00.0: Adding to iommu group 13VM.conf
Code:
root@pve:/etc/pve/qemu-server# cat 120.conf
bootdisk: scsi0
cores: 8
cpu: host
hostpci0: 2b:00,pcie=1
ide2: none,media=cdrom
machine: q35
memory: 16384
name: freenas
net0: virtio=D2:47:96:3F:C0:85,bridge=vmbr0,firewall=1
numa: 0
onboot: 1
ostype: other
scsi0: vm:120/vm-120-disk-0.qcow2,discard=on,size=16G,ssd=1
scsihw: virtio-scsi-pci
smbios1: uuid=e3a0128d-f36e-461b-84c2-31ac1d44321f
sockets: 1
startup: order=2,up=80,down=120
vmgenid: 388d9bc0-b3c3-4af8-9f5b-bb7c76141d05PVE Version
Code:
root@pve:/etc/pve/qemu-server# pveversion --verbose
proxmox-ve: 6.1-2 (running kernel: 5.3.13-1-pve)
pve-manager: 6.1-5 (running version: 6.1-5/9bf06119)
pve-kernel-5.3: 6.1-1
pve-kernel-helper: 6.1-1
pve-kernel-5.3.13-1-pve: 5.3.13-1
pve-kernel-5.3.10-1-pve: 5.3.10-1
ceph-fuse: 12.2.11+dfsg1-2.1+b1
corosync: 3.0.2-pve4
criu: 3.11-3
glusterfs-client: 5.5-3
ifupdown: 0.8.35+pve1
ksm-control-daemon: 1.3-1
libjs-extjs: 6.0.1-10
libknet1: 1.13-pve1
libpve-access-control: 6.0-5
libpve-apiclient-perl: 3.0-2
libpve-common-perl: 6.0-10
libpve-guest-common-perl: 3.0-3
libpve-http-server-perl: 3.0-3
libpve-storage-perl: 6.1-3
libqb0: 1.0.5-1
libspice-server1: 0.14.2-4~pve6+1
lvm2: 2.03.02-pve3
lxc-pve: 3.2.1-1
lxcfs: 3.0.3-pve60
novnc-pve: 1.1.0-1
proxmox-mini-journalreader: 1.1-1
proxmox-widget-toolkit: 2.1-2
pve-cluster: 6.1-3
pve-container: 3.0-16
pve-docs: 6.1-3
pve-edk2-firmware: 2.20191127-1
pve-firewall: 4.0-9
pve-firmware: 3.0-4
pve-ha-manager: 3.0-8
pve-i18n: 2.0-3
pve-qemu-kvm: 4.1.1-2
pve-xtermjs: 3.13.2-1
qemu-server: 6.1-4
smartmontools: 7.1-pve2
spiceterm: 3.1-1
vncterm: 1.6-1
zfsutils-linux: 0.8.2-pve2
Last edited:
