LSI SAS card passthrough error

Discussion in 'Proxmox VE: Installation and configuration' started by flipper203, Jun 24, 2019.

  1. flipper203

    flipper203 New Member

    Joined:
    Jun 24, 2019
    Messages:
    7
    Likes Received:
    0
    Hello,

    I am trying to passthrough my LSI SAS9211-4i PCIE card to a VM on my brand new proxmox installation. I followed the official guide https://pve.proxmox.com/wiki/Pci_passthrough
    My setup is a HP microserver gen8 with a 1260l xeon, last proxmox instaleld on 2 SSD (in zfs mirror) connected to the internal HP controller. On the LSI card I have the four storage drives installed.

    Everything is going well until I try to start the VM, regarding the parameters to change to set the passthrough. I get the foillowing error and the VM don't boot:
    Code:
    Virtual Environment 5.4-6
        Virtual Machine 103 (2pFreenas) on node 'pve'
        Logs
        ()
        kvm: -device vfio-pci,host=07:00.0,id=hostpci0,bus=ich9-pcie-port-1,addr=0x0: vfio error: 0000:07:00.0: failed to setup container for group 1: failed to set iommu for container: Operation not permitted
        TASK ERROR: start failed: command '/usr/bin/kvm -id 103 -name 2pFreenas -chardev 'socket,id=qmp,path=/var/run/qemu-server/103.qmp,server,nowait' -mon 'chardev=qmp,mode=control' -chardev 'socket,id=qmp-event,path=/var/run/qmeventd.sock,reconnect=5' -mon 'chardev=qmp-event,mode=control' -pidfile /var/run/qemu-server/103.pid -daemonize -smbios 'type=1,uuid=0c2366f3-9576-45ce-9f92-508441ea3ed5' -smp '2,sockets=1,cores=2,maxcpus=2' -nodefaults -boot 'menu=on,strict=on,reboot-timeout=1000,splash=/usr/share/qemu-server/bootsplash.jpg' -vnc unix:/var/run/qemu-server/103.vnc,x509,password -cpu kvm64,+lahf_lm,+sep,+kvm_pv_unhalt,+kvm_pv_eoi,enforce -m 10192 -device 'vmgenid,guid=06ceb1f6-d7db-4cec-9025-d70335dbe1b2' -readconfig /usr/share/qemu-server/pve-q35.cfg -device 'usb-tablet,id=tablet,bus=ehci.0,port=1' -device 'vfio-pci,host=07:00.0,id=hostpci0,bus=ich9-pcie-port-1,addr=0x0' -device 'VGA,id=vga,bus=pcie.0,addr=0x1' -device 'virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x3' -iscsi 'initiator-name=iqn.1993-08.org.debian:01:a95eca963df1' -device 'virtio-scsi-pci,id=scsihw0,bus=pci.0,addr=0x5' -drive 'file=/dev/zvol/rpool/data/vm-103-disk-0,if=none,id=drive-scsi0,format=raw,cache=none,aio=native,detect-zeroes=on' -device 'scsi-hd,bus=scsihw0.0,channel=0,scsi-id=0,lun=0,drive=drive-scsi0,id=scsi0,bootindex=100' -netdev 'type=tap,id=net0,ifname=tap103i0,script=/var/lib/qemu-server/pve-bridge,downscript=/var/lib/qemu-server/pve-bridgedown,vhost=on' -device 'virtio-net-pci,mac=4E:77:B9:7A:E5:19,netdev=net0,bus=pci.0,addr=0x12,id=net0,bootindex=300' -machine 'type=q35'' failed: exit code 1
    If I add the following option driver=vfio for the hostpci0, the VM starts but I get the followwing problem:
    Code:
    vm 103 - unable to parse value of 'hostpci0' - format error
        driver: property is not defined in schema and the schema does not allow additional properties
    I also had to put the options vfio_iommu_type1 allow_unsafe_interrupts=1

    Any help on this?
     
  2. dcsapak

    dcsapak Proxmox Staff Member
    Staff Member

    Joined:
    Feb 1, 2016
    Messages:
    3,592
    Likes Received:
    325
    the official documentation is https://pve.proxmox.com/wiki/PCI(e)_Passthrough

    please post the vm config ( qm config ID ) as well as the iommu groups, and the dmesg output

    this is not a valid option in our config
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  3. flipper203

    flipper203 New Member

    Joined:
    Jun 24, 2019
    Messages:
    7
    Likes Received:
    0
    So, I followed the above link and I still have an error:

    Code:
    root@pve:~# dmesg | grep -e DMAR -e IOMMU -e AMD-Vi
    [ 0.000000] ACPI: DMAR 0x00000000F1DE4A80 0003B4 (v01 HP ProLiant 00000001 \xffffffd2? 0000162E)
    [ 0.000000] DMAR: IOMMU enabled
    [ 0.000000] DMAR: Host address width 39
    [ 0.000000] DMAR: DRHD base: 0x000000fed90000 flags: 0x1
    [ 0.000000] DMAR: dmar0: reg_base_addr fed90000 ver 1:0 cap c9008020660262 ecap f010da
    [ 0.000000] DMAR: RMRR base: 0x000000f1ffd000 end: 0x000000f1ffffff
    [ 0.000000] DMAR: RMRR base: 0x000000f1ff6000 end: 0x000000f1ffcfff
    [ 0.000000] DMAR: RMRR base: 0x000000f1f93000 end: 0x000000f1f94fff
    [ 0.000000] DMAR: RMRR base: 0x000000f1f8f000 end: 0x000000f1f92fff
    [ 0.000000] DMAR: RMRR base: 0x000000f1f7f000 end: 0x000000f1f8efff
    [ 0.000000] DMAR: RMRR base: 0x000000f1f7e000 end: 0x000000f1f7efff
    [ 0.000000] DMAR: RMRR base: 0x000000000f4000 end: 0x000000000f4fff
    [ 0.000000] DMAR: RMRR base: 0x000000000e8000 end: 0x000000000e8fff
    [ 0.000000] DMAR: RMRR base: 0x000000f1dee000 end: 0x000000f1deefff
    [ 0.000000] DMAR-IR: IOAPIC id 8 under DRHD base 0xfed90000 IOMMU 0
    [ 0.000000] DMAR-IR: HPET id 0 under DRHD base 0xfed90000
    [ 0.000000] DMAR-IR: x2apic is disabled because BIOS sets x2apic opt out bit.
    [ 0.000000] DMAR-IR: Use 'intremap=no_x2apic_optout' to override the BIOS setting.
    [ 0.000000] DMAR-IR: Enabled IRQ remapping in xapic mode
    [ 0.720514] DMAR: No ATSR found
    [ 0.720827] DMAR: dmar0: Using Queued invalidation
    [ 0.720835] DMAR: Setting RMRR:
    [ 0.721122] DMAR: Setting identity map for device 0000:01:00.0 [0xf1dee000 - 0xf1deefff]
    [ 0.721440] DMAR: Setting identity map for device 0000:01:00.2 [0xf1dee000 - 0xf1deefff]
    [ 0.721749] DMAR: Setting identity map for device 0000:01:00.4 [0xf1dee000 - 0xf1deefff]
    [ 0.722076] DMAR: Setting identity map for device 0000:00:1f.2 [0xe8000 - 0xe8fff]
    [ 0.722387] DMAR: Setting identity map for device 0000:07:00.0 [0xe8000 - 0xe8fff]
    [ 0.722718] DMAR: Setting identity map for device 0000:03:00.0 [0xe8000 - 0xe8fff]
    [ 0.723031] DMAR: Setting identity map for device 0000:03:00.1 [0xe8000 - 0xe8fff]
    [ 0.723360] DMAR: Setting identity map for device 0000:04:00.0 [0xe8000 - 0xe8fff]
    [ 0.723369] DMAR: Setting identity map for device 0000:01:00.0 [0xe8000 - 0xe8fff]
    [ 0.723379] DMAR: Setting identity map for device 0000:01:00.2 [0xe8000 - 0xe8fff]
    [ 0.723390] DMAR: Setting identity map for device 0000:00:1f.2 [0xf4000 - 0xf4fff]
    [ 0.723392] DMAR: Setting identity map for device 0000:07:00.0 [0xf4000 - 0xf4fff]
    [ 0.723394] DMAR: Setting identity map for device 0000:03:00.0 [0xf4000 - 0xf4fff]
    [ 0.723395] DMAR: Setting identity map for device 0000:03:00.1 [0xf4000 - 0xf4fff]
    [ 0.723396] DMAR: Setting identity map for device 0000:04:00.0 [0xf4000 - 0xf4fff]
    [ 0.723398] DMAR: Setting identity map for device 0000:01:00.0 [0xf4000 - 0xf4fff]
    [ 0.723399] DMAR: Setting identity map for device 0000:01:00.2 [0xf4000 - 0xf4fff]
    [ 0.723401] DMAR: Setting identity map for device 0000:00:1f.2 [0xf1f7e000 - 0xf1f7efff]
    [ 0.723410] DMAR: Setting identity map for device 0000:07:00.0 [0xf1f7e000 - 0xf1f7efff]
    [ 0.723418] DMAR: Setting identity map for device 0000:03:00.0 [0xf1f7e000 - 0xf1f7efff]
    [ 0.723427] DMAR: Setting identity map for device 0000:03:00.1 [0xf1f7e000 - 0xf1f7efff]
    [ 0.723436] DMAR: Setting identity map for device 0000:04:00.0 [0xf1f7e000 - 0xf1f7efff]
    [ 0.723445] DMAR: Setting identity map for device 0000:01:00.0 [0xf1f7e000 - 0xf1f7efff]
    [ 0.723450] DMAR: Setting identity map for device 0000:01:00.2 [0xf1f7e000 - 0xf1f7efff]
    [ 0.723455] DMAR: Setting identity map for device 0000:00:1f.2 [0xf1f7f000 - 0xf1f8efff]
    [ 0.723457] DMAR: Setting identity map for device 0000:07:00.0 [0xf1f7f000 - 0xf1f8efff]
    [ 0.723459] DMAR: Setting identity map for device 0000:03:00.0 [0xf1f7f000 - 0xf1f8efff]
    [ 0.723460] DMAR: Setting identity map for device 0000:03:00.1 [0xf1f7f000 - 0xf1f8efff]
    [ 0.723462] DMAR: Setting identity map for device 0000:04:00.0 [0xf1f7f000 - 0xf1f8efff]
    [ 0.723464] DMAR: Setting identity map for device 0000:01:00.0 [0xf1f7f000 - 0xf1f8efff]
    [ 0.723466] DMAR: Setting identity map for device 0000:01:00.2 [0xf1f7f000 - 0xf1f8efff]
    [ 0.723468] DMAR: Setting identity map for device 0000:00:1f.2 [0xf1f8f000 - 0xf1f92fff]
    [ 0.723470] DMAR: Setting identity map for device 0000:07:00.0 [0xf1f8f000 - 0xf1f92fff]
    [ 0.723472] DMAR: Setting identity map for device 0000:03:00.0 [0xf1f8f000 - 0xf1f92fff]
    [ 0.723473] DMAR: Setting identity map for device 0000:03:00.1 [0xf1f8f000 - 0xf1f92fff]
    [ 0.723475] DMAR: Setting identity map for device 0000:04:00.0 [0xf1f8f000 - 0xf1f92fff]
    [ 0.723476] DMAR: Setting identity map for device 0000:01:00.0 [0xf1f8f000 - 0xf1f92fff]
    [ 0.723478] DMAR: Setting identity map for device 0000:01:00.2 [0xf1f8f000 - 0xf1f92fff]
    [ 0.723480] DMAR: Setting identity map for device 0000:00:1f.2 [0xf1f93000 - 0xf1f94fff]
    [ 0.723481] DMAR: Setting identity map for device 0000:07:00.0 [0xf1f93000 - 0xf1f94fff]
    [ 0.723483] DMAR: Setting identity map for device 0000:03:00.0 [0xf1f93000 - 0xf1f94fff]
    [ 0.723484] DMAR: Setting identity map for device 0000:03:00.1 [0xf1f93000 - 0xf1f94fff]
    [ 0.723486] DMAR: Setting identity map for device 0000:04:00.0 [0xf1f93000 - 0xf1f94fff]
    [ 0.723487] DMAR: Setting identity map for device 0000:01:00.0 [0xf1f93000 - 0xf1f94fff]
    [ 0.723489] DMAR: Setting identity map for device 0000:01:00.2 [0xf1f93000 - 0xf1f94fff]
    [ 0.723490] DMAR: Setting identity map for device 0000:01:00.0 [0xf1ff6000 - 0xf1ffcfff]
    [ 0.723492] DMAR: Setting identity map for device 0000:01:00.2 [0xf1ff6000 - 0xf1ffcfff]
    [ 0.723493] DMAR: Setting identity map for device 0000:01:00.4 [0xf1ff6000 - 0xf1ffcfff]
    [ 0.723810] DMAR: Setting identity map for device 0000:00:1a.0 [0xf1ffd000 - 0xf1ffffff]
    [ 0.724160] DMAR: Setting identity map for device 0000:00:1d.0 [0xf1ffd000 - 0xf1ffffff]
    [ 0.724172] DMAR: Prepare 0-16MiB unity mapping for LPC
    [ 0.724482] DMAR: Setting identity map for device 0000:00:1f.0 [0x0 - 0xffffff]
    [ 0.724573] DMAR: Intel(R) Virtualization Technology for Directed I/O
    [ 1734.255087] vfio-pci 0000:07:00.0: Device is ineligible for IOMMU domain attach due to platform RMRR requirement. Contact your platform vendor.
    Code:
    root@pve:~# find /sys/kernel/iommu_groups/ -type l
    /sys/kernel/iommu_groups/7/devices/0000:00:1c.7
    /sys/kernel/iommu_groups/5/devices/0000:00:1c.4
    /sys/kernel/iommu_groups/13/devices/0000:01:00.4
    /sys/kernel/iommu_groups/13/devices/0000:01:00.2
    /sys/kernel/iommu_groups/13/devices/0000:01:00.0
    /sys/kernel/iommu_groups/13/devices/0000:01:00.1
    /sys/kernel/iommu_groups/3/devices/0000:00:1a.0
    /sys/kernel/iommu_groups/11/devices/0000:03:00.0
    /sys/kernel/iommu_groups/11/devices/0000:03:00.1
    /sys/kernel/iommu_groups/1/devices/0000:00:01.0
    /sys/kernel/iommu_groups/1/devices/0000:07:00.0
    /sys/kernel/iommu_groups/8/devices/0000:00:1d.0
    /sys/kernel/iommu_groups/6/devices/0000:00:1c.6
    /sys/kernel/iommu_groups/4/devices/0000:00:1c.0
    /sys/kernel/iommu_groups/12/devices/0000:04:00.0
    /sys/kernel/iommu_groups/2/devices/0000:00:06.0
    /sys/kernel/iommu_groups/10/devices/0000:00:1f.2
    /sys/kernel/iommu_groups/10/devices/0000:00:1f.0
    /sys/kernel/iommu_groups/0/devices/0000:00:00.0
    /sys/kernel/iommu_groups/9/devices/0000:00:1e.0
    Code:
    root@pve:~# cat /etc/pve/qemu-server/103.conf
    balloon: 4096
    bootdisk: scsi0
    cores: 2
    hostpci0: 07:00,pcie=on
    machine: q35
    memory: 10192
    name: 2pFreenas
    net0: virtio=4E:77:B9:7A:E5:19,bridge=vmbr0,firewall=1
    numa: 0
    ostype: l26
    parent: afterInitFreenas
    scsi0: local-zfs:vm-103-disk-0,size=32G
    scsihw: virtio-scsi-pci
    smbios1: uuid=0c2366f3-9576-45ce-9f92-508441ea3ed5
    sockets: 1
    vmgenid: 06ceb1f6-d7db-4cec-9025-d70335dbe1b2
    
    [afterInitFreenas]
    balloon: 4096
    bootdisk: scsi0
    cores: 2
    ide2: local:iso/FreeNAS-11.2-U4.1.iso,media=cdrom
    memory: 10192
    name: 2pFreenas
    net0: virtio=4E:77:B9:7A:E5:19,bridge=vmbr0,firewall=1
    numa: 0
    onboot: 1
    ostype: l26
    runningmachine: pc-i440fx-3.0
    scsi0: local-zfs:vm-103-disk-0,size=32G
    scsihw: virtio-scsi-pci
    smbios1: uuid=0c2366f3-9576-45ce-9f92-508441ea3ed5
    snaptime: 1558993031
    sockets: 1
    vmgenid: 06ceb1f6-d7db-4cec-9025-d70335dbe1b2
    vmstate: local-zfs:vm-103-state-afterInitFreenas
    I also blacklisted the mpt3sas driver.
    When I try to start the VM, I get the following error:

    Code:
    kvm: -device vfio-pci,host=07:00.0,id=hostpci0,bus=ich9-pcie-port-1,addr=0x0: vfio error: 0000:07:00.0: failed to setup container for group 1: failed to set iommu for container: Operation not permitted
    TASK ERROR: start failed: command '/usr/bin/kvm -id 103 -name 2pFreenas -chardev 'socket,id=qmp,path=/var/run/qemu-server/103.qmp,server,nowait' -mon 'chardev=qmp,mode=control' -chardev 'socket,id=qmp-event,path=/var/run/qmeventd.sock,reconnect=5' -mon 'chardev=qmp-event,mode=control' -pidfile /var/run/qemu-server/103.pid -daemonize -smbios 'type=1,uuid=0c2366f3-9576-45ce-9f92-508441ea3ed5' -smp '2,sockets=1,cores=2,maxcpus=2' -nodefaults -boot 'menu=on,strict=on,reboot-timeout=1000,splash=/usr/share/qemu-server/bootsplash.jpg' -vnc unix:/var/run/qemu-server/103.vnc,x509,password -cpu kvm64,+lahf_lm,+sep,+kvm_pv_unhalt,+kvm_pv_eoi,enforce -m 10192 -device 'vmgenid,guid=06ceb1f6-d7db-4cec-9025-d70335dbe1b2' -readconfig /usr/share/qemu-server/pve-q35.cfg -device 'usb-tablet,id=tablet,bus=ehci.0,port=1' -device 'vfio-pci,host=07:00.0,id=hostpci0,bus=ich9-pcie-port-1,addr=0x0' -device 'VGA,id=vga,bus=pcie.0,addr=0x1' -device 'virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x3' -iscsi 'initiator-name=iqn.1993-08.org.debian:01:a95eca963df1' -device 'virtio-scsi-pci,id=scsihw0,bus=pci.0,addr=0x5' -drive 'file=/dev/zvol/rpool/data/vm-103-disk-0,if=none,id=drive-scsi0,format=raw,cache=none,aio=native,detect-zeroes=on' -device 'scsi-hd,bus=scsihw0.0,channel=0,scsi-id=0,lun=0,drive=drive-scsi0,id=scsi0,bootindex=100' -netdev 'type=tap,id=net0,ifname=tap103i0,script=/var/lib/qemu-server/pve-bridge,downscript=/var/lib/qemu-server/pve-bridgedown,vhost=on' -device 'virtio-net-pci,mac=4E:77:B9:7A:E5:19,netdev=net0,bus=pci.0,addr=0x12,id=net0,bootindex=300' -machine 'type=q35'' failed: exit code 1
    Any help on this, I don't like the following message:
    Code:
    vfio-pci 0000:07:00.0: Device is ineligible for IOMMU domain attach due to platform RMRR requirement. Contact your platform vendor
    If I remember, before blacklisting the driver I didn't have this message....
     
  4. flipper203

    flipper203 New Member

    Joined:
    Jun 24, 2019
    Messages:
    7
    Likes Received:
    0
    I did again the update-initramfs and rebooted and I get the following dmesg without error now:
    Code:
    root@pve:~# dmesg | grep -e DMAR -e IOMMU
    [    0.000000] ACPI: DMAR 0x00000000F1DE4A80 0003B4 (v01 HP     ProLiant 00000001 \xffffffd2?   0000162E)
    [    0.000000] DMAR: IOMMU enabled
    [    0.000000] DMAR: Host address width 39
    [    0.000000] DMAR: DRHD base: 0x000000fed90000 flags: 0x1
    [    0.000000] DMAR: dmar0: reg_base_addr fed90000 ver 1:0 cap c9008020660262 ecap f010da
    [    0.000000] DMAR: RMRR base: 0x000000f1ffd000 end: 0x000000f1ffffff
    [    0.000000] DMAR: RMRR base: 0x000000f1ff6000 end: 0x000000f1ffcfff
    [    0.000000] DMAR: RMRR base: 0x000000f1f93000 end: 0x000000f1f94fff
    [    0.000000] DMAR: RMRR base: 0x000000f1f8f000 end: 0x000000f1f92fff
    [    0.000000] DMAR: RMRR base: 0x000000f1f7f000 end: 0x000000f1f8efff
    [    0.000000] DMAR: RMRR base: 0x000000f1f7e000 end: 0x000000f1f7efff
    [    0.000000] DMAR: RMRR base: 0x000000000f4000 end: 0x000000000f4fff
    [    0.000000] DMAR: RMRR base: 0x000000000e8000 end: 0x000000000e8fff
    [    0.000000] DMAR: RMRR base: 0x000000f1dee000 end: 0x000000f1deefff
    [    0.000000] DMAR-IR: IOAPIC id 8 under DRHD base  0xfed90000 IOMMU 0
    [    0.000000] DMAR-IR: HPET id 0 under DRHD base 0xfed90000
    [    0.000000] DMAR-IR: x2apic is disabled because BIOS sets x2apic opt out bit.
    [    0.000000] DMAR-IR: Use 'intremap=no_x2apic_optout' to override the BIOS setting.
    [    0.000000] DMAR-IR: Enabled IRQ remapping in xapic mode
    [    0.720582] DMAR: No ATSR found
    [    0.720893] DMAR: dmar0: Using Queued invalidation
    [    0.720902] DMAR: Setting RMRR:
    [    0.721187] DMAR: Setting identity map for device 0000:01:00.0 [0xf1dee000 - 0xf1deefff]
    [    0.721500] DMAR: Setting identity map for device 0000:01:00.2 [0xf1dee000 - 0xf1deefff]
    [    0.721806] DMAR: Setting identity map for device 0000:01:00.4 [0xf1dee000 - 0xf1deefff]
    [    0.722137] DMAR: Setting identity map for device 0000:00:1f.2 [0xe8000 - 0xe8fff]
    [    0.722446] DMAR: Setting identity map for device 0000:07:00.0 [0xe8000 - 0xe8fff]
    [    0.722772] DMAR: Setting identity map for device 0000:03:00.0 [0xe8000 - 0xe8fff]
    [    0.723088] DMAR: Setting identity map for device 0000:03:00.1 [0xe8000 - 0xe8fff]
    [    0.723415] DMAR: Setting identity map for device 0000:04:00.0 [0xe8000 - 0xe8fff]
    [    0.723424] DMAR: Setting identity map for device 0000:01:00.0 [0xe8000 - 0xe8fff]
    [    0.723434] DMAR: Setting identity map for device 0000:01:00.2 [0xe8000 - 0xe8fff]
    [    0.723444] DMAR: Setting identity map for device 0000:00:1f.2 [0xf4000 - 0xf4fff]
    [    0.723446] DMAR: Setting identity map for device 0000:07:00.0 [0xf4000 - 0xf4fff]
    [    0.723448] DMAR: Setting identity map for device 0000:03:00.0 [0xf4000 - 0xf4fff]
    [    0.723449] DMAR: Setting identity map for device 0000:03:00.1 [0xf4000 - 0xf4fff]
    [    0.723451] DMAR: Setting identity map for device 0000:04:00.0 [0xf4000 - 0xf4fff]
    [    0.723452] DMAR: Setting identity map for device 0000:01:00.0 [0xf4000 - 0xf4fff]
    [    0.723454] DMAR: Setting identity map for device 0000:01:00.2 [0xf4000 - 0xf4fff]
    [    0.723455] DMAR: Setting identity map for device 0000:00:1f.2 [0xf1f7e000 - 0xf1f7efff]
    [    0.723464] DMAR: Setting identity map for device 0000:07:00.0 [0xf1f7e000 - 0xf1f7efff]
    [    0.723473] DMAR: Setting identity map for device 0000:03:00.0 [0xf1f7e000 - 0xf1f7efff]
    [    0.723481] DMAR: Setting identity map for device 0000:03:00.1 [0xf1f7e000 - 0xf1f7efff]
    [    0.723490] DMAR: Setting identity map for device 0000:04:00.0 [0xf1f7e000 - 0xf1f7efff]
    [    0.723499] DMAR: Setting identity map for device 0000:01:00.0 [0xf1f7e000 - 0xf1f7efff]
    [    0.723504] DMAR: Setting identity map for device 0000:01:00.2 [0xf1f7e000 - 0xf1f7efff]
    [    0.723509] DMAR: Setting identity map for device 0000:00:1f.2 [0xf1f7f000 - 0xf1f8efff]
    [    0.723511] DMAR: Setting identity map for device 0000:07:00.0 [0xf1f7f000 - 0xf1f8efff]
    [    0.723512] DMAR: Setting identity map for device 0000:03:00.0 [0xf1f7f000 - 0xf1f8efff]
    [    0.723514] DMAR: Setting identity map for device 0000:03:00.1 [0xf1f7f000 - 0xf1f8efff]
    [    0.723516] DMAR: Setting identity map for device 0000:04:00.0 [0xf1f7f000 - 0xf1f8efff]
    [    0.723518] DMAR: Setting identity map for device 0000:01:00.0 [0xf1f7f000 - 0xf1f8efff]
    [    0.723520] DMAR: Setting identity map for device 0000:01:00.2 [0xf1f7f000 - 0xf1f8efff]
    [    0.723522] DMAR: Setting identity map for device 0000:00:1f.2 [0xf1f8f000 - 0xf1f92fff]
    [    0.723523] DMAR: Setting identity map for device 0000:07:00.0 [0xf1f8f000 - 0xf1f92fff]
    [    0.723525] DMAR: Setting identity map for device 0000:03:00.0 [0xf1f8f000 - 0xf1f92fff]
    [    0.723527] DMAR: Setting identity map for device 0000:03:00.1 [0xf1f8f000 - 0xf1f92fff]
    [    0.723528] DMAR: Setting identity map for device 0000:04:00.0 [0xf1f8f000 - 0xf1f92fff]
    [    0.723530] DMAR: Setting identity map for device 0000:01:00.0 [0xf1f8f000 - 0xf1f92fff]
    [    0.723532] DMAR: Setting identity map for device 0000:01:00.2 [0xf1f8f000 - 0xf1f92fff]
    [    0.723533] DMAR: Setting identity map for device 0000:00:1f.2 [0xf1f93000 - 0xf1f94fff]
    [    0.723535] DMAR: Setting identity map for device 0000:07:00.0 [0xf1f93000 - 0xf1f94fff]
    [    0.723536] DMAR: Setting identity map for device 0000:03:00.0 [0xf1f93000 - 0xf1f94fff]
    [    0.723537] DMAR: Setting identity map for device 0000:03:00.1 [0xf1f93000 - 0xf1f94fff]
    [    0.723539] DMAR: Setting identity map for device 0000:04:00.0 [0xf1f93000 - 0xf1f94fff]
    [    0.723541] DMAR: Setting identity map for device 0000:01:00.0 [0xf1f93000 - 0xf1f94fff]
    [    0.723542] DMAR: Setting identity map for device 0000:01:00.2 [0xf1f93000 - 0xf1f94fff]
    [    0.723544] DMAR: Setting identity map for device 0000:01:00.0 [0xf1ff6000 - 0xf1ffcfff]
    [    0.723545] DMAR: Setting identity map for device 0000:01:00.2 [0xf1ff6000 - 0xf1ffcfff]
    [    0.723547] DMAR: Setting identity map for device 0000:01:00.4 [0xf1ff6000 - 0xf1ffcfff]
    [    0.723863] DMAR: Setting identity map for device 0000:00:1a.0 [0xf1ffd000 - 0xf1ffffff]
    [    0.724211] DMAR: Setting identity map for device 0000:00:1d.0 [0xf1ffd000 - 0xf1ffffff]
    [    0.724221] DMAR: Prepare 0-16MiB unity mapping for LPC
    [    0.724531] DMAR: Setting identity map for device 0000:00:1f.0 [0x0 - 0xffffff]
    [    0.724622] DMAR: Intel(R) Virtualization Technology for Directed I/O
    but when I attach it to a VM, I get the same message as in the previous post. What am I missing ?
     
  5. dcsapak

    dcsapak Proxmox Staff Member
    Staff Member

    Joined:
    Feb 1, 2016
    Messages:
    3,592
    Likes Received:
    325
    this means that either your mainboard or the mainboard/device combo has some platform restriction for passthrough, sometimes a bios/firmware upgrade can help, but if not, there is not much you can do
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  6. flipper203

    flipper203 New Member

    Joined:
    Jun 24, 2019
    Messages:
    7
    Likes Received:
    0
    the thing is when I reboot, before I attatch the device to a VM and try to boot it, I don't get this message in the dmesg command
     
  7. dcsapak

    dcsapak Proxmox Staff Member
    Staff Member

    Joined:
    Feb 1, 2016
    Messages:
    3,592
    Likes Received:
    325
    yes, when the vm tries to passthrough it, the error occurs ... i am not sure what you are saying?
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  8. flipper203

    flipper203 New Member

    Joined:
    Jun 24, 2019
    Messages:
    7
    Likes Received:
    0
    If I reboot the proxmox host, without any vm configured as passthrough, I get the following result:
    Code:
    root@pve:~# dmesg | grep -e DMAR -e IOMMU
    [    0.000000] ACPI: DMAR 0x00000000F1DE4A80 0003B4 (v01 HP     ProLiant 00000001 \xffffffd2?   0000162E)
    [    0.000000] DMAR: IOMMU enabled
    [    0.000000] DMAR: Host address width 39
    [    0.000000] DMAR: DRHD base: 0x000000fed90000 flags: 0x1
    [    0.000000] DMAR: dmar0: reg_base_addr fed90000 ver 1:0 cap c9008020660262 ecap f010da
    [    0.000000] DMAR: RMRR base: 0x000000f1ffd000 end: 0x000000f1ffffff
    [    0.000000] DMAR: RMRR base: 0x000000f1ff6000 end: 0x000000f1ffcfff
    [    0.000000] DMAR: RMRR base: 0x000000f1f93000 end: 0x000000f1f94fff
    [    0.000000] DMAR: RMRR base: 0x000000f1f8f000 end: 0x000000f1f92fff
    [    0.000000] DMAR: RMRR base: 0x000000f1f7f000 end: 0x000000f1f8efff
    [    0.000000] DMAR: RMRR base: 0x000000f1f7e000 end: 0x000000f1f7efff
    [    0.000000] DMAR: RMRR base: 0x000000000f4000 end: 0x000000000f4fff
    [    0.000000] DMAR: RMRR base: 0x000000000e8000 end: 0x000000000e8fff
    [    0.000000] DMAR: RMRR base: 0x000000f1dee000 end: 0x000000f1deefff
    [    0.000000] DMAR-IR: IOAPIC id 8 under DRHD base  0xfed90000 IOMMU 0
    [    0.000000] DMAR-IR: HPET id 0 under DRHD base 0xfed90000
    [    0.000000] DMAR-IR: x2apic is disabled because BIOS sets x2apic opt out bit.
    [    0.000000] DMAR-IR: Use 'intremap=no_x2apic_optout' to override the BIOS setting.
    [    0.000000] DMAR-IR: Enabled IRQ remapping in xapic mode
    [    0.720453] DMAR: No ATSR found
    [    0.720759] DMAR: dmar0: Using Queued invalidation
    [    0.720767] DMAR: Setting RMRR:
    [    0.721055] DMAR: Setting identity map for device 0000:01:00.0 [0xf1dee000 - 0xf1deefff]
    [    0.721369] DMAR: Setting identity map for device 0000:01:00.2 [0xf1dee000 - 0xf1deefff]
    [    0.721674] DMAR: Setting identity map for device 0000:01:00.4 [0xf1dee000 - 0xf1deefff]
    [    0.722001] DMAR: Setting identity map for device 0000:00:1f.2 [0xe8000 - 0xe8fff]
    [    0.722312] DMAR: Setting identity map for device 0000:07:00.0 [0xe8000 - 0xe8fff]
    [    0.722641] DMAR: Setting identity map for device 0000:03:00.0 [0xe8000 - 0xe8fff]
    [    0.722953] DMAR: Setting identity map for device 0000:03:00.1 [0xe8000 - 0xe8fff]
    [    0.723282] DMAR: Setting identity map for device 0000:04:00.0 [0xe8000 - 0xe8fff]
    [    0.723292] DMAR: Setting identity map for device 0000:01:00.0 [0xe8000 - 0xe8fff]
    [    0.723302] DMAR: Setting identity map for device 0000:01:00.2 [0xe8000 - 0xe8fff]
    [    0.723313] DMAR: Setting identity map for device 0000:00:1f.2 [0xf4000 - 0xf4fff]
    [    0.723315] DMAR: Setting identity map for device 0000:07:00.0 [0xf4000 - 0xf4fff]
    [    0.723316] DMAR: Setting identity map for device 0000:03:00.0 [0xf4000 - 0xf4fff]
    [    0.723318] DMAR: Setting identity map for device 0000:03:00.1 [0xf4000 - 0xf4fff]
    [    0.723319] DMAR: Setting identity map for device 0000:04:00.0 [0xf4000 - 0xf4fff]
    [    0.723320] DMAR: Setting identity map for device 0000:01:00.0 [0xf4000 - 0xf4fff]
    [    0.723322] DMAR: Setting identity map for device 0000:01:00.2 [0xf4000 - 0xf4fff]
    [    0.723323] DMAR: Setting identity map for device 0000:00:1f.2 [0xf1f7e000 - 0xf1f7efff]
    [    0.723332] DMAR: Setting identity map for device 0000:07:00.0 [0xf1f7e000 - 0xf1f7efff]
    [    0.723341] DMAR: Setting identity map for device 0000:03:00.0 [0xf1f7e000 - 0xf1f7efff]
    [    0.723350] DMAR: Setting identity map for device 0000:03:00.1 [0xf1f7e000 - 0xf1f7efff]
    [    0.723358] DMAR: Setting identity map for device 0000:04:00.0 [0xf1f7e000 - 0xf1f7efff]
    [    0.723367] DMAR: Setting identity map for device 0000:01:00.0 [0xf1f7e000 - 0xf1f7efff]
    [    0.723372] DMAR: Setting identity map for device 0000:01:00.2 [0xf1f7e000 - 0xf1f7efff]
    [    0.723377] DMAR: Setting identity map for device 0000:00:1f.2 [0xf1f7f000 - 0xf1f8efff]
    [    0.723379] DMAR: Setting identity map for device 0000:07:00.0 [0xf1f7f000 - 0xf1f8efff]
    [    0.723381] DMAR: Setting identity map for device 0000:03:00.0 [0xf1f7f000 - 0xf1f8efff]
    [    0.723382] DMAR: Setting identity map for device 0000:03:00.1 [0xf1f7f000 - 0xf1f8efff]
    [    0.723384] DMAR: Setting identity map for device 0000:04:00.0 [0xf1f7f000 - 0xf1f8efff]
    [    0.723386] DMAR: Setting identity map for device 0000:01:00.0 [0xf1f7f000 - 0xf1f8efff]
    [    0.723388] DMAR: Setting identity map for device 0000:01:00.2 [0xf1f7f000 - 0xf1f8efff]
    [    0.723390] DMAR: Setting identity map for device 0000:00:1f.2 [0xf1f8f000 - 0xf1f92fff]
    [    0.723391] DMAR: Setting identity map for device 0000:07:00.0 [0xf1f8f000 - 0xf1f92fff]
    [    0.723393] DMAR: Setting identity map for device 0000:03:00.0 [0xf1f8f000 - 0xf1f92fff]
    [    0.723395] DMAR: Setting identity map for device 0000:03:00.1 [0xf1f8f000 - 0xf1f92fff]
    [    0.723396] DMAR: Setting identity map for device 0000:04:00.0 [0xf1f8f000 - 0xf1f92fff]
    [    0.723398] DMAR: Setting identity map for device 0000:01:00.0 [0xf1f8f000 - 0xf1f92fff]
    [    0.723400] DMAR: Setting identity map for device 0000:01:00.2 [0xf1f8f000 - 0xf1f92fff]
    [    0.723401] DMAR: Setting identity map for device 0000:00:1f.2 [0xf1f93000 - 0xf1f94fff]
    [    0.723403] DMAR: Setting identity map for device 0000:07:00.0 [0xf1f93000 - 0xf1f94fff]
    [    0.723404] DMAR: Setting identity map for device 0000:03:00.0 [0xf1f93000 - 0xf1f94fff]
    [    0.723406] DMAR: Setting identity map for device 0000:03:00.1 [0xf1f93000 - 0xf1f94fff]
    [    0.723407] DMAR: Setting identity map for device 0000:04:00.0 [0xf1f93000 - 0xf1f94fff]
    [    0.723409] DMAR: Setting identity map for device 0000:01:00.0 [0xf1f93000 - 0xf1f94fff]
    [    0.723410] DMAR: Setting identity map for device 0000:01:00.2 [0xf1f93000 - 0xf1f94fff]
    [    0.723412] DMAR: Setting identity map for device 0000:01:00.0 [0xf1ff6000 - 0xf1ffcfff]
    [    0.723413] DMAR: Setting identity map for device 0000:01:00.2 [0xf1ff6000 - 0xf1ffcfff]
    [    0.723415] DMAR: Setting identity map for device 0000:01:00.4 [0xf1ff6000 - 0xf1ffcfff]
    [    0.723731] DMAR: Setting identity map for device 0000:00:1a.0 [0xf1ffd000 - 0xf1ffffff]
    [    0.724105] DMAR: Setting identity map for device 0000:00:1d.0 [0xf1ffd000 - 0xf1ffffff]
    [    0.724116] DMAR: Prepare 0-16MiB unity mapping for LPC
    [    0.724426] DMAR: Setting identity map for device 0000:00:1f.0 [0x0 - 0xffffff]
    [    0.724516] DMAR: Intel(R) Virtualization Technology for Directed I/O
    So, no error. If I attach the PCIe device to the VM, and try to start it, I get the error above when trying to start the VM, and the dmesg command outputs the error again.
     
  9. dcsapak

    dcsapak Proxmox Staff Member
    Staff Member

    Joined:
    Feb 1, 2016
    Messages:
    3,592
    Likes Received:
    325
    yes the error occurs only when you try to passthrough... this is normal...
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  10. flipper203

    flipper203 New Member

    Joined:
    Jun 24, 2019
    Messages:
    7
    Likes Received:
    0
  1. This site uses cookies to help personalise content, tailor your experience and to keep you logged in if you register.
    By continuing to use this site, you are consenting to our use of cookies.
    Dismiss Notice