I tried to be as explanative to the title. Possible problem is with the hba adapter (LSI SAS 2008) i am trying to passthrough to a TrueNas Scale VM.
- Checked passthrough requirements in bios (VT-X VT-d)
- set in /etc/kernel/cmdline (I use uefi boot) the extra parameters
root=ZFS=rpool/ROOT/pve-1 boot=zfs intel_iommu=on pcie_acs_override=downstream,multifunction (had to use those because the iommu group 1 which the hba existed was shared with something else of the motherboard. So this helped to split even more the groups and now hba resides in group 10 alone).
-Verifed IOMMU is enabled with dmesg | grep -e DMAR -e IOMMU
-inspected the IOMMU grouping numbers
for d in /sys/kernel/iommu_groups/*/devices/*; do n=${d#*/iommu_groups/*}; n=${n%%/*}; printf 'IOMMU group %s ' "$n"; lspci -nns "${d##*/}"; done;
There was where I figured out the hba shared group with another pci device and used the option downstream,multifunction
-In pve-blacklist.conf already has by itself blacklist nvidiafb which is irrelevant to the case
I didnt create /etc/modprobe.d/blacklist.conf file in order to blacklist drivers (thought that this is for gpu drivers only. Dont know how to check which drivers the hba is using) So in /etc/modprobe.d/ there is only the file pve-blacklist.conf and no blacklist.conf or vfio.conf o
Also /etc/initramfs-tools/modules is empty and nano /etc/modules as well (from extra options)
Last after VM creation I added from gui ->hardware the pci device by enabling the options Pci-Express All functions and Rom- Bar (maybe that causing the problem??)
Even though the disks upon the hba passed to the VM and created array etc etc... if I start the VM from cli with qm start 100 I get the following (still works though)
My questions are
1.a simple pci device pass through needs more steps like a gpu?
2.what does it mean ethtool: autonegotiation is unset or enabled, the speed and duplex are not writable. ????
3.why is that transition happening so many times
kernel: fwbr100i0: port 1(fwln100i0) entered blocking state
kernel: fwbr100i0: port 1(fwln100i0) entered forwarding state ???
4.Main problem I suppose
QEMU[7664]: kvm: vfio-pci: Cannot read device rom at 0000:01:00.0
QEMU[7664]: Device option ROM contents are probably invalid (check dmesg). proxyNAS QEMU[7664]: Skip option ROM probe with rombar=0, or load from file with romfile=
kernel: vfio-pci 0000:01:00.0: Invalid PCI ROM header signature: expecting 0xaa55, got 0xffff
extra message vfio-pci 0000:01:00.0: Invalid PCI ROM header signature: expexting 0xaa55, got 0xffff
5.Any other weird messages you see?
Sorry for the long post
New Edit
After a while in syslog I get this
All the above disks are the ones passed through to the VM
ps Asrock H370ac/itx mobo / 32gb ram /zfs mirror for OS / cpu 8700 genuine engineering sample
- Checked passthrough requirements in bios (VT-X VT-d)
- set in /etc/kernel/cmdline (I use uefi boot) the extra parameters
root=ZFS=rpool/ROOT/pve-1 boot=zfs intel_iommu=on pcie_acs_override=downstream,multifunction (had to use those because the iommu group 1 which the hba existed was shared with something else of the motherboard. So this helped to split even more the groups and now hba resides in group 10 alone).
-Verifed IOMMU is enabled with dmesg | grep -e DMAR -e IOMMU
-inspected the IOMMU grouping numbers
for d in /sys/kernel/iommu_groups/*/devices/*; do n=${d#*/iommu_groups/*}; n=${n%%/*}; printf 'IOMMU group %s ' "$n"; lspci -nns "${d##*/}"; done;
There was where I figured out the hba shared group with another pci device and used the option downstream,multifunction
-In pve-blacklist.conf already has by itself blacklist nvidiafb which is irrelevant to the case
I didnt create /etc/modprobe.d/blacklist.conf file in order to blacklist drivers (thought that this is for gpu drivers only. Dont know how to check which drivers the hba is using) So in /etc/modprobe.d/ there is only the file pve-blacklist.conf and no blacklist.conf or vfio.conf o
Also /etc/initramfs-tools/modules is empty and nano /etc/modules as well (from extra options)
Last after VM creation I added from gui ->hardware the pci device by enabling the options Pci-Express All functions and Rom- Bar (maybe that causing the problem??)
Even though the disks upon the hba passed to the VM and created array etc etc... if I start the VM from cli with qm start 100 I get the following (still works though)
Code:
proxyNAS qm[7651]: start VM 100: UPID:proxyNAS:00001DE3:0000632D:618EC03A:qmstart:100:root@pam:
proxyNAS systemd[1]: Started 100.scope.
proxyNAS systemd-udevd[7673]: Using default interface naming scheme 'v247'.
proxyNAS systemd-udevd[7673]: ethtool: autonegotiation is unset or enabled, the speed and duplex are not writable.
proxyNAS kernel: device tap100i0 entered promiscuous mode
proxyNAS systemd-udevd[7672]: Using default interface naming scheme 'v247'.
proxyNAS systemd-udevd[7672]: ethtool: autonegotiation is unset or enabled, the speed and duplex are not writable.
proxyNAS systemd-udevd[7673]: ethtool: autonegotiation is unset or enabled, the speed and duplex are not writable.
proxyNAS systemd-udevd[7691]: ethtool: autonegotiation is unset or enabled, the speed and duplex are not writable.
proxyNAS systemd-udevd[7691]: Using default interface naming scheme 'v247'.
proxyNAS kernel: fwbr100i0: port 1(fwln100i0) entered blocking state
proxyNAS kernel: device fwln100i0 entered promiscuous mode
proxyNAS kernel: fwbr100i0: port 1(fwln100i0) entered blocking state
proxyNAS kernel: fwbr100i0: port 1(fwln100i0) entered forwarding state
proxyNAS kernel: vmbr0: port 2(fwpr100p0) entered blocking state
proxyNAS kernel: vmbr0: port 2(fwpr100p0) entered disabled state
proxyNAS kernel: device fwpr100p0 entered promiscuous mode
proxyNAS kernel: device eno1 entered promiscuous mode
proxyNAS kernel: vmbr0: port 2(fwpr100p0) entered blocking state
proxyNAS kernel: vmbr0: port 2(fwpr100p0) entered forwarding state
proxyNAS kernel: fwbr100i0: port 2(tap100i0) entered blocking state
proxyNAS kernel: fwbr100i0: port 2(tap100i0) entered disabled state
proxyNAS kernel: fwbr100i0: port 2(tap100i0) entered blocking state
proxyNAS kernel: fwbr100i0: port 2(tap100i0) entered forwarding state
proxyNAS qm[7650]: <root@pam> end task UPID:proxyNAS:00001DE3:0000632D:618EC03A:qmstart:100:root@pam: OK
proxyNAS QEMU[7664]: kvm: vfio-pci: Cannot read device rom at 0000:01:00.0
proxyNAS QEMU[7664]: Device option ROM contents are probably invalid (check dmesg). proxyNAS QEMU[7664]: Skip option ROM probe with rombar=0, or load from file with romfile=
proxyNAS kernel: vfio-pci 0000:01:00.0: Invalid PCI ROM header signature: expecting 0xaa55, got 0xffff
extra message displayed only from within proxmox shell by attaching a monitor to the server
vfio-pci 0000:01:00.0: Invalid PCI ROM header signature: expexting 0xaa55, got 0xffffMy questions are
1.a simple pci device pass through needs more steps like a gpu?
2.what does it mean ethtool: autonegotiation is unset or enabled, the speed and duplex are not writable. ????
3.why is that transition happening so many times
kernel: fwbr100i0: port 1(fwln100i0) entered blocking state
kernel: fwbr100i0: port 1(fwln100i0) entered forwarding state ???
4.Main problem I suppose
QEMU[7664]: kvm: vfio-pci: Cannot read device rom at 0000:01:00.0
QEMU[7664]: Device option ROM contents are probably invalid (check dmesg). proxyNAS QEMU[7664]: Skip option ROM probe with rombar=0, or load from file with romfile=
kernel: vfio-pci 0000:01:00.0: Invalid PCI ROM header signature: expecting 0xaa55, got 0xffff
extra message vfio-pci 0000:01:00.0: Invalid PCI ROM header signature: expexting 0xaa55, got 0xffff
5.Any other weird messages you see?
Sorry for the long post
New Edit
After a while in syslog I get this
Code:
smartd[1804]: Device: /dev/sda [SAT], removed ATA device: No such device
smartd[1804]: Device: /dev/sdb [SAT], removed ATA device: No such device
smartd[1804]: Device: /dev/sdc [SAT], removed ATA device: No such device
smartd[1804]: Device: /dev/sdd [SAT], removed ATA device: No such device
smartd[1804]: Device: /dev/sde [SAT], removed ATA device: No such device
smartd[1804]: Device: /dev/sdf [SAT], removed ATA device: No such deviceAll the above disks are the ones passed through to the VM
ps Asrock H370ac/itx mobo / 32gb ram /zfs mirror for OS / cpu 8700 genuine engineering sample
Last edited: