pci passthrough, VM unable to boot.

W

why-be-banned

Member
Apr 4, 2023
59
7
13
I passed through an i226 NIC to VM ,
boot error:

error writing '1' to '/sys/bus/pci/devices/0000:01:00.0/reset': Inappropriate ioctl for device
failed to reset PCI device '0000:01:00.0', but trying to continue as not all devices need a reset
kvm: -device vfio-pci,host=0000:01:00.0,id=hostpci0,bus=pci.0,addr=0x10: vfio 0000:01:00.0: error getting device from group 13: No such device
Verify all devices in group 13 are bound to vfio-<bus> or pci-stub and not already in use
TASK ERROR: start failed: QEMU exited with code 1
Click to expand...

VM config:
root@pve:~# qm config 300
boot: order=scsi0
cores: 4
cpu: x86-64-v2-AES
hostpci0: 0000:01:00.0
memory: 700
meta: creation-qemu=9.0.2,ctime=1733239234
name: openwrt
net0: virtio=E0:EF:02:5F:53:C0,bridge=vmbr0
numa: 0
ostype: l26
scsi0: local:300/vm-300-disk-0.raw,discard=on,iothread=1,size=123392K,ssd=1
scsihw: virtio-scsi-single
smbios1: uuid=2898e8da-851d-4d08-b5dc-077aa2190fb1
sockets: 1
startup: down=10
vmgenid: a591a78d-8d98-419a-ac6e-4087f801647e
Click to expand...

passthrough configuration:

root@pve:~# cat /etc/modules
# /etc/modules: kernel modules to load at boot time.
#
# This file contains the names of kernel modules that should be loaded
# at boot time, one per line. Lines beginning with "#" are ignored.
# Parameters can be specified after the module name.

vfio
vfio_iommu_type1
vfio_pci
vfio_virqfd
Click to expand...

root@pve:~# cat /etc/default/grub
# If you change this file, run 'update-grub' afterwards to update
# /boot/grub/grub.cfg.
# For full documentation of the options in this file, see:
# info -f grub -n 'Simple configuration'

GRUB_DEFAULT=0
GRUB_TIMEOUT=5
GRUB_DISTRIBUTOR=`lsb_release -i -s 2> /dev/null || echo Debian`
GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on iommu=pt intel_pstate=disable"
GRUB_CMDLINE_LINUX=""
Click to expand...

批注 2025-05-20 141424.jpg
 
Last edited:
Verify all devices in group 13 are bound to vfio-<bus> or pci-stub and not already in use
Click to expand...

That looks like an IOMMU group issue.

Have you checked that the NIC/slot is actually isolated in its own group as shown here, with the command:
Code: 
pvesh get /nodes/{nodename}/hardware/pci --pci-class-blacklist ""
 
BTW - from your above image it appears you have not setup any repos to update your PVE instance. This is probably not advised.

Also you appear to have the latest PVE Manager Version 8.4.1/2a5fa54a8503f96d but not the latest kernel. This is odd. New install from latest ISO of April 9th 2025 should probably (I may be wrong) have a newer kernel than your 6.8.12-4 from Nov 2024. Some kernel pinning activity?
 
gfngfn256 said:
That looks like an IOMMU group issue.

Have you checked that the NIC/slot is actually isolated in its own group as shown here, with the command:
Code: 
pvesh get /nodes/{nodename}/hardware/pci --pci-class-blacklist ""
Click to expand...

Thank you. please help check it:

root@pve:~# pvesh get /nodes/pve/hardware/pci --pci-class-blacklist ""
┌──────────┬────────┬──────────────┬────────────┬────────┬───────────────────────────────────────────────────────────────────────────────────────────────┬──────┬─────────
│ class │ device │ id │ iommugroup │ vendor │ device_name │ mdev │ subsyste
╞══════════╪════════╪══════════════╪════════════╪════════╪═══════════════════════════════════════════════════════════════════════════════════════════════╪══════╪═════════
│ 0x010601 │ 0x31e3 │ 0000:00:12.0 │ 4 │ 0x8086 │ Celeron/Pentium Silver Processor SATA Controller │ │ 0x7270
├──────────┼────────┼──────────────┼────────────┼────────┼───────────────────────────────────────────────────────────────────────────────────────────────┼──────┼─────────
│ 0x010802 │ 0x002a │ 0000:05:00.0 │ 17 │ 0x1e0f │ │ │ 0x00fb
├──────────┼────────┼──────────────┼────────────┼────────┼───────────────────────────────────────────────────────────────────────────────────────────────┼──────┼─────────
│ 0x020000 │ 0x125c │ 0000:01:00.0 │ 13 │ 0x8086 │ Ethernet Controller I226-V │ │ 0x0000
├──────────┼────────┼──────────────┼────────────┼────────┼───────────────────────────────────────────────────────────────────────────────────────────────┼──────┼─────────
│ 0x020000 │ 0x125c │ 0000:02:00.0 │ 14 │ 0x8086 │ Ethernet Controller I226-V │ │ 0x0000
├──────────┼────────┼──────────────┼────────────┼────────┼───────────────────────────────────────────────────────────────────────────────────────────────┼──────┼─────────
│ 0x020000 │ 0x125c │ 0000:03:00.0 │ 15 │ 0x8086 │ Ethernet Controller I226-V │ │ 0x0000
├──────────┼────────┼──────────────┼────────────┼────────┼───────────────────────────────────────────────────────────────────────────────────────────────┼──────┼─────────
│ 0x020000 │ 0x125c │ 0000:04:00.0 │ 16 │ 0x8086 │ Ethernet Controller I226-V │ │ 0x0000
├──────────┼────────┼──────────────┼────────────┼────────┼───────────────────────────────────────────────────────────────────────────────────────────────┼──────┼─────────
│ 0x030000 │ 0x3185 │ 0000:00:02.0 │ 0 │ 0x8086 │ GeminiLake [UHD Graphics 600] │ │ 0x0000
├──────────┼────────┼──────────────┼────────────┼────────┼───────────────────────────────────────────────────────────────────────────────────────────────┼──────┼─────────
│ 0x040300 │ 0x3198 │ 0000:00:0e.0 │ 2 │ 0x8086 │ Celeron/Pentium Silver Processor High Definition Audio │ │ 0x7270
├──────────┼────────┼──────────────┼────────────┼────────┼───────────────────────────────────────────────────────────────────────────────────────────────┼──────┼─────────
│ 0x060000 │ 0x31f0 │ 0000:00:00.0 │ 1 │ 0x8086 │ Gemini Lake Host Bridge │ │ 0x0000
├──────────┼────────┼──────────────┼────────────┼────────┼───────────────────────────────────────────────────────────────────────────────────────────────┼──────┼─────────
│ 0x060100 │ 0x31e8 │ 0000:00:1f.0 │ 12 │ 0x8086 │ Celeron/Pentium Silver Processor LPC Controller │ │ 0x7270
├──────────┼────────┼──────────────┼────────────┼────────┼───────────────────────────────────────────────────────────────────────────────────────────────┼──────┼─────────
│ 0x060400 │ 0x31d8 │ 0000:00:13.0 │ 5 │ 0x8086 │ Gemini Lake PCI Express Root Port │ │ 0x7270
├──────────┼────────┼──────────────┼────────────┼────────┼───────────────────────────────────────────────────────────────────────────────────────────────┼──────┼─────────
│ 0x060400 │ 0x31d9 │ 0000:00:13.1 │ 6 │ 0x8086 │ Gemini Lake PCI Express Root Port │ │ 0x7270
├──────────┼────────┼──────────────┼────────────┼────────┼───────────────────────────────────────────────────────────────────────────────────────────────┼──────┼─────────
│ 0x060400 │ 0x31da │ 0000:00:13.2 │ 7 │ 0x8086 │ Gemini Lake PCI Express Root Port │ │ 0x7270
├──────────┼────────┼──────────────┼────────────┼────────┼───────────────────────────────────────────────────────────────────────────────────────────────┼──────┼─────────
│ 0x060400 │ 0x31db │ 0000:00:13.3 │ 8 │ 0x8086 │ Gemini Lake PCI Express Root Port │ │ 0x7270
├──────────┼────────┼──────────────┼────────────┼────────┼───────────────────────────────────────────────────────────────────────────────────────────────┼──────┼─────────
│ 0x060400 │ 0x31d6 │ 0000:00:14.0 │ 9 │ 0x8086 │ Gemini Lake PCI Express Root Port │ │ 0x7270
├──────────┼────────┼──────────────┼────────────┼────────┼───────────────────────────────────────────────────────────────────────────────────────────────┼──────┼─────────
│ 0x060400 │ 0x31d7 │ 0000:00:14.1 │ 10 │ 0x8086 │ Gemini Lake PCI Express Root Port │ │ 0x7270
├──────────┼────────┼──────────────┼────────────┼────────┼───────────────────────────────────────────────────────────────────────────────────────────────┼──────┼─────────
│ 0x078000 │ 0x319a │ 0000:00:0f.0 │ 3 │ 0x8086 │ Celeron/Pentium Silver Processor Trusted Execution Engine Interface │ │ 0x7270
├──────────┼────────┼──────────────┼────────────┼────────┼───────────────────────────────────────────────────────────────────────────────────────────────┼──────┼─────────
│ 0x0c0330 │ 0x31a8 │ 0000:00:15.0 │ 11 │ 0x8086 │ Celeron/Pentium Silver Processor USB 3.0 xHCI Controller │ │ 0x7270
├──────────┼────────┼──────────────┼────────────┼────────┼───────────────────────────────────────────────────────────────────────────────────────────────┼──────┼─────────
│ 0x0c0500 │ 0x31d4 │ 0000:00:1f.1 │ 12 │ 0x8086 │ Celeron/Pentium Silver Processor Gaussian Mixture Model │ │ 0x7270
├──────────┼────────┼──────────────┼────────────┼────────┼───────────────────────────────────────────────────────────────────────────────────────────────┼──────┼─────────
│ 0x118000 │ 0x318c │ 0000:00:00.1 │ 1 │ 0x8086 │ Celeron/Pentium Silver Processor Dynamic Platform and Thermal Framework Processor Participant │ │ 0x7270
└──────────┴────────┴──────────────┴────────────┴────────┴───────────────────────────────────────────────────────────────────────────────────────────────┴──────┴─────────
root@pve:~#
Click to expand...
 

Attachments

  • 批注 2025-05-20 170924.jpg
    批注 2025-05-20 170924.jpg
    257.9 KB · Views: 3
Last edited:
OK. Although that I226-V NIC on bus 0000:01:00.0 does appear to have it's own IOMMU group of 13 - this does not automatically mean that it can be passed through.

Things to consider:
  • Are you using that NIC elsewhere? How is Proxmox NW setup? It may already be using it.
  • You appear to have 4 I226-V's NICs. It is possible that they cannot be segregated - due to either the specific card or MB/bus implementation.
  • Your MB is not correctly supporting passthrough for that specific NIC.
  • Read here on the various things to look out for.
Good luck.
 
gfngfn256 said:
OK. Although that I226-V NIC on bus 0000:01:00.0 does appear to have it's own IOMMU group of 13 - this does not automatically mean that it can be passed through.

Things to consider:
  • Are you using that NIC elsewhere? How is Proxmox NW setup? It may already be using it.
  • You appear to have 4 I226-V's NICs. It is possible that they cannot be segregated - due to either the specific card or MB/bus implementation.
  • Your MB is not correctly supporting passthrough for that specific NIC.
  • Read here on the various things to look out for.
Good luck.
Click to expand...

Thank you.

I use old kernel because the new kernel unable to boot, so I pin the old kernel.

I don't use this NIC elsewhere.

批注 2025-05-20 214447.jpg

enp1s0 is the NIC passed through.

passthrough integrated graphics works fine, but passthrough the NIC doesn't.

I do a fresh install of pve8.4, and it still doesn't work.

I do a fresh install of pve7.4, and it works ! passthrough the NIC works!

so, it's the problem in pve8.4 ?
 
Last edited:
why-be-banned said:
enp1s0 is the NIC passed through
Click to expand...
Are you 100% sure that this enp1s0 is the NIC on bus 0000:01:00.0 ?

You can check the logical name against the bus by using:
Code: 
lshw -class network

Are those NICS different physical cards or are they on a card incorporating numerous NICS? Maybe they are on the physical MB? This itself may hinder segregated PCI passthrough.

why-be-banned said:
I use old kernel because the new kernel unable to boot, so I pin the old kernel.
Click to expand...
You should still have update repos setup.

why-be-banned said:
so, it's the problem in pve8.4 ?
Click to expand...
Not necessarily. You may have gotten (lucky) a different logical name for that NIC on a different install. See above.
 
You must log in or register to reply here.
Top Bottom