# Intel Arc Pro B60 (Battlemage G21) PCIe link trains to Gen1 x1 under Linux `xe` driver, despite full-speed link at BIOS POST and on a different system
## Summary
An Intel Arc Pro B60 (24GB) consistently trains its PCIe link down to the
**minimum possible state (Gen1, x1)** when the `xe` driver loads under Linux,
on a system where BIOS POST itself reports the same device at a healthy
**Gen4 x8** link before the OS ever loads. The same card, moved to an
unrelated Windows system, links up at **Gen4 x8** even under Windows' generic
"Microsoft Basic Display Adapter" fallback driver (i.e. before any real
Intel/xe-equivalent driver touches it). This isolates the regression to
something in the Linux boot / `xe` driver PCI enumeration path on this
specific card family, not the card, not the motherboard, and not general
firmware/BIOS-level link training.
## Hardware / software
- GPU: Intel Arc Pro B60, 24GB (Battlemage G21, PCI ID `8086:e211`, device
reports "battlemage discrete display version 14.01 stepping B0")
- Card topology: card has an onboard PCIe switch (`8086:e2ff` upstream,
`8086:e2f0` downstream) between the slot and the actual GPU die — i.e. the
GPU function sits *behind* an on-card switch, not directly on the slot.
- Host: Proxmox VE, kernel `7.0.14-4-pve` (`xe` driver, in-tree)
- Motherboard: Gigabyte B760M DS3H AX DDR4 (Rev 1.x), BIOS F24 (also
reproduced on F22), CPU: Intel Core i5-12400
- GPU passed through to an unprivileged LXC container via `/dev/dri` device
nodes (not relevant to the link-training issue itself, which is visible
even from the Proxmox host directly)
## Symptom
`current_link_speed` / `current_link_width` for the GPU function
(`0000:03:00.0`) report `2.5 GT/s` / `x1` — both **current and reported
max** — i.e. the kernel believes Gen1 x1 is this link's ceiling, not just its
negotiated state:
```
$ cat /sys/bus/pci/devices/0000:03:00.0/current_link_speed
2.5 GT/s PCIe
$ cat /sys/bus/pci/devices/0000:03:00.0/current_link_width
1
$ cat /sys/bus/pci/devices/0000:03:00.0/max_link_speed
2.5 GT/s PCIe
$ cat /sys/bus/pci/devices/0000:03:00.0/max_link_width
1
```
The upstream hops are unaffected — CPU root port to the card's onboard
switch (`0000:01:00.0`) consistently trains at Gen4 x8 (its expected ceiling
given the platform), every time, on every boot. Only the switch's
**downstream** port to the GPU function (`0000:02:01.0` → `0000:03:00.0`)
collapses to the link floor.
Practical impact: `llama-bench` (llama.cpp, SYCL backend) on a 3B model shows
prompt-processing throughput that looks roughly sane (pp512 ≈ 51-93 tok/s
depending on which GPU was actually in use) but token-generation throughput
of **5.6-8.65 tok/s** — consistent with a starved, latency-bound link,
since tg does one small round-trip per token while pp batches and tolerates
latency better.
A heavier model (27B, full offload, `-ngl 99`) triggered a burst of kernel
`Fence expiration time out` errors from the GPU and the process hung
indefinitely — plausibly the same marginal link failing outright under
sustained transfer, though this hasn't been isolated as rigorously as the
link-speed regression itself.
## What's been ruled out (each tested with a full reboot, checking
`current_link_speed`/`width` fresh each time)
| Tried | Result |
|---|---|
| BIOS update F22 → F24 | No change |
| CMOS clear | No change |
| Resizable BAR / Above 4G Decoding / VT-d / XMP confirmed enabled | No change |
| `pci=realloc` kernel parameter | No change |
| Physical reseat of the card | No change |
| New power cable, different PSU connector/rail | No change |
| `xe.max_vfs=0` (disable SR-IOV PF mode) | "Running in SR-IOV PF mode" log line disappears, confirming the parameter took effect — **but link speed and the PCODE mailbox error below are unaffected** |
| Different physical system (Windows, generic driver only) | **Card links at Gen4 x8** — proves the card itself is healthy |
| Same motherboard's own BIOS PCIe status page (pre-OS) | Reports **Gen4 x8** for the same device — proves the board/firmware trains it correctly before Linux ever loads |
This combination — healthy at BIOS POST, healthy on unrelated hardware,
consistently degraded specifically when the Linux `xe` driver initializes on
this board — points at something in the Linux PCI/`xe` enumeration path,
not the hardware.
## Other log evidence
The kernel's initial resource allocation for this device is messy — it
reserves VF BAR space for all 24 SR-IOV `TotalVFs` the card advertises
(even with `xe.max_vfs=0` set — this reservation happens at the generic PCI
core level, reading the hardware's SR-IOV capability structure, independent
of the driver's own VF-creation parameter), fails to fit the combined
window on the first pass, and only succeeds after a release-and-reassign
retry:
```
pci 0000:03:00.0: VF BAR 0 [mem 0x00000000-0x17ffffff 64bit pref]: contains BAR 0 for 24 VFs
pci 0000:03:00.0: VF BAR 2 [mem size 0xc00000000 64bit pref]: can't assign; no space
pci 0000:03:00.0: VF BAR 2 [mem size 0xc00000000 64bit pref]: failed to assign
[... repeats several times ...]
pci 0000:02:01.0: bridge window [mem size 0x801000000 64bit pref] to [bus 03] add_size c18000000 add_align 800000000
pci 0000:02:01.0: bridge window [mem 0x4800000000-0x5c18ffffff 64bit pref]: assigned
```
Also present, every boot, right after the `xe` driver attaches:
```
xe 0000:03:00.0: [drm] Using mailbox commands for power limits
xe 0000:03:00.0: [drm] *ERROR* PCODE Mailbox failed: -6 Illegal Command
xe 0000:03:00.0: [drm] Thermal mailbox not supported by card firmware
```
Uncertain whether this mailbox failure is causally related to the link
training regression or a separate, cosmetic issue (other users report the
same PCODE error without a link-speed symptom), but it's the only other
Linux/`xe`-specific anomaly present alongside the degraded link.
## Question for maintainers / community
Is there a known interaction between the `xe` driver's SR-IOV/VF-BAR
resource-allocation retry path and downstream PCIe switch link training on
Battlemage cards that sit behind an on-card switch (as opposed to being
directly on the slot)? Given the card and motherboard are both independently
proven healthy outside this specific Linux/`xe` boot path, this looks like a
driver/kernel PCI-core issue rather than a hardware fault.
Happy to gather more diagnostics (`lspci -vvv`, full `dmesg`, specific
kernel versions/patches to test) on request.
## Summary
An Intel Arc Pro B60 (24GB) consistently trains its PCIe link down to the
**minimum possible state (Gen1, x1)** when the `xe` driver loads under Linux,
on a system where BIOS POST itself reports the same device at a healthy
**Gen4 x8** link before the OS ever loads. The same card, moved to an
unrelated Windows system, links up at **Gen4 x8** even under Windows' generic
"Microsoft Basic Display Adapter" fallback driver (i.e. before any real
Intel/xe-equivalent driver touches it). This isolates the regression to
something in the Linux boot / `xe` driver PCI enumeration path on this
specific card family, not the card, not the motherboard, and not general
firmware/BIOS-level link training.
## Hardware / software
- GPU: Intel Arc Pro B60, 24GB (Battlemage G21, PCI ID `8086:e211`, device
reports "battlemage discrete display version 14.01 stepping B0")
- Card topology: card has an onboard PCIe switch (`8086:e2ff` upstream,
`8086:e2f0` downstream) between the slot and the actual GPU die — i.e. the
GPU function sits *behind* an on-card switch, not directly on the slot.
- Host: Proxmox VE, kernel `7.0.14-4-pve` (`xe` driver, in-tree)
- Motherboard: Gigabyte B760M DS3H AX DDR4 (Rev 1.x), BIOS F24 (also
reproduced on F22), CPU: Intel Core i5-12400
- GPU passed through to an unprivileged LXC container via `/dev/dri` device
nodes (not relevant to the link-training issue itself, which is visible
even from the Proxmox host directly)
## Symptom
`current_link_speed` / `current_link_width` for the GPU function
(`0000:03:00.0`) report `2.5 GT/s` / `x1` — both **current and reported
max** — i.e. the kernel believes Gen1 x1 is this link's ceiling, not just its
negotiated state:
```
$ cat /sys/bus/pci/devices/0000:03:00.0/current_link_speed
2.5 GT/s PCIe
$ cat /sys/bus/pci/devices/0000:03:00.0/current_link_width
1
$ cat /sys/bus/pci/devices/0000:03:00.0/max_link_speed
2.5 GT/s PCIe
$ cat /sys/bus/pci/devices/0000:03:00.0/max_link_width
1
```
The upstream hops are unaffected — CPU root port to the card's onboard
switch (`0000:01:00.0`) consistently trains at Gen4 x8 (its expected ceiling
given the platform), every time, on every boot. Only the switch's
**downstream** port to the GPU function (`0000:02:01.0` → `0000:03:00.0`)
collapses to the link floor.
Practical impact: `llama-bench` (llama.cpp, SYCL backend) on a 3B model shows
prompt-processing throughput that looks roughly sane (pp512 ≈ 51-93 tok/s
depending on which GPU was actually in use) but token-generation throughput
of **5.6-8.65 tok/s** — consistent with a starved, latency-bound link,
since tg does one small round-trip per token while pp batches and tolerates
latency better.
A heavier model (27B, full offload, `-ngl 99`) triggered a burst of kernel
`Fence expiration time out` errors from the GPU and the process hung
indefinitely — plausibly the same marginal link failing outright under
sustained transfer, though this hasn't been isolated as rigorously as the
link-speed regression itself.
## What's been ruled out (each tested with a full reboot, checking
`current_link_speed`/`width` fresh each time)
| Tried | Result |
|---|---|
| BIOS update F22 → F24 | No change |
| CMOS clear | No change |
| Resizable BAR / Above 4G Decoding / VT-d / XMP confirmed enabled | No change |
| `pci=realloc` kernel parameter | No change |
| Physical reseat of the card | No change |
| New power cable, different PSU connector/rail | No change |
| `xe.max_vfs=0` (disable SR-IOV PF mode) | "Running in SR-IOV PF mode" log line disappears, confirming the parameter took effect — **but link speed and the PCODE mailbox error below are unaffected** |
| Different physical system (Windows, generic driver only) | **Card links at Gen4 x8** — proves the card itself is healthy |
| Same motherboard's own BIOS PCIe status page (pre-OS) | Reports **Gen4 x8** for the same device — proves the board/firmware trains it correctly before Linux ever loads |
This combination — healthy at BIOS POST, healthy on unrelated hardware,
consistently degraded specifically when the Linux `xe` driver initializes on
this board — points at something in the Linux PCI/`xe` enumeration path,
not the hardware.
## Other log evidence
The kernel's initial resource allocation for this device is messy — it
reserves VF BAR space for all 24 SR-IOV `TotalVFs` the card advertises
(even with `xe.max_vfs=0` set — this reservation happens at the generic PCI
core level, reading the hardware's SR-IOV capability structure, independent
of the driver's own VF-creation parameter), fails to fit the combined
window on the first pass, and only succeeds after a release-and-reassign
retry:
```
pci 0000:03:00.0: VF BAR 0 [mem 0x00000000-0x17ffffff 64bit pref]: contains BAR 0 for 24 VFs
pci 0000:03:00.0: VF BAR 2 [mem size 0xc00000000 64bit pref]: can't assign; no space
pci 0000:03:00.0: VF BAR 2 [mem size 0xc00000000 64bit pref]: failed to assign
[... repeats several times ...]
pci 0000:02:01.0: bridge window [mem size 0x801000000 64bit pref] to [bus 03] add_size c18000000 add_align 800000000
pci 0000:02:01.0: bridge window [mem 0x4800000000-0x5c18ffffff 64bit pref]: assigned
```
Also present, every boot, right after the `xe` driver attaches:
```
xe 0000:03:00.0: [drm] Using mailbox commands for power limits
xe 0000:03:00.0: [drm] *ERROR* PCODE Mailbox failed: -6 Illegal Command
xe 0000:03:00.0: [drm] Thermal mailbox not supported by card firmware
```
Uncertain whether this mailbox failure is causally related to the link
training regression or a separate, cosmetic issue (other users report the
same PCODE error without a link-speed symptom), but it's the only other
Linux/`xe`-specific anomaly present alongside the degraded link.
## Question for maintainers / community
Is there a known interaction between the `xe` driver's SR-IOV/VF-BAR
resource-allocation retry path and downstream PCIe switch link training on
Battlemage cards that sit behind an on-card switch (as opposed to being
directly on the slot)? Given the card and motherboard are both independently
proven healthy outside this specific Linux/`xe` boot path, this looks like a
driver/kernel PCI-core issue rather than a hardware fault.
Happy to gather more diagnostics (`lspci -vvv`, full `dmesg`, specific
kernel versions/patches to test) on request.