The PCI-E link speed of the Ryzen 9000 series is abnormal.

M

masato

Member
Mar 4, 2023
35
4
13
am building a RYZEN9 9950X environment to create a new GPU pass-through environment with the Rypzen9 9950X, but I feel that the PCI-E link speed is abnormal.

Motherboard: Gibagyte X870E Aorus Elite WiFi7
CPU: Reyzen9 9950X
GPU: Nvidia GTX750Ti, RTX4060Ti, AMD Radeon RX580
MEMORY: 128GB

When Linux boots, the PCI-E endpoint is determined to be legacy and is linked at Gen1.

The problem is that the SATA controller and USB 3.x controller are also linked at Gen1x1, so the connection bandwidth with the device is not reached.

The SATA controller has a link speed of 2.5GT/s x1, while the device is 6Gbps, so there is insufficient bandwidth.
Since lnkCap is displayed as 2.5GT/s x1, is it correct to say that it looks like a controller with a bandwidth below SATA?

The same is true for USB3.x, which is 5Gbps, 10Gbps, or 20Gbps compared to 2.5GT/s x1, so the bandwidth is insufficient.
LnkCap is also displayed as 2.5GT/s x1.

What about your motherboard?

dmesg|grep -e PCI |grep -e Legacy
Code: 
[    0.487533] pci 0000:01:00.0: [10de:1380] type 00 class 0x030000 PCIe Legacy Endpoint
[    0.499681] pci 0000:13:00.0: [1022:43fd] type 00 class 0x0c0330 PCIe Legacy Endpoint
[    0.499933] pci 0000:14:00.0: [1022:43f6] type 00 class 0x010601 PCIe Legacy Endpoint
[    0.500192] pci 0000:15:00.0: [1022:43fd] type 00 class 0x0c0330 PCIe Legacy Endpoint
[    0.500363] pci 0000:16:00.0: [1022:43f6] type 00 class 0x010601 PCIe Legacy Endpoint
[    0.502101] pci 0000:79:00.0: [1b21:2426] type 00 class 0x0c0330 PCIe Legacy Endpoint
[    0.502289] pci 0000:7a:00.0: [1b21:2425] type 00 class 0x0c0340 PCIe Legacy Endpoint
[    0.502580] pci 0000:7b:00.0: [1002:13c0] type 00 class 0x030000 PCIe Legacy Endpoint
[    0.502702] pci 0000:7b:00.1: [1002:1640] type 00 class 0x040300 PCIe Legacy Endpoint

for pcie in $(dmesg|grep -e PCI |grep -e Legacy | awk '{print $4}' | sed -e s/0000:// -e s/:$//); do lspci -vvvs $pcie | grep -e "^[0-9]" -e "Lnk" -e "Subsystem:" -e "IOMMU group"; done
Code: 
01:00.0 VGA compatible controller: NVIDIA Corporation GM107 [GeForce GTX 750 Ti] (rev a2) (prog-if 00 [VGA controller])
        Subsystem: NVIDIA Corporation GM107 [GeForce GTX 750 Ti]
        IOMMU group: 16
                LnkCap: Port #0, Speed 8GT/s, Width x16, ASPM L0s L1, Exit Latency L0s <512ns, L1 <4us
                LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
                LnkSta: Speed 2.5GT/s (downgraded), Width x8 (downgraded)
                LnkCap2: Supported Link Speeds: 2.5-8GT/s, Crosslink- Retimer- 2Retimers- DRS-
                LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis-
                LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete+ EqualizationPhase1+
                LnkCtl3: LnkEquIntrruptEn- PerformEqu-
13:00.0 USB controller: Advanced Micro Devices, Inc. [AMD] Device 43fd (rev 01) (prog-if 30 [XHCI])
        Subsystem: ASMedia Technology Inc. Device 1142
        IOMMU group: 42
                LnkCap: Port #0, Speed 2.5GT/s, Width x1, ASPM L0s L1, Exit Latency L0s <4us, L1 <64us
                LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
                LnkSta: Speed 2.5GT/s, Width x1
                LnkCap2: Supported Link Speeds: 2.5GT/s, Crosslink- Retimer+ 2Retimers+ DRS-
                LnkCtl2: Target Link Speed: 2.5GT/s, EnterCompliance- SpeedDis-
                LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete- EqualizationPhase1-
14:00.0 SATA controller: Advanced Micro Devices, Inc. [AMD] 600 Series Chipset SATA Controller (rev 01) (prog-if 01 [AHCI 1.0])
        Subsystem: ASMedia Technology Inc. 600 Series Chipset SATA Controller
        IOMMU group: 43
                LnkCap: Port #0, Speed 2.5GT/s, Width x1, ASPM L0s L1, Exit Latency L0s <4us, L1 <64us
                LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
                LnkSta: Speed 2.5GT/s, Width x1
                LnkCap2: Supported Link Speeds: 2.5GT/s, Crosslink- Retimer+ 2Retimers+ DRS-
                LnkCtl2: Target Link Speed: 2.5GT/s, EnterCompliance- SpeedDis-
                LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete- EqualizationPhase1-
15:00.0 USB controller: Advanced Micro Devices, Inc. [AMD] Device 43fd (rev 01) (prog-if 30 [XHCI])
        Subsystem: ASMedia Technology Inc. Device 1142
        IOMMU group: 44
                LnkCap: Port #0, Speed 2.5GT/s, Width x1, ASPM L0s L1, Exit Latency L0s <4us, L1 <64us
                LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
                LnkSta: Speed 2.5GT/s, Width x1
                LnkCap2: Supported Link Speeds: 2.5GT/s, Crosslink- Retimer+ 2Retimers+ DRS-
                LnkCtl2: Target Link Speed: 2.5GT/s, EnterCompliance- SpeedDis-
                LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete- EqualizationPhase1-
16:00.0 SATA controller: Advanced Micro Devices, Inc. [AMD] 600 Series Chipset SATA Controller (rev 01) (prog-if 01 [AHCI 1.0])
        Subsystem: ASMedia Technology Inc. 600 Series Chipset SATA Controller
        IOMMU group: 45
                LnkCap: Port #0, Speed 2.5GT/s, Width x1, ASPM L0s L1, Exit Latency L0s <4us, L1 <64us
                LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
                LnkSta: Speed 2.5GT/s, Width x1
                LnkCap2: Supported Link Speeds: 2.5GT/s, Crosslink- Retimer+ 2Retimers+ DRS-
                LnkCtl2: Target Link Speed: 2.5GT/s, EnterCompliance- SpeedDis-
                LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete- EqualizationPhase1-
79:00.0 USB controller: ASMedia Technology Inc. Device 2426 (rev 01) (prog-if 30 [XHCI])
        Subsystem: ASMedia Technology Inc. Device 2426
        IOMMU group: 51
                LnkCap: Port #0, Speed 2.5GT/s, Width x1, ASPM L0s L1, Exit Latency L0s <4us, L1 <64us
                LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
                LnkSta: Speed 2.5GT/s, Width x1
                LnkCap2: Supported Link Speeds: 2.5GT/s, Crosslink- Retimer+ 2Retimers+ DRS-
                LnkCtl2: Target Link Speed: 2.5GT/s, EnterCompliance- SpeedDis-
                LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete- EqualizationPhase1-
7a:00.0 USB controller: ASMedia Technology Inc. Device 2425 (rev 01) (prog-if 40 [USB4 Host Interface])
        Subsystem: ASMedia Technology Inc. Device 2425
        IOMMU group: 52
                LnkCap: Port #0, Speed 2.5GT/s, Width x1, ASPM L0s L1, Exit Latency L0s <4us, L1 <64us
                LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
                LnkSta: Speed 2.5GT/s, Width x1
                LnkCap2: Supported Link Speeds: 2.5GT/s, Crosslink- Retimer+ 2Retimers+ DRS-
                LnkCtl2: Target Link Speed: 2.5GT/s, EnterCompliance- SpeedDis-
                LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete- EqualizationPhase1-
7b:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Granite Ridge [Radeon Graphics] (rev c1) (prog-if 00 [VGA controller])
        Subsystem: Gigabyte Technology Co., Ltd Granite Ridge [Radeon Graphics]
        IOMMU group: 53
                LnkCap: Port #0, Speed 16GT/s, Width x16, ASPM L0s L1, Exit Latency L0s <64ns, L1 <1us
                LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
                LnkSta: Speed 16GT/s, Width x16
                LnkCap2: Supported Link Speeds: 2.5-16GT/s, Crosslink- Retimer+ 2Retimers+ DRS-
                LnkCtl2: Target Link Speed: 16GT/s, EnterCompliance- SpeedDis-
                LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete+ EqualizationPhase1+
                LnkCtl3: LnkEquIntrruptEn- PerformEqu-
7b:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Rembrandt Radeon High Definition Audio Controller
        Subsystem: Advanced Micro Devices, Inc. [AMD/ATI] Rembrandt Radeon High Definition Audio Controller
        IOMMU group: 54
                LnkCap: Port #0, Speed 16GT/s, Width x16, ASPM L0s L1, Exit Latency L0s <64ns, L1 <1us
                LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
                LnkSta: Speed 16GT/s, Width x16
                LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete- EqualizationPhase1-
 
I face similar problem, may i know do you solve it successfully?
My gpu passthorugh also downgrade to
lspci -vvv -s 00:01.1 | grep -i 'LnkSta'
LnkSta: Speed 2.5GT/s, Width x16
 
You (both sso and masato) told us nothing about what version of Proxmox you were running and how you configured the VM.
Additionally, nothing about what "Linux" you were booting (ancient kernel version?).

Should we get the good old crystal ball, to get things going?
Or would you rather add details, so that the whole thing could start to make some sense?
 
It seems that recent GeForce models are designed to lower the PCI-E Gen when idle due to power-saving features.
Regarding the SATA controller and USB 3.x controller, when I contacted AMD technical support, I was told that although it appears to be connected at PCI-E Gen 2x1, internally it is connected via a proprietary interface, so there is no problem with data transfer speeds.

Proxmox VE 9.1.4
Kernel 6.17.4-2-pve
 
The RTX4060Ti was listed as Gen2x8 because the GPU was not being used in benchmarks, etc.
When the GPU is in use, it is linked at Gen4x8.
SATA and USB3.x are designed to return false information that the CPU's IOD is linked at Gen2x1, but in reality they are not connected via PCI-E, but via a proprietary interface with a bandwidth that exceeds that of SATA and USB.
(Answer from AMD technical support)
 
The same thing happened with MSI's X670E Gaming Plus WiFi, so maybe it's a Ryzen 7000 series or later?
This wasn't the case with the Ryzen 5000 series (X570).
 
Another point is that if the CPU ID is set to a Ryzen or later CPU, PCI-E information will be displayed as a question mark in GPU-Z on Windows 11 with GPU pass-through.
Even if the CPU is a Ryzen 9000 series, it will work normally if recognized as Optron, so it seems that PCI-E is handled differently in Windows 11.
 
Did their answers solve the problem?

I don't understand why you started bragging about it.
 
Last edited:
uzumo said:
Did their answers solve the problem?

I don't understand why you started bragging about it.
Click to expand...
Who are you referring to when you say "their answer"?
Do you mean that the X670E and X870E motherboards you own are not linked with USB3.x and SATA Gen2x1?
 
You must log in or register to reply here.
Top Bottom
Back