Proxmox 6.2.16-19-pv kernel brakes lower cpu package states

[etnicor]

Did a fresh install of proxmox and my i3 13100 cpu can reach C6 when on 6.2.16-3-pve kernel.

However when I upgrade kernel from 6.2.16-3-pve to 6.2.16-19-pv I can no longer reach c6 cpu package state. I just get to c3 now.

Tried the 6.5.3-1-pv kernel aswell and have the same problem there.

Anyone have any idea on what has changed or what could be going on?

Edit: Tested all 8.x versions, so something happened with kernel after 6.2.16-5-pve. I guess 5 to 19 is quite a big jump.
6.2.16-3-pve(8.0.2) cpu reach c6
6.2.16-4-pve(8.0.3) cpu reach c6
6.2.16-5-pve(8.0.4) cpu reach c6
6.2.16-19-pve(8.0.5) cpu fail to reach c6


Tested using latest powertop version compiled from source to verify cpu package C-state and s-tui to monitor package power consumption.

 

[etnicor]

@t.lamprecht Moving discussion from 6.5 kernel back here. Finding is that cpu package state get limited to C3 in kernel proxmox-kernel-6.2.16-16-pve.

Running this on consumer hardware, but I believe lower c-state is relevant for power savings professionally aswell nowdays.

CPU: Intel i3-13100
Motherboard: Asus PRIME B760M-A D4
nvme1: samsung 970 evo plus
nvme2: samsung 990 pro

ASPM enabled in BIOS and obviously works or c6 state would not be reached on "older" kernels.


Maybe relevant

00:06.0 PCI bridge: Intel Corporation 12th Gen Core Processor PCI Express x4 Controller #0 (rev 05) (prog-if 00 [Normal decode])
                LnkCap: Port #5, Speed 16GT/s, Width x4, ASPM L1, Exit Latency L1 <16us
                LnkCtl: ASPM L1 Enabled; RCB 64 bytes, Disabled- CommClk+
00:1a.0 PCI bridge: Intel Corporation Device 7a48 (rev 11) (prog-if 00 [Normal decode])
                LnkCap: Port #25, Speed 16GT/s, Width x4, ASPM L1, Exit Latency L1 <64us
                LnkCtl: ASPM L1 Enabled; RCB 64 bytes, Disabled- CommClk+
00:1c.0 PCI bridge: Intel Corporation Device 7a38 (rev 11) (prog-if 00 [Normal decode])
                LnkCap: Port #1, Speed 8GT/s, Width x1, ASPM L0s L1, Exit Latency L0s <1us, L1 <4us
                LnkCtl: ASPM L0s L1 Enabled; RCB 64 bytes, Disabled- CommClk-
00:1c.2 PCI bridge: Intel Corporation Device 7a3a (rev 11) (prog-if 00 [Normal decode])
                LnkCap: Port #3, Speed 8GT/s, Width x1, ASPM L1, Exit Latency L1 <64us
                LnkCtl: ASPM L1 Enabled; RCB 64 bytes, Disabled- CommClk+
00:1d.0 PCI bridge: Intel Corporation Device 7a36 (rev 11) (prog-if 00 [Normal decode])
                LnkCap: Port #15, Speed 16GT/s, Width x1, ASPM L0s L1, Exit Latency L0s <1us, L1 <4us
                LnkCtl: ASPM L0s L1 Enabled; RCB 64 bytes, Disabled- CommClk-
01:00.0 Non-Volatile memory controller: Samsung Electronics Co Ltd Device a80c (prog-if 02 [NVM Express])
                LnkCap: Port #0, Speed 16GT/s, Width x4, ASPM L1, Exit Latency L1 <64us
                LnkCtl: ASPM L1 Enabled; RCB 64 bytes, Disabled- CommClk+
02:00.0 Non-Volatile memory controller: Samsung Electronics Co Ltd NVMe SSD Controller SM981/PM981/PM983 (prog-if 02 [NVM Express])
                LnkCap: Port #0, Speed 8GT/s, Width x4, ASPM L1, Exit Latency L1 <64us
                LnkCtl: ASPM L1 Enabled; RCB 64 bytes, Disabled- CommClk+
04:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8125 2.5GbE Controller (rev 05)
                LnkCap: Port #0, Speed 5GT/s, Width x1, ASPM L0s L1, Exit Latency L0s unlimited, L1 <64us
                LnkCtl: ASPM L1 Enabled; RCB 64 bytes, Disabled- CommClk+

Cheers

 

[etnicor]

Tested to remove NVME drives from system and installed proxmox on a sata ssd.
Also disabled VMD in BIOS.
Problem persisted.

So I guess patch " Enable ASPM for NVMe behind VMD (LP: #2034504)" can be ruled out.

 

[etnicor]

@t.lamprecht
I think I found the issue. Disabled the Realtek ethernet in BIOS and I now get to c6 states.

So most likely this patch is the issue, preventing cpu to go to lower sleep states.
Ethernet not stable 23.04 (RTL8168/8169) (LP: #2031537)

Can we patch out this, I haven't had any stability issues

 

[spirit]

So most likely this patch is the issue, preventing cpu to go to lower sleep states.
Ethernet not stable 23.04 (RTL8168/8169) (LP: #2031537)

Can we patch out this, I haven't had any stability issues

I think this patch is fixing all the problems reported by users with rtl8169 chipset for months (just search in the forum

 

[etnicor]

I think this patch is fixing all the problems reported by users with rtl8169 chipset for months (just search in the forum

AH I see, I have RTL8125 does it have the same issue?

 

[etnicor]

I had a Intel i226-v pcie network card in another computer.
Moved that one into this computer and I guess this solution works for now.

Would be nice if Realtek would work though. Don't have any high hopes here though.

 

[t.lamprecht]

As written over in the other thread:

I re-checked the patches and it seems this is not really an unexpected regression but rather a trade-off.
As most (all, but possibly hardware revision and/or firmware dependent) of those models have issues if Active-state power management (ASPM) is enabled, i.e., after a while they crash completely requiring a reboot to get network working again.
That is (understandably) considered quite a bit worse than not having power management, and on kernel side this the only thing that could be controlled, so it was disabled again.

Ideally the HW vendor would fix this for real, and deliver that, e.g., via a firmware update.
So we won't consider this a fixable bug for the time being, it's by design to make broken HW a bit less broken, or at least basically useable.