CPU doesn't reach C8 C-state in Proxmox (it does on Debian Bookworm)

[douteiful]

Hi everyone,

I'm using an Intel I226 NIC card, my CPU reaches C8 power saving state just fine under Debian Bookworm and Arch Linux, but not on Proxmox 8.4.5, even with Bookworm's kernel.

Working setups tested:
- Debian Bookworm with kernel 6.1.0-37-amd64: Reaches C8 fine
- Arch Linux with kernel 6.14: Reaches C8 fine

Non-working setups tested:
- Proxmox 8.4 with Debian's kernel 6.1.0-37-amd64: Only reaches C3
- Proxmox 8.4 with PVE kernel 6.8: Only reaches C3
- Proxmox 8.4 with PVE kernel 6.14: Only reaches C3

As you can see it doesn't seem to have to do with the kernel but with Proxmox. If I remove the card it reaches C10. If I connect the card, it reaches C8 on Debian/Arch, but only C3 on Proxmox. So Proxmox seems to be doing something different regarding power states on the Intel card? If it's not the kernel, what could it be?

I'd be thankful if you could help me troubleshoot the issue.

 

[douteiful]

Update: I've noticed that this problem also arises with Debian Bookworm when it's installed on disk. It only works well when running from the USB drive as a live ISO. This is really weird, but at least it's not related to Proxmox. I don't think it's the NVMe because when removing the Intel card it reaches C10. I'll take it elsewhere but it'd be still be nice if anyone has an idea.

 

[Impact]

I'll just repost what I shared to someone else somewhere else when they asked about saving power. Maybe you can gain something from it.

On the software side you can try things like echo powersupersave > /sys/module/pcie_aspm/parameters/policy.
Check clear; lspci -vv | awk '/ASPM/{print $0}' RS= | grep --color -P '(^[a-z0-9:.]+|ASPM .*abled)' for ASPM state.
The first command caused ASPM to be enabled for me but didn't really change much.
Some devices can stop your CPU from getting into deep C state. Check with powertop. Talking about powertop. I also run this script at bootup.
99% of that is just in case I connect a keyboard to it.

#!/bin/bash
powertop --auto-tune > /dev/null 2>&1


if [ ! -d "/sys/bus/usb/drivers/usbhid" ]; then
        echo "No HID device exists?"
        exit 0;
fi

HIDDEVICES=$(ls /sys/bus/usb/drivers/usbhid | grep -oE '^[0-9]+-[0-9\.]+' | sort -u)

for i in $HIDDEVICES; do
        MANUFACTURER=$(cat /sys/bus/usb/devices/$i/manufacturer)
        PRODUCT=$(cat /sys/bus/usb/devices/$i/product)
        IDVENDOR=$(cat /sys/bus/usb/devices/$i/idProduct)
        IDPRODUCT=$(cat /sys/bus/usb/devices/$i/idVendor)
        BUSNUM=$(cat /sys/bus/usb/devices/$i/busnum)
        DEVNUM=$(cat /sys/bus/usb/devices/$i/devnum)
        echo  "Enabling \"$PRODUCT\" ($IDPRODUCT:$IDVENDOR) (Bus 00$BUSNUM Dev 00$DEVNUM) from manufacturer \"$MANUFACTURER\""
        echo 'on' > /sys/bus/usb/devices/$i/power/control
done

On the hardware side you can check for things like PPC Adjustment, C-state, ASPM, Aggressive device sleep, Power Supply Idle Control, ERP and so on in the UEFI. Also see here: https://youtu.be/MucGkPUMjNo?si=uIomzFXZEJQRqVT7&t=562