CPU power throttle back to save energy

[lpallard]

Hello fellow PVE users. I am wondering if there is a way to throttle back CPU speed or set CPU power states to save energy in PVE?

My PVE server has the following hardware config:

Supermicro H8DCL-iF (dual socket C32 motherboard)
2x Opteron 4334 6 core CPU
4x8GB DDR3-1600 ECC RAM
4x Hitachi SAS 15k600 hard drives
1x IBM M5016 SAS RAID controller + 24mm 5000RPM fan
1x Intel Quad Gigabit controller + 24mm 5000RPM fan
8x 80mm fan (throttled to speed approx 1800RPM to keep temps acceptable)

Power usage is around 230W in average.

1. Based on your experience, would you deem 230W acceptable for such hardware config?
2. Is there any way to scale CPU frequency back when not in use or use power states such as C states with Intel CPU's? In Freenas, you can create a "tunable" parameter and assign it a variable for freenas to scale a CPU core or apply a C-state to it. Can you do this in PVE?

Server load is around 0.8 (according to the graph on PVE's webGUI) and CPU load is around 15-18 (I assume percents?) so I would not say this is a "loaded" or busy server. Sure there are peaks because I have 10 VM's so workload is well distributed with occasional peaks.


Thanks!

 

[wolfgang]

Hi,

To 1 Yes I would say this is normal for this System.

To 2 No this brings stability problems, so it is not recommended.

 

[Stewge]

The Opteron CPUs are serious power hogs (comparatively).

To be honest if the power consumption is a problem for you, a single new quad-core Xeon (ie 1230v5) could probably outperform that setup on less than half the power.

 

[lpallard]

The Opteron CPUs are serious power hogs (comparatively).

To be honest if the power consumption is a problem for you, a single new quad-core Xeon (ie 1230v5) could probably outperform that setup on less than half the power.

Thanks Wolfgang for replying and specifying this is not good practice. Weird because on the freenas forums, they are all for it... I guess between virtualization and storage things are significantly different.

For now I will leave the PVE server as-is. Its been running 24/7 for 385 days now without a glitch whatsoever..

Yes I agree the Opterons are power hogs compared to Xeon's and are less powerful (processing wise that is). That being said, when I built this server the financials indicated it would be cheaper to opt for Opterons instead of Xeons. 3 years later, if I had to do it all over again, I would have to examine this closely.

 

[Bent]

For future time travelers:

The issue is that Proxmox sets the cpu govenor to 'performance'. Which of course offers you the best performance and avoids "micro lags" caused by the CPU having to scale up dynamically but this is also not the ideal setting for saving energy.

My personal recommendation is to set the CPU governor to 'powersave' which is absolutely fine for newer Intel CPUs (anything from the past 5 years). This will scale the CPU frequency down whenever possible and increase it dynamically on load so there is almost no loss in performance. Intels p_state driver does the demand based scaling and does a really good job at it.

tl;dr:

To achieve this without any additional application you can run the following:

echo "powersave" | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor

I recommend adding this to a cronjob '@reboot' if you want this setting to be permanent across reboots.

To monitor the current CPU frequency:

cat /sys/devices/system/cpu/cpu*/cpufreq/scaling_cur_freq

If you want more infos about intel_pstate check 'pyamsoft/pstate-frequency' on GitHub (can't post links sorry).

This can in many cases save a lot of power. In my case it saves around 40-50W per node (when idle / light load).*

Especially at home or in larger deployments this can offer serious cost savings at little to no loss in performance.

* to me this has not shown any signs of stability issues or any other unwanted effects. The powersave govenor is also the govenor most distributions use by default.

 

[AlexLup]

Man, I was seriously contemplating sleep & Wake-On-LAN solutions for a while, so glad I found this!

 

[mdbraber]

@Bent thanks for this tip!

 

[sdet00]

For future time travelers:

The issue is that Proxmox sets the cpu govenor to 'performance'. Which of course offers you the best performance and avoids "micro lags" caused by the CPU having to scale up dynamically but this is also not the ideal setting for saving energy.

My personal recommendation is to set the CPU governor to 'powersave' which is absolutely fine for newer Intel CPUs (anything from the past 5 years). This will scale the CPU frequency down whenever possible and increase it dynamically on load so there is almost no loss in performance. Intels p_state driver does the demand based scaling and does a really good job at it.

tl;dr:

To achieve this without any additional application you can run the following:

echo "powersave" | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor

I recommend adding this to a cronjob '@reboot' if you want this setting to be permanent across reboots.

(snip)

I've performed this on a handful of systems, and so far I have mixed results:

1. On both AMD systems that I tried this on ( AMD Turion X2 TL-50 and AMD Turion II N54L) CPU frequencies were stuck at 800Mhz and performance was pretty bad. I've only tried this on Intel systems that are running 3rd Generation Pentium or better, so I can't comment too much about older CPUs.
2. Every time I reboot my server, I get an email that just says "powersave". To avoid this, I use this in my Crontab instead:

echo "powersave" | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor >/dev/null 2>&1

I also use "S-TUI" as a command to monitor my clock frequencies and temps, as it gives a bit more info and allows you to also stress the system and watch clock speeds change.

 

[sukerman]

PVE 6.1

My Ryzen 5 3600 does not scale when set to powersave stays at 2Ghz, but is does scale when set to conservative or on demand.

Use this to find out available governors.

cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_available_governors

e.g : conservative ondemand userspace powersave performance schedutil

Ondemand and Conservative appear to be the same, but:

>"The ondemand (available since 2.6.10) and conservative (since 2.6.12)
> are governors based on in kernel implementations of CPU scaling
> algorithms: they scale the CPU frequencies according to the needs
> (like does the userspace frequency scaling daemons, but in kernel).
> They differs in the way they scale up and down. The ondemand governor
> switches to the highest frequency immediately when there is load,
> while the conservative governor increases frequency step by step.
> Likewise they behave the other way round for stepping down frequency
> when the CPU is idle. The conservative governor is good for battery
> powered environments on AMD64 (but may not work on older ThinkPads
> like the T21). Ondemand may not work on older laptops without Enhanced
> SpeedStep due to latency reasons. Anyway, for recent enough Intel CPU,
> ondemand is the one recommended for power efficiency (over userspace,
> and even over "powersave") by the Intel's kernel developer Arjan van
> de Ven"

So for me I set :

echo "ondemand" | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor

You can watch if your CPU frequencies are scaling in another window:

watch cat /sys/devices/system/cpu/cpu*/cpufreq/scaling_cur_freq

sysbench wasn't available in the repo so I used stress-ng to make it scale.

apt install stress-ng
stress-ng --cpu 8 --cpu-method matrixprod --metrics-brief --perf -t 60

HOWEVER .... this setting does not survive a reboot, so add to cron:

@reboot echo "ondemand" | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor

ALSO....

The terminal screen didn't power off so I added this to turn off after a minute

@reboot setterm -blank 1

 

[AndrwHmmr]

So just to give this "debate" , some data:
My Ryzen 5 3600 (idle)
Powersave ~38 Watt
On Demand ~39 Watt
Performance ~40 Watt

You can see the effects on frequency in my prometheus graphs, even sitting at nearly 4GhZ all the time it does not consume more than 1 W more than on demand... So there is really no need from my point of view. Thanks AMD!

Ryzen 5 3600
128GB DDR4
X470D4U2-2T
2 SSDs Mirror
be Quiet! Straight Power 450 W Gold PSU

EDIT: So I do realise that this may very well be because of PSU efficiency stuff at low loads...

 

[AndrwHmmr]

So maybe there is not much difference in power draw from the wall, but!
I noticed a definite improvement in terms of thermals with my Ryzen 3600 with the on demand govenor... Will keep this now.

The graphs show that the fans do not ramp up that often for the time period the on demand govenor is active...
Nice. So now the PSU is burning all the Watts the CPU doesnt use, but it is quieter. A big win for me. (The temp of my server cabinet plays a role too of course but...)

 

[HeyRob]

I also have a Ryzen 5 3600X. Been running in my NAS at between 97-101 watts load (power data provided by TP-Link smart outlet) depending on usage. After setting my CPU cores to ondemand using sukerman’s suggestion, it instantly dropped to between 81-85 watts consumed at the outlet. I would say that is significant heat and consumption reduction. Thank you sukerman!

 

[michaelvv]

So just to give this "debate" , some more data:
My Ryzen 5 3600 (idle)
On Demand ~62 Watt

There is not a real difference using powersave or ondemand.

Hardware.

Ryzen 5 3600
MSI B550 Tomahawk mainboard
MSI GT 710 1GD3H LP 1GB DDR3
2*16 Gb Corsair LPX DDR-2666 memory
1 Tb ADATA SX-8200 Pro Nvmi
1 6Tb WD RED Harddisk
PSU Corsair CX650M

And just to confirm my previous Intel Xeon-E-2146G CPU on a ASUS-WS-C246-PRO board
With same hardware , was using 45-47 Watt a clear advantage to Intel here...

When stressing the cpu running stress -c 12 , the power usage rises to 130 watt.

 

[HeyRob]

Agreed, I didn’t notice much difference between powersave or ondemand throttling. Clearly a beneficial difference between either of those two and un-throttled though.

Hardware:
Purpose:
NAS (OMV VM with disk-passthru) + HomeLab VM’s

Ryzen5 3600X
Asrock X570M PRO4 Micro ATX
2*32GB HyperX Predator DDR4
3*6TB WD Red’s (MergerFS/SnapRAID for redundancy)
2*2TB RAID1 Seagate 5400’s
1 1TB SSD for VM OS’s
1 256 GB SSD for ProxMox/boot drive
Corsair RMX 850 watt Gold PSU
Fractal Design Node 804 case

 

[kylezhang]

To change powersave governor on boot:

sudo apt install cpufrequtils lm-sensors

Then edit /etc/default/cpufrequtils, add line:
GOVERNOR="powersave"

Also can create a script to change governor when we want. Create and edit /usr/local/bin/cpufreq-set-all . Add the following script
#!/bin/bash
MAX_CPU=$((`nproc --all` - 1))
for i in $(seq 0 $MAX_CPU); do
echo "Changing CPU $i with parameter $@";
cpufreq-set -c $i $@ ;
done

Then add execute permission:
chmod +x /usr/local/bin/cpufreq-set-all

Then we can switch governor by the following command:
cpufreq-set-all -g powersave
cpufreq-set-all -g performance