Proxmox Xeon CPU upgrade choice

[newpain01]

I am doing a new Proxmox server build and while I am at it, I've been thinking to also upgrade the CPU on my Dell Precision T3610 workstation. It currently runs an Intel Xeon E5-1620 v2 @ 3.70GHz which appears to be a 4 core CPU, even though Proxmox says 8 x Intel Xeon.

My use case for this machine will be Proxmox, Frigate for video surveillance, Plex, TrueNAS, Pihole, HomeAssistant and various other VMs and containers I might need.
The socket is FCLGA2011, so it looks like I have a few options for CPUs, one of which might be an Intel Xeon E5-2697 v2 @ 2.70GHz with 12 cores or an Intel Xeon E5-2667 v2 @ 3.30GHz with 8 cores. They all have the same 130W TDP, so I guess power consumption should remain the same.

I am wondering if having more cores but lower frequency will give me a noticeable performance boost for the VMs or my current 4 cores with higher frequency (3.7GhHz) should handle it well? I can pick these up used on ebay for $25 - 2667 or $45 - 2697, so it's not a huge investment, but the question is, will I notice any difference?

 

[Dunuin]

I am doing a new Proxmox server build and while I am at it, I've been thinking to also upgrade the CPU on my Dell Precision T3610 workstation. It currently runs an Intel Xeon E5-1620 v2 @ 3.70GHz which appears to be a 4 core CPU, even though Proxmox says 8 x Intel Xeon.

PVE shows threads, not cores.

My use case for this machine will be Proxmox, Frigate for video surveillance, Plex, TrueNAS, Pihole, HomeAssistant and various other VMs and containers I might need.
The socket is FCLGA2011, so it looks like I have a few options for CPUs, one of which might be an Intel Xeon E5-2697 v2 @ 2.70GHz with 12 cores or an Intel Xeon E5-2667 v2 @ 3.30GHz with 8 cores. They all have the same 130W TDP, so I guess power consumption should remain the same.

I am wondering if having more cores but lower frequency will give me a noticeable performance boost for the VMs or my current 4 cores with higher frequency (3.7GhHz) should handle it well? I can pick these up used on ebay for $25 - 2667 or $45 - 2697, so it's not a huge investment, but the question is, will I notice any difference?

That really depends on the workload. Some stuff (like game servers, routers, ...) will need single threaded performance, where its better to have fewer but higher clocked cores. Usually guests will feel snappier with higher clocks. But when running a lot of guests or workloads that can make good use of parallelization (video decoding/encoding in software, compression, ...), then it is good to have more cores, even when they are lower clocked. I probably would get one with a lot of cores, as from what you told us the most load should come from TrueNAS, Frigate and Plex whoch should all make good use of parallelization.

 

[Neobin]

For a rough performance overview:
https://www.cpubenchmark.net/compar...ntel-Xeon-E5-2667-v2-vs-Intel-Xeon-E5-2697-v2

The E5-2667 v2 would be a no-brainer compared to your current one, because of the same single core performance, but almost double the multi core performance. Of course: 8C/16T vs. 4C/8T

The E5-2697 v2 would have ~14% more multi core performance, but ~11% less single core performance compared to the E5-2667 v2. 12C/24T vs. 8C/16T

I would definitely upgrade, but to be honest, I am not really sure with which one of the both.

PS.: Do not forget to also plan with an appropriate/good amount of RAM.

 

[newpain01]

Yeah, I definitely feel I can get a significant performance boost for not a lot of money, so that's why I am looking to upgrade, but again, I couldn't decide which one to go with, so I thought I should ask here for some opinions. I don't plan on running any game servers, but might run Pfsense and experiment with different network stuff as I learn more. It's mostly TrueNAS (probably 25+TB in ZFS RAID 6), home surveillance, automation, plex, audio streaming and learning stuff related to Docker, containers and various aspects of software development.

@Neobin I already have 96GB 1600MHz ECC RAM on this system, that should be enough for whatever I might run. I will be getting some enterprise grade SSDs for running this system as discussed in another thread I created earlier and will be getting 6-8 6TB hard drives hopefully on black friday if they are on sale (probably going with Seagate Exon 6TB (ST6000NM0115) since those are enterprise grade and CMR).

I wonder if the stock cooler from the E5-1620 is enough for any of these CPUs?

 

[Neobin]

A compromise could be the E5-2690 v2:
https://www.cpubenchmark.net/compar...ntel-Xeon-E5-2690-v2-vs-Intel-Xeon-E5-2697-v2

I wonder if the stock cooler from the E5-1620 is enough for any of these CPUs?

Since all have the same TDP, I would say yes.
Only make sure to give both a good clean job and a nice and fresh thermal paste application.

 

[apoc]

Usually i am a big advocate for high clock speed per core.
In your case (single socket) I'd prefer more cores / threads as I am convinced that this reduces the scheduling overhead of the hypervisor. My finding also was that for home and lab use na individual clockspeed per core also is not so relevant. I have downgraded my 16C 2,3GHZ to a 16C 1.8 GHZ processor to safe energy. Not seen noticeable differences for my usecase

 

[newpain01]

I have downgraded my 16C 2,3GHZ to a 16C 1.8 GHZ processor to safe energy. Not seen noticeable differences for my usecase

Interesting! I would expect some slight performance decrease since the clock speed is significantly lower. How is the difference in power consumption?

Since all of the CPUs listed above are same TDP, I don't expect a big difference in energy consumption, even though I would also like to save energy since I will be running this 24/7.

 

[apoc]

Interesting! I would expect some slight performance decrease since the clock speed is significantly lower.

That is exactly my point. Of course it is noticeable on paper. We talk about 500mhz per core less. That is 20+ %.
However on my usage profile it does not make a difference. I am running 18 VMs with 1-4 vCPUs, mainly Linux but one Windows guest. On that one i can notice a difference when applying software updates. Everything else just behaves as before. So in the end, especially with single socket you might be better of with more cores.

How is the difference in power consumption?

About 35W less. Server pulled 160w out the socket. Now has 125-130w. That was the main reason for me, as it runs 24/7. It makes a difference of about 200 kWh per year - and that alone was worth the price. I got this processor for around 35 EUR - roi in 2 years easily

 

[newpain01]

Nice! That's a big difference there in power consumption. My machine currently pulls 60W at the socket, but I don't have any hard drives yet (still building the system). Planning to add 6-8 drives, so I think it will most likely double that consumption.

I think the E5-2697 v2 @ 2.70GHz with 12 cores is the highest that my board can support, 8 cores more than what I currently have, should make a noticeable performance difference even though they are running at 1000mhz less. Maybe no difference in power though, but that's ok.

 

[Dunuin]

You can also manually limit the MHz of your CPU cores.

Get available frequencies:
cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_available_frequencies

Set CPU cores upper clock limit to 2.1 GHz:
for file in /sys/devices/system/cpu/cpu*/cpufreq/scaling_max_freq; do echo "2100000" > $file; done
You can also put it with a "@reboot" line in crontab so it gets automatically set after boot.

Used that to save some power, because otherwise my Xeon E5 2683v4 would always boost up to 3Ghz. Now with schedutil governor its always varies between min clock and 2.1 GHz.
So no big problem if you got no low power xeon.

You can change the governor with echo "schedutil" | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor. Otherwise PVE might always clock at the highest possible frequency when using the default "performance" CPU governor.

 

[apoc]

Planning to add 6-8 drives, so I think it will most likely double that consumption.

Depends on the drivetype but can easily double.
Especially 3,5" HDDs with 7200 rpm can pull significant power.

That is the reason why I have decided to use 2,5" mobile HDDs (using WD Black since 6 years pretty successfully) - of course their capacity is limited in comparison to their larger brothers but they also need significant less power.

 

[Dunuin]

Depends on the drivetype but can easily double.
Especially 3,5" HDDs with 7200 rpm can pull significant power.

That is the reason why I have decided to use 2,5" mobile HDDs (using WD Black since 6 years pretty successfully) - of course their capacity is limited in comparison to their larger brothers but they also need significant less power.

Not sure if that really saves power. At least not if you want more than a few TBs of storage. Biggest WD black with CMR is 1TB and consumes 1.8W. If you want 22TB thats 22x 1TB 2,5" HDD (=39,6W) or 1x 22TB 3,5" WD Red Pro HDD hat just needs 6.8W. So watt per TB ratio is way better when using the big disks.

 

[newpain01]

Exactly, higher capacity and lower number of drives is the way to go, but I wouldn't go under 4 drives as I want to use RAID 6.

 

[apoc]

Biggest WD black with CMR is 1TB and consumes 1.8W. If you want 22TB thats 22x 1TB 2,5" HDD (=39,6W) or 1x 22TB 3,5" WD Red Pro HDD hat just needs 6.8W. So wat per TB ratio is way better when using the big disks.

Come on
you never would use just one HDD - you know that.

Also capacity is not everything - the IO and potential recovery speeds, density (space in the case) might also be important. so one 22TB drive is not exactly a good example.
As said "capacity is limited" - just wanted to provide a hint.
What is working for me is not necessarily working for others.

 

[newpain01]

cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_available_frequencies

The `scaling_available_frequencies` file doesn't exist on my system.

Set CPU cores upper clock limit to 2.1 GHz:
for file in /sys/devices/system/cpu/cpu*/cpufreq/scaling_max_freq; do echo "2100000" > $file; done
You can also put it with a "@reboot" line in crontab so it gets automatically set after boot.

Nice, thanks!

You can change the governor with echo "schedutil" | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor. Otherwise PVE might always clock at the highest possible frequency when using the default "performance" CPU governor.

I've been using

bash -c "$(wget -qLO - https://github.com/tteck/Proxmox/raw/main/misc/scaling-governor.sh)"

to update the scaling_governor to powersaver, but it looks like it resets to performance on reboot?

 

[Dunuin]

Come on
you never would use just one HDD - you know that.

Thats true
But even 4x 4TB 3,5" disks would consume less electricity than 16x 2.5" 1TB disks. But sure, better performance per watt you will get with 2.5" disks. But if I would want performance + low power + high density I personally would get SSDs instead

I've been using

bash -c "$(wget -qLO - https://github.com/tteck/Proxmox/raw/main/misc/scaling-governor.sh)"

to update the scaling_governor to powersaver, but it looks like it resets to performance on reboot?

Have a look at the script your are executing: "NOTE: Settings return to default after reboot"
So yes, you have to run it after each reboot.

 

[RolandK]

check your real cpu load/demand by looking at the statistic graphs and consider buying some darn cheap E5-2650L v2 (with 70W TDP, you get those for 12 euros here in germany) and better stop wasting unnecessary amounts of energy

need for cpu power is often highly overrated and prices for energy are climbing up. furthermore, we have a climate problem to solve....

 

[newpain01]

check your real cpu load/demand by looking at the statistic graphs and consider buying some darn cheap E5-2650L v2 (with 70W TDP, you get those for 12 euros here in germany) and better stop wasting unnecessary amounts of energy

Thanks for the suggestion, that looks like a good option with a good amount of cores and low TDP, almost half of my current CPU. Price on ebay in my area is just $8, so that's a plus too. I'm am all about saving energy.

we have a climate problem to solve....

Hey, I'm doing my part. I have a 7.3KW solar system on my roof and export extra electricity 3/4 of a year. If high capacity batteries were priced lower, I would run all my server stuff off of that during the night as well.

 

[apoc]

But if I would want performance + low power + high density I personally would get SSDs instead

True and again this depends in the workload/usage. In my case due to the high amount of writes (backups, ZFS) I doubt that the SSDs will stand up to the job for a long time.
My HDDs got replaced after a runtime of up to 9years (remember, home usage) which is also something to be considered if we are talking about sustainability (which we ultimately talk about).
IIRC a SSD need to survive at least 4 years to catch up the energy deficit it has to an HDD on production. At the speed ZFS eats my SSD pool (MLC, Mirror) I doubt SSDs will survive this time.

E5-2650L v2 (with 70W TDP, you get those for 12 euros here in germany)

I have totally overlooked that model while checking for alternatives.
I'd choose that one.

Hey, I'm doing my part. I have a 7.3KW solar system on my roof

I'd love to do that, not possible with our current situation (rent) :/

 

[newpain01]

I am wondering what would be the difference in power consumption between the E5-2650L and the E5-2697 (but set the CPU cores to 1.7GHz as @Dunuin mentioned above with scaling_max_freq, to match the frequency of the E5-2650L)