low energy HW suggestions

[crc-error-79]

Hello all,

in order to reduce the power consumptions, I want to move my two proxmox systems into one single pc.

But I am not sure which hardware pick to achive my goal.

The current "cluster" is composed by:

1
- m/b asrock itx with dual lan
- cpu i7 7700,
- ram 16 gb ddr4
- main storage 2x480 (mirror zfs)
- secondary storage 2x1000 ssd (mirror zfs) - (only testing and vm's local backups)
- 2x10GB sfp+ intel x520 nic
average power consumption of the system 32-40W with 2vm and 4 lxc

2 -
- m/b asrock with single lan
- cpu i7 7700
- 32 gb dd4
- main storage 2x1000 ssd (mirror zfs)
- secondary storage 1x1000 ssd - (local backups and read cache for truenas - not optimal I know but I had no sata connector available for a dedicated solution)
- vm truenas 2x3000 hdd (3.5" - mirror zfs),
- vm truenas 1x2000 hdd (2.5" - frigate recordings)
- 1x10GB sfp+ nic card
average power consumption of the system 40-45W with 3vm and 4 lxc

It is an home lab, so nothing critical and with daily backup (local and on a synology nas), so to me it is not mendatory have ecc memory, redundancy, etc.

Available (my ex proxmox server), I have a Ryzen 5900 + 64 gb ddr4 ecc but the cpu + 2ssd with nothing running than proxmox (and governor in powersave) it consume in idle 70-80W.

Since I need quicksync for hw transcoding and at least 6 cores, the idea is to use a i5 12400 on a matx board + 64gb + dual nic 10g + a lsi controller for the disks.
Or mabye an used xeon but which one?
Could a system like this consume less than my 2 i7s?

On ebays there are a lot of used xeons but most of them have no gpu (so no quicksync) and the only info I found is the TPM with means nothing so I don't know the real power consumptions..

Do you guys have any suggestions?

 

[LnxBil]

Do you guys have any suggestions?

Not a perfect fit for your requirements, but a ORDROID H3+ is a very good device with very low power consumption (11W) with 64-GB RAM. It has only 2 SATA, 1 NVMe, 4C/8T and 2x2.5 GBit, so not exatcly up to your specs yet a noteworthy mention.

 

[crc-error-79]

Thank you, I have already consider it. as well as all the mini pc: fujitsu, dell, mac mini intel or passively cooled itx pc (with aluminium case and multiple nics) but all of them lacks on expansion, I can't add a 10gbe nic or an extra sata/sas controller for hdd storage..
I saw that there are a lot of Supermicro soc with Intel Atom but I don't know how powerful this cpu is, if it can run a windows vm or not

 

[LnxBil]

I saw that there are a lot of Supermicro soc with Intel Atom but I don't know how powerful this cpu is, if it can run a windows vm or not

Yeah, those were too expensive for my taste. The problem is really to have server components with low wattage. I tried a single 10 GBE card (at least the ones. I have), which took 10-30W if used, so having them will double or quadrupe my wattage of the mini itx machine I used at that time. There were two other Atom-based machines in use for over decade with onboard 5xSATA.

I just threw the 10 GBE out. 2x2,5 GBit is enough if I invest in a 2.5 GBit switch, currently GBit is enough for me. Yes 10 GBE would be better, but the increaseed wattage is just not enough gain for the power it draws.

 

[crc-error-79]

Yeah, those were too expensive for my taste. The problem is really to have server components with low wattage. I tried a single 10 GBE card (at least the ones. I have), which took 10-30W if used, so having them will double or quadrupe my wattage of the mini itx machine I used at that time. There were two other Atom-based machines in use for over decade with onboard 5xSATA.

I just threw the 10 GBE out. 2x2,5 GBit is enough if I invest in a 2.5 GBit switch, currently GBit is enough for me. Yes 10 GBE would be better, but the increaseed wattage is just not enough gain for the power it draws.

My card consumes less than a spinning hdd, without it the power consumption goes down 5-6W in average, sure I have to say that I am not using the whole bandwidth...
It is only used during backups or when I move the data from my pc or nas, but it remain in that range.
Maybe mine consume less because I am using the SFP+ with dac instead of the base-t.. I don't know..


What do you think about the i5 12400 + microatx board & other stff. Do you think that it will consume less then 60-70W?

 

[LnxBil]

Maybe mine consume less because I am using the SFP+ with dac instead of the base-t.. I don't know..

me too.

What do you think about the i5 12400 + microatx board & other stff. Do you think that it will consume less then 60-70W?

I have no idea. This heavily depends on the used hardware on the board. I had a Xeon board on which the CPU used not much, but all the other chips were hot to the touch and used a lot. Have you tried looking for power benchmarks online?

 

[Dunuin]

It is an home lab, so nothing critical and with daily backup (local and on a synology nas), so to me it is not mendatory have ecc memory, redundancy, etc.

If you ever had a bad RAM module you won't want non-ECC again. Here 3 DIMMs failed, one after another and that corrupted hundred of GBs of data on my TrueNAS over a few weeks. If you don't keep your backups for months, you are screwed, as your backups will also contain the corrupted data. Would have been so much easier with ECC RAM and EDAC monitoring. Then I could have seen it early corrupting data instead of ousing it for weeks without noticing anything.

a lsi controller for the disks.

Here the LSI card adds ~10W when idling.
The problem is really all the devices you add. Here, the biggest home server, without any disks/PCI cards, consumes 35W when idling. Add PCIe cards and disks and it is idling at 112W. Start some idling guests and it goes up to 145W.
But still better than using 2 servers...at least from a power consumption perspective. I combined two of my PVE servers earlier this year. The remaining server now consumes an additional 40W but I don't need to run the second server anymore which was using ~80W. So 40W saved.

I saw that there are a lot of Supermicro soc with Intel Atom but I don't know how powerful this cpu is, if it can run a windows vm or not

I wouldn't get an Atom CPU again. Tried an Atom J3710 CPU (Consumer grade, 6.5W TDP, quadcore, 2.64 GHz) and it was totally struggling to run ZFS, to back up to a PBS or to run more than a few linux VMs. It can't even handle suricata IPS of OPNsense when no other guests are running.
After some months I replaced it with a small old Xeon board (E3-1230v3 on a X10SLL-F). The J3710, with running guests, was consuming 17-18W (10W without guests). Now with the Xeon and the same workload it's more like an average of 25W-33W (~20W without guests). But at least it now got ECC RAM and services won't become unresponsive under load.
Also, keep all your IT infrastructure in mind. Here, when also counting the switches, routers, APs, UPS and so on this adds another 35W to the idle consumption. With that in mind, the additional 8-15W of the Xeon server isn't bad. Total idle consumtion might climb from 52W to 62W. But at least it also got 4 times the performance when needed.
I will probably replace that later with some Intel i3/i5 with iGPU + ECC support on mATX, unless Minisforum or whatever starts designing efficient MiniPCs that at least can handle ECC RAM + 2x 2,5" slots for a ZFS mirror.

There is a quite old German list of low idle power hardware:
https://docs.google.com/spreadsheets/d/1LHvT2fRp7I6Hf18LcSzsNnjp10VI-odvwZpQZKv_NCI/edit#gid=0
Maybe something similar but more recent exists?

But to really save electricity it is best to only let your servers run when really needed. So for example a small power-efficient host, as much stripped down as possible, without all the nice-to-haves
Iike 10Gbit NICs or proper HBA cards and with everything disabled in BIOS that isn't absolutely needed, for all the guests that need to run 24/7. And an additional bigger host for all the other VMs/LXCs that could be started on demand so it doesn't need to run overnight when everyone is sleeping.
It's by far not as convenient as running everything 24/7, but really saves money...here it saves ~500€ per year...probably way more if you also take saved SSD/HDD wear into account.

 

[crc-error-79]

I have no idea. This heavily depends on the used hardware on the board. I had a Xeon board on which the CPU used not much, but all the other chips were hot to the touch and used a lot. Have you tried looking for power benchmarks online?

CPU usage is very low, the vm as well as the lxc stay in idle most of the time, the one that uses more cpu is (highest value I saw was below 60%) is the lxc with jellyfin during transcoding with quicksync or the pfsense with vpn on.
On the others the cpu is always below 5%.
The power hungry vms (a fedora I use to check manuals/videos when I work and a Windows 10 with tia portal [siemens plc development tool] ) are nearly always off.

Could I ask you which xeon you have and how is the consumption?

If you ever had a bad RAM module you won't want non-ECC again. Here 3 DIMMs failed, one after another and that corrupted hundred of GBs of data on my TrueNAS over a few weeks. If you don't keep your backups for months, you are screwed, as your backups will also contain the corrupted data. Would have been so much easier with ECC RAM and EDAC monitoring. Then I could have seen it early corrupting data instead of ousing it for weeks without noticing anything.

Here the LSI card adds ~10W when idling.
The problem is really all the devices you add. Here, the biggest home server, without any disks/PCI cards, consumes 35W when idling. Add PCIe cards and disks and it is idling at 112W. Start some idling guests and it goes up to 145W.
But still better than using 2 servers...at least from a power consumption perspective. I combined two of my PVE servers earlier this year. The remaining server now consumes additional 40W but I don't need to run the second server anymore which was using ~80W. So 40W saved.

That is right I wasn't consider that the LSI is very energy hungry..
I could consider to change the hdd/ssd I have with ssd with bigger capacity, so instead of 4x1TB I could use 2x2TB or 2x4TB..

In years I have never had an issue with bad ram, maybe I was lucky but I understand your point.
In basement I found an old synlogy 916 and an HP microserver (I think it is an amd turion) both with 4 hdd bays and ecc, I could power them and see the consumption, if is low I can move the truenas from a vm to the baremetal or use the synology.
So I can move the others vm on the first i7 and remove the second

Thank you very much for the feedback (and the link, I will check it out).
25W-33W is very good, I will jump on ebay to see if there is some good price. Even if the downside is the 32gb of max ram and no gpu onboard so no transcoding..

The consumption of the other IT equipments are around 60-70W (UPS + 2 ubiquiti rack switches, 10 cameras + 3x APs + other small switches all in poe from the 24 pro switch).
One of the secondary goal I want achive is to increase the surveillance system uptime (frigate on a vm) during the blackouts.
Every Watt removed from the home-lab, are minutes added to the UPS.
But to do this I don't want to completely cut the system performances, it should be balanced

 

[Dunuin]

25W-33W is very good, I will jump on ebay to see if there is some good price. Even if the downside is the 32gb of max ram and no gpu onboard so no transcoding..

I really would get something newer. That thing is now over 10 years old, so not very power efficient by today's standards with its 22nm. And if you don't care that much about single-threaded performance I would get a low-power version of a Xeon CPU.
Would be interesting to know what something like a Supermicro X12STH-F + Xeon E-2374G or Xeon E-2378G is consuming when idling.

 

[crc-error-79]

I really would get something newer. That thing is now over 10 years old, so not very power efficient by today's standards with its 22nm. And if you don't care that much about single-threaded performance I would get a low-power version of a Xeon CPU.
Would be interesting to know what something like a Supermicro X12STH-F + Xeon E-2374G or Xeon E-2378G is consuming when idling.

Thank you, I am total newbie about xeon (and it's chipsets) world and since these stuff are almost out of mainstream (website, youtube, etc) it is difficult to really understand how well or bad they work.
I will check it
Do you think that these new cpu will have consume similar to yours?

 

[crc-error-79]

I am looking for an used xeon, just one question, could them vendor looked?
Example if I buy an used dell or lenovo workstation, can I use the processor on another motherboard?

 

[Dunuin]

Thank you, I am total newbie about xeon (and it's chipsets) world and since these stuff are almost out of mainstream (website, youtube, etc) it is difficult to really understand how well or bad they work.

I will check it

Do you think that these new cpu will have consume similar to yours?

There are usually two types of Xeon CPUs. The big chips with a lot of PCIe lanes and cores specially design for servers. And the small CPUs that are based on the consumer i3/i5/i7/i9 CPUs...just with enterprise stuff like ECC support not locked. From what I have seen so far the idle consumption of these smaller CPUs is better. Simpler hardware = lower idle power consumption.

 

[crc-error-79]

There are usually two types of Xeon CPUs. The big chips with a lot of PCIe lanes and cores specially design for servers. And the small CPUs that are based on the consumer i3/i5/i7/i9 CPUs...just with enterprise stuff like ECC support not locked. From what I have seen so far the idle consumption of these smaller CPUs is better. Simpler hardware = lower idle power consumption.

Thank you, have you saw my previous post?

On ebay I found an old (5 years old) Lenovo with a xeon E with integrated gpu here in Europe, and it is cheaper than the combo cpu + motherboard shipped from China or Us.
My guess is there is some lock on the chip like dell do on the amd, am I right? Or there are no vendor lock?

 

[pengu1n]

It's a shame you're not in England. I have a few powered off that could go to a loving home. Older Xeons with ECC ram, HBAs, etc. One is a 4U storage server that can hold 24 hard drives. All powered off due to cost of electricity. Same as you want to do.
Anyway, older xeons are not vendor locked.

 

[crc-error-79]

It's a shame you're not in England. I have a few powered off that could go to a loving home. Older Xeons with ECC ram, HBAs, etc. One is a 4U storage server that can hold 24 hard drives. All powered off due to cost of electricity. Same as you want to do.
Anyway, older xeons are not vendor locked.

the 24 bays server is a little too much for me. eheheh
In Italy is the same, the electricity cost has doubled in few months..
That is why I am trying to "shrink" the homelab but found the right combination between performance & energy used is a very hard work..