[SOLVED] Electricity bill doubled...how to redesign homelab?

[Dunuin]

Hi,

Just got the next electricity bill raise. Started my PVE server with 0.21€ per KWh, now its at 0.266€ per KWh and next month and onwards it will be 0.462€ per KWh. So 262€ per month for a single person household....checked other electricity providers and seen prices there from 0.40€ up to 0.95€ per KWh...
Right now my homelab (1x PVE node, 2x TrueNAS, 4x router, 4x switch) uses roughly 220W average with spikes to 300W under load. So that would be 74 to 101€ per month just for the homelab.
I really don't like it, but looks like I'm forced to shutdown my homelab until the war is over and prices will drop again. And all my 3x 4U servers depend on each other, so I would need to shutdown all of them or let them all running 24/7.

Server 1 is my Backup NAS (bare metal TrueNAS Core) that can be ignored as it already only running a few minutes/hours per week to receive the ZFS replication.
Hardware:
Case: 1x Supermicro Case
Mainboard: 1x Supermicro X10SLL-F
CPU: 1x Intel Xeon E3-1230v3 (4 core, 3.3-3.7 GHz)
RAM: 2x 8GB DDR3 ECC
Boot-Pool: 2x 120GB SSDs as ZFS mirror
HDD-Pool: 4x 8TB HDDs as raidz1 (WD White)
PSU: BeQuiet 300W
Heatsink: 92mm tower cooler (Arctic)

Server 2 is my Main NAS (bare metal TrueNAS Core) which stores all my files for all other clients and is required by most VMs. It also runs some important guests for HA/failover like DNS server (Pihole), router (OPNsense) and PBS.
Hardware:
Case: 1x Inter-Tech 4U-4088-S
Mainboard: 1x Supermicro X10SLL-F
CPU: 1x Intel Xeon E3-1230v3 (4 core, 3.3-3.7 GHz)
RAM: 4x 8GB DDR3 ECC
Boot-Pool: 2x 120GB SSDs as ZFS mirror
SSD-Pool: 5x 400GB SSDs as raidz1 (5x Intel S3710 400GB)
HDD-Pool: 4x 8TB HDDs as raidz1 (WD White) + 2x 200GB SSDs (Intel S3700) as mirrored special device
Unused Pool (planned for bare metal PVE when virtualizing TrueNAS): 2x 200GB SSD (Intel S3710)
NIC (storage backend): Mellanox Connect-X3 (MCX311A-XCAT)
HBA: 2x PERC H310 (flashed to LSI 9211-8i IT-Mode)
PSU: BeQuiet 300W
Heatsink: 92mm tower cooler (be Quiet)

Server 3 is my PVE node which is running 6 LXCs and 21 VMs.
Hardware:
Case: 1x Inter-Tech 4U-4088-S
Mainboard: 1x Supermicro X10SRM-F
CPU: 1x Intel Xeon E5-2683 v4 (16 cores, 2.1-3 GHz)
RAM: 4x 16GB DDR4 2400 ECC (Samsung M393A2K40BB1-CRC)
Boot-Pool: 2x 100GB SSDs as mdraid mirror (Intel S3700 100GB)
VM-Pool1: 5x 200GB SSDs as raidz1 (5x Intel S3710)
VM-Pool2: 1x 200GB SSD as LVM-thin (Intel S3710)
HDD-Pool: 2x 3TB HDDs as ZFS mirror (Seagate ST3000DM001)
GPU: Geforce GT710
NIC (storage backend): Mellanox Connect-X3 (MCX311A-XCAT)
NIC (frontend): Intel i350-T4
PSU: Supermicro 900W 80+ gold
Heatsink: 92mm tower cooler (SNK-P0050AP4)

Other Homelab parts:
Switches:
1x HP Aruba Instant On 1930 (JL682A)
3x D-Link DGS-1008D (Ver. I1)
Routers:
2x D-Link DIR-860L (Ver. B1 mit OpenWRT 19.07)
1x Fritzbox 7560
1x TP-Link WDR4900 (Ver. 1.3 mit OpenWRT)
UPS:
1x Bluewalker Powerwalker VI 800 STL

So I would need to shutdown all 3 server for a year or so but there are some services I really rely on. This would be:
- Nextcloud VM (3GB RAM + 1 vCPUs) because it syncs my password safe between devices...would really be screwed if that isn't working for a longer time...but it also stores data on the NAS which also would be shutdown
- Pihole LXC (384MB RAM + 1 vCPU) for DNS so nternet will work
- OPNsense VM (4GB RAM + 4 vCPUs) because otherwise there would be no internet or VLAN routing at home
- reverse proxy VM (384MB RAM + 1 vCPU) because services like Nextcloud rely on it
- Docker VM (8GB RAM + 4 vCPUs) because its running paperless-ngx which is needed to view/store my papers/letters (also stores stuff on NAS)
- DokuWiki LXC (512MB RAM + 1 vCPU) to document stuff and read notes (also stores data on NAS)

...so I don't see how to solve this in a good way.

Option 1:
Shutdown all 3 servers, get some energy efficient cheap box with 16 or better 32GB ECC RAM and at least slots for 2x 2.5" SSDs so It could run the 6 VMs/LXCs mentioned above from the mirrored SSDs. And I also would have to set it up as a NAS and reconfigure those guests and my workstation, laptop and smartphone to store the home folders, nextcloud data, dokuwiki data, digitalized letters on these SSDs. This would be a lot of work and kind of pointless because I would have to revert everything before using the big server again. Later when switching back to the big server I could use this small node as spare node in case the main PVE node would be failing to get atleast the most important guests running.
This would save the most electricity but I also can't access most of my data or services without booting up all the big servers again every time I want to access something.
Anyone can recommend an affordable secondhand computer model that would fit and works fine with PVE?

Option 2:
Only shutdown the PVE node and continue using the Main NAS server. Right now that TrueNAS server is using maxed out RAM (32GB) and an old quadcore Xeon (E3-1230v3) for 4x 8TB HDDs + 5x 400GB SSDs + 5x 200GB SSD + 2x 120GB SSDs on two LSI 2008 HBAs. My plans for the future were to virtualize TrueNAS and also run PVE bare metal on it but the hardware is too low end for that. The 32GB RAM and CPU are just enough to run ZFS, PBS and my secondary Pihole and OPNsense VMs. So for it to be able to run PVE + virtualized TrueNAS I would need to upgrade it first to something with at least 64GB RAM and more cores.
My PVE node got 64GB RAM + 16 core CPU (Xeon E5-2683 v4 but also got a spare 8 core E5-2620v4) which would be fine for that. So I could disassemble my PVE node and use the Mainboard + RAM + CPU for it. But then I would loose the option to temporarily boot up the PVE node in case I need to access one of my other 21 guests. I'm already fighting with running out of RAM on that PVE node with its fully populated RAM slot and I got PCIe cards I can't use because I only got 3 PCIe slots and all are full and 6 or 7 would be nice to have. In the future I planned to upgrade that PVE node anyway to something like a Supermicro X10SRL-F with 128GB RAM. But that isn't cheap either and it's stupid to buy "new" hardware that I want to be shutdown 99% of the time because of saving electricity.

Option 3:
Disassemble the PVE node + Main NAS and get something completely new that can work as my NAS and PVE node in a single server. I already thought about that but failed to find a silent (homelab is in the living room) PSU that can power 10x 3,5" HDDs + 20x 2.5" SSDs. With that I would also loose the ability to run my OPNsense and Pihole HA and nothing would be working in case that single server got a problem. I still would keep the backup NAS, but that server is only powered on once per week so not really helpful for failover. And powerfull but energy efficient enterprise grade hardware is probably too expensive. I already paid like 3500 to 4000€ for the homelab and nearly all hardware (except for the managed switch and HDDs) was second hand I got cheap.


How would you cut down or remodel my homelab to safe electricity?
Do you also got such an insane bill raise? How do you handle this?
Do you think second hand server hardware wil become cheaper now as people and hosting companies will be forced to replace the old hardware because its not reasonable anymore to let it running?

 

[guletz]

Hi @Dunuin ,

Nice homelab, is more like a Data Center...

Very hard decision. In the worst case scenario, I would keep all servers up, but I would power VMs, only when is need. I would buy a cheap 2 x raspberry pi where I would run only PiHole. Also I would by a decent hw router an used instead of OPNsense.

And I guess you do not need to run your homeLab, 24h/24. I think that during the night you do not work at your homeData-Center!

You could also change your CPU guvernor power from performance to some variant who is electricity friendly.

I also run a homeLab who is power in the morning (1 Proxmox node), and once/week I power the second node for replicate my primary node.

But during the night power is down.

During the day, I boot the OwnCould and other VMs only I need it.

Good luck / Bafta.

 

[LnxBil]

Very good points from guletz. I also switched to a simple Atom-based machine with two enterprise SSDs running on 9W idle.

But during the night power is down.

During the day, I boot the OwnCould and other VMs only I need it.

You may also use a photovoltaik system on your balcony. Here in Germany, we have a huge boom with so called balcony power plants, a 600W out-plug-ready system to reduce your electrical bill over the day.

 

[guletz]

I also switched to a simple Atom-based

Hi @LnxBil

Could you be kind and share your atom configuration?

Thx .

Here in Germany, we have a huge boom with so called balcony power plants, a 600W out-plug-ready system to reduce your electrical bill over the day.

Also could you give more information about a such sistem?

Good luck/Bafta !

 

[LnxBil]

Could you be kind and share your atom configuration?

Sure:

• ASRock J4205, Mini-ITX, passively cooled
• 2x 8GB-RAM DDR3 SO-DIMM (maximum possible)
• PicoPSU 90W (smallest available)
• 2x Samsung PM863 240 GB, ZFS mirror, additional cooling sinks
• everything in an open case, mounted at my wall.

Smallest and cheapest system I could find with 16GB-RAM. Any "enterprisy" hardware costs A LOT more (high 3/low 4-digit) only for the mainboard (e.g. Supermicro X9). My system runs for about 4 years now. Power usage checked with an outlet power measure device, 9W idle, up to 15W for a loaded system. The purpose is all network infrastructure, a couple of Docker containers, different vpn gateways, filer with most commonly used files and of course PVE.

Everything else is running on server hardware, that is behind switchable plugs and only gets started up if I need it.

Until I bought the Mini-ITX system, I used an old IBM T61 laptop with 8 GB-RAM and two SSHD which needed about 20W. It was nice to haven an integrated USV (also called battery).

Also could you give more information about a such sistem?

e.g. Hoymiles HM-600 is the highest available wattage system for the German regulation about Balkonkraftwerke (balcony power plant). It's a micro-inverter / module-mounted inverter with two independent module inputs of up to max. 370W DC per panel/input and it converts to upmost 600W AC. It has an "open" circuit cable at which you can plug any 240V plug (e.g. the german Schutzkontaktstecker, Wieland or Beretti plug). The cost of such a system (microinverter, 2x 370W panels, cables) is currently about 800 Euros and the demand is huge (therefore the high prices). It's main goal is to have it on your balcony without any installation at your root (although you still can), but it is often hung from your balcony or setup directly on your balcony on the ground. This is very tenant-friendly and german court has ruled that it can always be used (on the balcony) if there is no "physical harm" to the building in any way. Of course Germany, you have to register it with your energy provider and with the government.

The idea behind it is that you can provide your basic load and more so that you have to buy less expensive eletricity from your energy provider.
Under common conditions (1000kWh/kWPeak/year, average Germany yield), you will generate 600 kWh of electricity per year.