Advice; squeezing usefulness out of old boxes

[floodping]

I was allocated a stack of decommissioned HP servers that only ever saw extremely light usage. Fun! But they belong to a non-profit, and my budget for this side-project is precisely $0. Almost all are single-chip, four-core Xeons. They do not have SSDs nor hardware RAID. I would still like to put them to a good use of some sort.

I'm new to Proxmox, but I'm inclined to try it here. I'm also new to ZFS, but wanted to try it here to get RAID1 given there's no PVE software-raid nor, in my case, hardware raid.

Each box has:
2 × 4 TB spinning rust HDDs
1 × 32 GB on-mlb 'enterprise' microSD card, many spares
variously 12 to 20 GB of ECC RAM

Now I'm just bursting with bad ideas. Approaches I've considered, nested:

1. Default PVE install on microSD. Scheduled, audited backups; for inevitable card death. /swap choices:
   
   1. Low host,vm swapiness. ZFS gets the HDs entirely.
      
   2. Zero swap. Expect crashes. ZFS gets the HDs entirely.
      
   3. Move swap (and logs?) inside HD ZFS.
   4. Two partitions on each HD: 1 small for swap,etc and 1 large for partition-backed ZFS.
   5. Single partition on each HD, use file-backed ZFS.
   6. Don't use ZFS. Use drives whichever way best suits the VMs.
2. Ignore the motherboard's microSD card slot. As for the spinning drives:
   
   1. ZFS the spinning drives. Install Proxmox over Debian.
   2. Don't use ZFS at all.
3. Don't use Proxmox nor ZFS. Implement whatever uses I come up with manually on bare metal.

I know swap is a nuanced subject, but this isn't enterprise production. Loads will be very low, so I'd think "actually" running out of memory should be rare.

I'm partial to "idea 1.1". From the little I've seen, PVE installs so quickly and easily that it is almost expendable. There are no project requirements, but even if I come up with some great services, unexpected VM downtime will be a non-issue, as long as it isn't protracted. From the docs, restoring from backups looks utterly straightforward; Am I missing any gotchas? Is restoration secretly harder after a PVE reinstall or on a different node?
Is my sense of ZFS requirements way off?
Did I leave the best approach off the idea list completely?

Again, any guidance welcome. Cheers!

 

[Andrew Hart]

If it really is a "stack", I'd go for a proxmox ceph hyperconverged cluster. Allocate 1 hdd disk from each as your proxmox and 1 each as an osd for ceph. Assuming you have a spare gig switch that you can dedicate to ceph.
If you need high performance vm put it on lvm on the local hdd, if you need high available put on the ceph storage.

 

[LnxBil]

What type of machines are these? You could build better machines by combining them, more RAM, more CPUs more Disks ...

Best use would IMHO be use ZFS over all disks with PVE installed on top (I do not find this in your decision try).

 

[floodping]

Thanks for the replies!

I should have mentioned that it's unlikely I'll get to keep them together geographically. If future meetings on their needs go the way I anticipate, the boxes will end up in split into groups of one to six nodes, on sites with external bandwidth too poor for any sort of spanning technology.

Andrew Hart: I've never personally had a chance to even consider ceph, so I've been reading up since your post. Thanks for the idea; it looks amazing, now that I have enough nodes to warrant it. I'll hope for new projects which will allow them to stay together.

LnxBil: Most are these weird half-depth, 1U, HP Proliant gen8 DL-series boxes. They only have two 3.5 drive bays so, while the nodes with 12 GB RAM are painful, I'd lose a lot of storage capacity by dumping nodes just to steal their ram. As for your recommended "ZFS over all disks with PVE installed on top"... would that still be your suggestion given just two spinning drives, and no clustering?


Looking back at it, my decision tree was just me trying
1: to figure out what to do with swap if I only have microSD cards and ZFS, both of which are specifically recommended against.
2: to ascertain if booting PVE from the microSD card is as bad as it sounds in my head. I think the temptation is from remembering that back in the day I'd boot ESXi from [non-ssd] flash drives and routinely see a full day go by without a single write to the boot media. I'm assuming PVE has no configuration or provision for low-write/embedded hardware constraints. Oh, and I'm tempted to put PVE on the SD cards because of how shockingly easy it was to install it.

Any additional thoughts or recommendations appreciated. Thanks!

 

[alexskysilk]

I would still like to put them to a good use of some sort.

I'd say good use would be target practice. More seriously, why would you invest any time or effort into these? without a specific problem to solve its impossible to answer your questions.

 

[LnxBil]

I'm assuming PVE has no configuration or provision for low-write/embedded hardware constraints.

Unfortunately no.

LnxBil: Most are these weird half-depth, 1U, HP Proliant gen8 DL-series boxes. They only have two 3.5 drive bays so, while the nodes with 12 GB RAM are painful, I'd lose a lot of storage capacity by dumping nodes just to steal their ram. As for your recommended "ZFS over all disks with PVE installed on top"... would that still be your suggestion given just two spinning drives, and no clustering?

Ceph without SSDs and only two disks does not provide any fun (nor performance), really!

 

[Andrew Hart]

Unfortunately no.

Ceph without SSDs and only two disks does not provide any fun (nor performance), really!

There are two disks on each of the servers. Since each of the servers will only have gig ethernet anyhow, ssd would be a waste of time.
The servers probably don't even have trim support.

 

[floodping]

LnxBil, Andrew Hart: Yep. Ceph is a bad fit for this old hardware. Thanks. No SSDs. Two 1 gig nics. Little ram. Horrible firmware/hardware support for... anything.

I will be learning more and floating some ceph tests with some more spendy orgs though. I'm gathering that ceph is flat-out designed to scale, so if we like performance (and manageability) on a 15 node trial, adding more will make things better, not worse? Anyway—thanks again for the recommendations!

❧ ❧ ❧​

I'd say good use would be target practice. More seriously, why would you invest any time or effort into these?

I'd say you should find a better target thread for your elitism. "More seriously," these are owned by a not-for-profit org where every dollar I spend is taken away from helping improve people's futures. Their original grant precludes selling them. So they're just garbage? I'll assume you typed that in a friendly, jokey manner, so I'm just poking back. But yeah—if your Enterprise Datacenter blinders get so tight you truly can't even offer suggestions anymore for small offices, volunteers, freelancers, charities, and homelabs, then why post in this thread?

I can think of dozens of uses for them. Combine them all into one big write-only, subpoena-able archiver for a central site. Or they'll want them split them up. Offline field office fileshares, backup destination, digital signage manager, lan stuff… dhcp, ntp, printserver. Normal office internal webserver, archiver, firewall, lan security, vpn, voip, directory svc, git, collabora, wiki, mattermost, pos/kiosk head, wordpress, work-order/ticketing, alternate-os user vm host, …

without a specific problem to solve its impossible to answer your questions.

The docs say to expect server blocks or high IO load if swap is on ZFS. Given low swapiness and careful pve/vm memory curation…
Which is worse; swap on an SD card, or on ZFS?
Which is worse; a swap partition on the same drive as partition-backed ZFS, or giving ZFS direct whole drives, but putting swap in it?
Which is worse; keeping swap somewhere it doesn't belong (flash media, ZFS, …) or turning it to 0?
With no HA/shared storage/clustering, is PVE still truly as easy to nuke and recover from backups as it looks?
Will a freshly-installed PVE have any trouble with a VMs from other PVE instances… any restore gotchas?
Will a freshly-installed PVE have any trouble with a ZFS pool created by a previous install or elsewhere?

LXC, KVM, and even ZFS predate ubiquitous SSDs. Current documentation is still full of mentions of how to best deploy limited-in-quantity SSDs in a mostly spinning HD environment. I'm just polling for any community experience under tough olde-timey hardware constraints.

Thanks.

 

[Andrew Hart]

Here is a PVE ceph cluster I have:

3 servers of:
CPU(s): 8
Thread(s) per core: 1
Core(s) per socket: 4
Socket(s): 2
Model name: Intel(R) Xeon(R) CPU E5430 @ 2.66GHz

ceph status
cluster:
id: cf6aec56-7ab1-41b1-90c1-972d52cb9d90
health: HEALTH_OK
services:
mon: 3 daemons,
osd: 9 osds: 9 up, 9 in
data:
pools: 1 pools, 512 pgs
objects: 90.16k objects, 116GiB
usage: 225GiB used, 7.97TiB / 8.19TiB avail
pgs: 512 active+clean

Sure it isn't fast but it is using 11 year old processors.(Released November 11, 2007) and no SSDs.

ceph osd df
ID CLASS WEIGHT REWEIGHT SIZE USE AVAIL %USE VAR PGS
0 hdd 0.90959 1.00000 931GiB 25.4GiB 906GiB 2.73 1.02 116
1 hdd 0.90959 1.00000 931GiB 25.8GiB 906GiB 2.77 1.04 118
2 hdd 0.90959 1.00000 931GiB 22.8GiB 909GiB 2.45 0.91 105
3 hdd 0.90959 1.00000 931GiB 23.7GiB 908GiB 2.55 0.95 106
4 hdd 0.90959 1.00000 931GiB 22.2GiB 909GiB 2.38 0.89 99
5 hdd 0.90959 1.00000 931GiB 28.5GiB 903GiB 3.06 1.14 132
6 hdd 0.90959 1.00000 931GiB 27.7GiB 904GiB 2.98 1.11 128
7 hdd 0.90959 1.00000 931GiB 26.2GiB 905GiB 2.81 1.05 120
8 hdd 0.90959 1.00000 931GiB 22.3GiB 909GiB 2.39 0.89 100
TOTAL 8.19TiB 225GiB 7.97TiB 2.68

 

[alexskysilk]

I'll assume you typed that in a friendly, jokey manner, so I'm just poking back.

I did

"More seriously," these are owned by a not-for-profit org where every dollar I spend is taken away from helping improve people's futures. Their original grant precludes selling them. So they're just garbage?

In a real sense, yes. A tool that has no problem to solve is just taking up room. I understand that you are trying to find utility for these but from my perspective this is backwards; I'm tasked with solving problems, not finding problems to solve with my otherwise idle tools.

I can think of dozens of uses for them. Combine them all into one big write-only, subpoena-able archiver for a central site. Or they'll want them split them up. Offline field office fileshares, backup destination, digital signage manager, lan stuff… dhcp, ntp, printserver. Normal office internal webserver, archiver, firewall, lan security, vpn, voip, directory svc, git, collabora, wiki, mattermost, pos/kiosk head, wordpress, work-order/ticketing, alternate-os user vm host, …

If you have to think of them, you've already defined your situation as without a need. Needs are self evident. You CAN use the equipment for any number of reasons, but the calculus of whether you SHOULD is completely different:
1. What is the cost of deployment, in terms of setup time, installation in the DC, cost of DC space and power, and subsequent support and maintenance; equipment capex is a factor but far from the only one or even the largest.
2. What are the users expectations for the proposed solution? will it have sufficient resources to meet its requirements? Will it perform according to expectation? will it have the anticipated uptime?
3. What are the allowed downtime parameters? if the system is down for an hour, a day, a week? who's responsible to respond, and how much will that coverage cost?

I understand the non profit aspect of this, but users are people. People have expectations; people seldom remember what something cost but complain endlessly about how it sucks.

But onto the practical.

Which is worse; swap on an SD card, or on ZFS?
Which is worse; a swap partition on the same drive as partition-backed ZFS, or giving ZFS direct whole drives, but putting swap in it?

neither. never put swap on SD card; you can put it on ZFS but if you're trying to run proxmox on a ram constrained system you shouldnt use ZFS in the first place. If you must, you can use a zvol, but my advice is buy a super cheap SSD and use it instead.

With no HA/shared storage/clustering, is PVE still truly as easy to nuke and recover from backups as it looks?

Yeah, its pretty simple. vzdump (the proxmox backup mechanism) keeps all metadata with the backup so once restored it will have all details. If you intend to restore proxmox to bare metal just keep a copy of your interfaces file; the rest is fairly transient.

Will a freshly-installed PVE have any trouble with a VMs from other PVE instances… any restore gotchas?

as long as the bridges are named the same and are attached to the same network it should just work.

Will a freshly-installed PVE have any trouble with a ZFS pool created by a previous install or elsewhere?

Usually no. The only issues that may arise are due to cache policy; turning it off on the restored vm will usually fix it (although sometimes its the other way depending on the source dataset type.)

LXC, KVM, and even ZFS predate ubiquitous SSDs. Current documentation is still full of mentions of how to best deploy limited-in-quantity SSDs in a mostly spinning HD environment. I'm just polling for any community experience under tough olde-timey hardware constraints.

Oh for sure; the analogy that may serve here is if you happened to have a horse and buggy its possible to get around town with it but it wouldnt make sense to. It all WORKS but the overall cost makes the proposition moot.

 

[floodping]

Thanks again, all. So yeah; old hardware is old. At least now that I have a better sense what I'm looking at I'll be able to "give it to them straight" when their lofty goals come up. As for keeping some lit up, I actually like the horse and buggy picture... in today's world, at most they might find a fun, quaint, novelty sort of second life, and nobody can complain when the ride smells like manure.