installation of 2 servers for school. Solid advice welcome

[huuub]

So instead of ordering 1 expensive Dell/HP server I decided to buy 2 identical more consumer grade servers. Purpose is that if one fails the other takes over.
Main components:
MSI MPG X670E CARBON WIFI
AMD Ryzen 9 7950X3D Processor (32 threads)
192 gig DDR5 RAM
Crucial MX500 250GB SSD (boot)
2 * Lexar NM790 4TB NVMe M.2 SSD (data)
Seagate HDD NAS 3.5" 12TB ST12000VN0008 Ironwolf (Backups)
2 * 10 Gbit/s + 1 * 2.5 Gbit/s

Requirements :
It doenst need to be High available, If there is every 4 to 12 hours a backup/snapshot that is ok. Also if a server/VM fails it is ok that I can switch it manually.
So what I was thinking of is making a ZFS pool of the 2 NVMe drives on both servers. And a replication job that runs every 4 hours to the backup HD and to the other Server. There are no critical databases or something. If something happens I manually switch them. I have a 3th server if needed for quorum but I could also give one server one vote more I think because I do the switch manually anyway?

The questions:
I could go for Ceph but then I need someone who could confirm this would work good and reliable?
Is there a reliable way to make it completely automated with only 2 servers (not a real requirement but hand)?
I dont want to go for NFS since that brings in one point of failure.

Since this is the first time I am working with a cluster I can use some good advice or best practices.
Thanks and have a happy end of the year.

 

[leesteken]

A cluster of only two nodes is asking for problems; use at least a QDevice. Ceph really needs at least three nodes. Using cheap consumer SSDs will also give you headaches. Using an AMD CPU with 3D-cache (for only half the cores) does not make sense for servers. Would you not be better off with second-hand real/enterprise servers?

 

[Dunuin]

So what I was thinking of is making a ZFS pool of the 2 NVMe drives on both servers.

Don't buy consumer SSDs for ZFS.

I could go for Ceph but then I need someone who could confirm this would work good and reliable?

Not with only 2 nodes.

Is there a reliable way to make it completely automated with only 2 servers (not a real requirement but hand)?

A cluster needs at least 3 machines for quorum (2n +1). One could be a cheap qdevice.

 

[huuub]

A cluster of only two nodes is asking for problems; use at least a QDevice. Ceph really needs at least three nodes. Using cheap consumer SSDs will also give you headaches. Using an AMD CPU with 3D-cache (for only half the cores) does not make sense for servers. Would you not be better off with second-hand real/enterprise servers?

I can get the QDevice so indeed I will do that and indeed I will not use Ceph as I already thought.
The SSD is a concern. But the price also. Do you have alternatives for the storage (Which do not cost 700 euros for 4 Tb?).
These are not really intensive storage requirements.

 

[Dunuin]

The SSD is a concern. But the price also. Do you have alternatives for the storage (Which do not cost 700 euros for 4 Tb?).
These are not really intensive storage requirements.

SSDs without a power-loss protection (so every consumer/prosumer SSD) will really suck when doing any sync IO. You get like 100-1000ths of the performance and the SSD will wear very quickly. So especially bad when running stuff like DBs that do a lot of sync IO. But ZFS on its own got horrible overhead so you really want a durable SSD.
And enterprise SSDs aren'T that expensive anymore. A "Samsung PM9A3 3.84TB M.2" costs 449€. Look at the price per TB TBW and not at the price per TB capacity when buying SSDs. Then you will see cheap looking SSDs aren't actually cheap on the long run!

 

[huuub]

I will make the change to the "Samsung PM9A3 3.84TB M.2". And also will change to the regular one and not the 3D version of the CPU.
Any other changes you recommend?

 

[Dunuin]

ECC RAM instead of non-ECC in case the CPU + BIOS support it. But I think with DDR5 it isn't as easy anymore as with DDR4 ECC UDIMMs? Wasn't there a different keying so you can't simply swap ECC for non-ECC anymore?

But like already said, if you are on a limited budget and you want something powerful and reliable with a great bang for the buck (especially if you don't have to care that much about electricity costs and noise) second hand servers might be your best choice.

 

[leesteken]

The SSD is a concern. But the price also. Do you have alternatives for the storage (Which do not cost 700 euros for 4 Tb?).

Don't buy new/latest gaming hardware, which is not optimal for your use case anyway, and spend the money on ECC memory (traditionally not supported on MSI) and SSD's with PLP (I use Micron 7450) instead.

 

[Dunuin]

PS: M.2 isn't common for enterprise SSDs as it is a terrible formfactor for everything that isn't mobile hardware. There are only 3 option and all require the long 22110 slots if you want NVMe and 4TB: Samsung PM9A3, the older PM983 and the Micron 7450 Pro.
7450 Pro and PM9A3 more or less cost the same.

Have a look at your local refurbished server resellers. Maybe you find 3x identical ~5 year old way more powerful servers you could upgrade with some enterprise NVMe SSDs + 40Gbit dualport NICs for a proper ceph cluster and a less powerful forth server to set up a PBS with Enterprise SSDs for the same price.

 

[huuub]

PS: M.2 isn't common for enterprise SSDs as it is a terrible formfactor for everything that isn't mobile hardware. There are only 3 option and all require the long 22110 slots if you want NVMe and 4TB: Samsung PM9A3, the older PM983 and the Micron 7450 Pro.
7450 Pro and PM9A3 more or less cost the same.

Have a look at your local refurbished server resellers. Maybe you find 3x 5 year old way more powerful servers you could upgrade with some enterprise NVMe SSDs + 40Gbit NICs for a proper ceph cluster and a less powerful forth server to set up a PBS with Enterprise SSDs for the same price.

Hmm maybe you are right but well I went for the ECC and the PM9A3 changes. Very usefull.
It wont be with the 40Gbit NIC nor 3 servers and Ceph but I dont think we get 64 threads in total and almost 400 gigs of Ram with other options for a small 7000 euros. Off course all has it pros and contras (It wont be HA but we can live with the fact that everything is down for a few hours). I chose the Asrock X670E Taichi for support of ECC ram and the Kingston Fury Renegade KF560C32RSK2-96 6000 ECC.

 

[Dunuin]

I dont think we get 64 threads in total and almost 400 gigs of Ram with other options for a small 7000 euros.

Have a look at ebay auctions. Shouldn't cost more than 1000-2000€ with older hardware. Resellers providing warranty of cause want a bit more.
I've seen complete working 32 thread xeon servers with 256GB RAM for as low as 450€. Buy a bunch of these and you got a nice cluster. And they aren't maxed out. For a few hundred euros you could upgrade them to 64 threads and 512GB RAM. But at that price point they are quite old and not that reliable anymore (like dual socket E5 v2 Xeons). Probably better to pay a bit more and get something that is 5 years old instead of 10+ years as the single threaded performance won't be great.

 

[bbgeek17]

It doenst need to be High available, If there is every 4 to 12 hours a backup/snapshot that is ok. Also if a server/VM fails it is ok that I can switch it manually.
So what I was thinking of is making a ZFS pool of the 2 NVMe drives on both servers. And a replication job that runs every 4 hours to the backup HD and to the other Server.

If you want to use PVE interface to manage replication, I believe, the nodes need to be in the same cluster. So you still need 2+1 setup.

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox