Rate/roast my proposed homelab Ceph network

[jefm]

I've been messing with Proxmox on and off for years in homelab using old desktops as hosts and lately added a pair of second-hand enterprise servers. Just started messing with Ceph in virtualized hosts, had a lot of fun with that and I want to try running it on metal for home & lab use.
However I want to move away from the desktop-enterprise mishmash to three identical homemade AMD AMS4/5 boxes with multiple NICs running a meshed network, with four to six OSD's per node.

In a homelab forum, I was assured this would very not work, as "three nodes is not enough for Ceph" and "the networking is too complicated".
So I ask what an actual Pxmx forum thinks, as the documentation backs three nodes as viable for a VE with Ceph, and the meshed network is not only supported but has a literal howto page.

As for networking, I originally considered a jumble of 10Gb nics to handle the priv/pub/Corosync/vm bridge traffic. But 40Gb dual nics look accessible, and would greatly simplify all that meshing.
Disks for OSD's might be a weakness as I'll likely try to utilize many enterprise SATA hard drives that I already have.

Thanks for any advice, I've been out of the hardware game for a long time but it's been fun picking it back up again.

 

[esi_y]

Of course it WILL work, even TB networking based ones work.
https://forum.proxmox.com/threads/intel-nuc-13-pro-thunderbolt-ring-network-ceph-cluster.131107/

But the question is, is it for practice or do you look for something that you do not want to have much potential overhead administering...

 

[jefm]

Of course it WILL work, even TB networking based ones work.

That's priceless, thank you.

But the question is, is it for practice or do you look for something that you do not want to have much potential overhead administering...

The goals of this racket are:

1. to set up a PVE where I can make & break Vm's (containers etc) for computer & network studies
2. have enough compute power to spin up fake networks, DR simulations, etc
3. maintain storage and VM's for self-hosting cool services and things
4. learn how to administrate, back up and care for all of the above, in the chance that i get sysadmin work someday

"I'm jef and thanks for attending my TED talk"

 

[esi_y]

The goals of this racket are:

1. to set up a PVE where I can make & break Vm's (containers etc) for computer & network studies
2. have enough compute power to spin up fake networks, DR simulations, etc
3. maintain storage and VM's for self-hosting cool services and things
4. learn how to administrate, back up and care for all of the above, in the chance that i get sysadmin work someday

Then go ahead by all means! Keep backups of everything that's not for "studies" only at the least.

"I'm jef and thanks for attending my TED talk"

You're welcome, you should post a walk-through for others here once you are done.

 

[smueller]

That's priceless, thank you.


The goals of this racket are:

1. to set up a PVE where I can make & break Vm's (containers etc) for computer & network studies
2. have enough compute power to spin up fake networks, DR simulations, etc
3. maintain storage and VM's for self-hosting cool services and things
4. learn how to administrate, back up and care for all of the above, in the chance that i get sysadmin work someday

"I'm jef and thanks for attending my TED talk"

Hi Jef,
I'm Samuel form Thomas-Krenn.AG (Solutions Engineer for Datacenter Solutions)

I would love to help you if you have any issues or questions going along. If you want you can message me at smueller@thomas-krenn.com or via Discord. (Please request my Discord Tag via e-mail)

 

[bomzh]

Are you going to use SATA SSD or regular spinning rust? I think 10Gbit is more than enough for spinning rust.

 

[jefm]

you should post a walk-through for others here once you are done.

Sure, I would love to put up runbooks & configs somewhere, minus public IP addresses and that kind of thing, and/or maybe a build-journal thread if people on this forum are into that.

 

[jefm]

Are you going to use SATA SSD or regular spinning rust?

I might be able to add NVME or maybe SATA SSD for caching, but for the time being the heavy lifting will be on ol' CMR.
Much learning about latency, iops, and swapping OSDs around is foreseen.

I think 10Gbit is more than enough for spinning rust.

I'm inclined to agree especially with the caliber of drives that I have around. Although I did get some used 40Gbit stock for much cheaper then I thought it'd be and almost comparable with 10gb stuff I was looking at.