Modify existing fresh install or just reinstall to add ceph?

[SpacemanSpiff]

Hey all,

So I'm just getting into proxmox. I have some decent familiarity with NetApp and Nutanix from an old job and I was trying to setup a similar HCI type configuration in my home lab. I lucked into some nice, old enterprise hardware and I've balanced them out as best I can with some hardware I had. I couldn't make them identical though. Setup at the bottom of the post.

I currently have Proxmox installed and during the install, when it listed the drives, I gave 1 drive thinking that the others would be able to be committed to storage later but I may have been incorrect about that? The Proxmox GUI and CLI both show all the other drives connected but I'm not sure exactly how to add them to make them usable.

Additionally, trying to learn about Ceph, apparently I need an OSD for every drive? So I need like 9 OSD's per server (8 drives plus the IDSDM)? That sounds like...a lot, but then again I've never setup Ceph before. For NetApp/Nutanix, you need one controller per node, not per drive. Or maybe it was per drive but the controllers did it automatically in the background?

Regarding the networking, I only have a switch with 1G ports but I'd like to take advantage of the 10G NICs. I read online that I can just direct connect the two nodes. Unfortunately I didn't do this until after proxmox was installed. How do I add the 10G direct connect after the fact? This may be a moot point if I need to reinstall proxmox anyways.

So, bottom line, I currently have the hardware setup with proxmox installed and ceph installed inside it but not configured. I have one 1G NIC on each node connected to my network.

I would like to do the following:

1. Setup Ceph so I have shared storage between the nodes. (I assume this would present one large storage volume, like a pool or a LUN)
2. Configure the 10G NIC to be the direct interconnect between the nodes for cluster related tasks while leaving the 1G NICs connected to my network for all else.

So, with those goals in mind, do I need to reinstall proxmox entirely and change something during setup? Or can I mold what I currently have into what I'm trying to do?

Thank you for your help.

-------------------------------------

Node 1:
Dell PowerEdge R640
2x Intel(R) Xeon(R) Silver 4110 CPU @ 2.10GHz
144GB RAM
64GB IDSDM
2x 10G NIC
2x 1G NIC
(Storage in Attached Node1Storage screenshot.)

Node 2:
Dell PowerEdge R640
2x Intel(R) Xeon(R) Silver 4110 CPU @ 2.10GHz
128GB RAM
64GB IDSDM
4x10G NIC
4x 1G NIC
(Storage in Attached Node2Storage screenshot.)

 

[UdoB]

do I need to reinstall proxmox entirely and change something during setup?

No, there is no need to do that. Just follow the docs to add the ceph-repository. Then you can install the whole shebang - includings OSDs, MONs and MGRs - from the web gui. (Just for reference: the official docs are there: https://pve.proxmox.com/pve-docs/chapter-pveceph.html)

Disclaimer: I am not a Ceph user - my own adventure (in a homelab) was two years ago : https://forum.proxmox.com/threads/fabu-can-i-use-ceph-in-a-_very_-small-cluster.159671/