Suggestions/Feedback on Proxmox Cluster with Ceph

[framberries]

Howdy,

For context, I'm just barely diving into the world of using Ceph with Proxmox. I've previously had proxmox up and running for years on a single node, working fine the way it was. However, I recently acquired 2 more Intel NUC devices (my first proxmox node is an older NUC), and I'm interested in creating a proxmox cluster. My goals are to allow for high availability/migration of VMs and containers. How long this takes to rebuild in the case of a catastrophic failure, because this is a 'production' home lab, is not terribly important, but I'd prefer to keep data up to date if a device needs to migrate on such an event. An example, I host a few game servers, and while losing data is acceptable (say given up to a day) it's unideal to say the least.

With the goals and desires outlined above, I'm attempting to do some research into getting Ceph working in a proxmox cluster, but haven't seen my scenario outlined as a solution/usable setup yet. So if barking up the wrong tree or that's outlined somewhere, please link it so I can read it! Hardware is 3x Intel NUCs, each with a 2.5Gb ethernet adaptor, a single SSD for booting proxmox, and an NVME for VM/container storage. If I'm understanding Ceph correctly, it'd be the preferred method for storage since it'd be more up to date data wise than using a ZFS drive on each intel nuc with the same name, and replication configured. Everything I've read seems to indicate using a remote ceph cluster/storage on proxmox, or creating 12 OSDs (multiple drives per device?) across 3 devices. I haven't seen anything outlining using Ceph with a single NVME per node, as the storage. Am I on the right track, is it doable, or am I crazy?

Any feedback would be greatly appreciated! Thanks for reading my novel

 

[mfed]

Rest assured that it is entirely possible to run a Proxmox cluster on 3 nodes with a single NVMe drive. I actually run a similar configuration for several years. Note that you would basically end up with the ceph storage equivalent to the size of your single NVME drive, as the best practice is to have data stored on 3 drives. The ceph setup is very straightforward from the GUI, and it will allow you to use the storage on all nodes of the cluster.

 

[framberries]

Hey mfed,

Thanks for the confirmation and explanation! I'm going to try and bring up the nodes and see what I can do on my own. Did you use any particular resources (articles, youtube videos, etc) for getting you going on that same setup? I'd love to see a guide if at all possible

 

[mfed]

I built my cluster 6 years ago, so don't remember which resources I used at the time... I built a cluster in a nested lab several times, and it was very straightforward. If you have your single node environment I encourage you to test first in a virtual lab, you just need the three VMs with 3GB of memory, and two 16GB disks each (one for the install, and one for ceph).
After you install the proxmox on all of them, you join them to a proxmox cluster, and then use the GUI to create a ceph cluster - you would need to create a ceph monitor on each node, and then an OSD on each node, and then create a pool. GUI should do everything for you, and I think the defaults should give you a working configuration.

 

[framberries]

Awesome, thanks for your input @mfed! I will give it a whirl.