Ceph node without OSDs?

proxwolfe

Well-Known Member
Jun 20, 2020
487
45
48
49
Hi,

I have a three node home lab PVE cluster. Each node is also a CEPH node and has two OSDs (one being assigned to a "fast" pool for apps and one being assigned to a "slow" pool for data) (I know that's fewer than recommended and I am contemplating adding more OSDs but this is a home lab...) and I have a 3/2 rule. So I understand that a copy of each bit should reside on each node.

Now I am thinking about adding a fourth PVE node to the PVE cluster which, however, would not be up 24/7 but only for special compute tasks. While I would like to have the VMs that should run on this node in the CEPH app pool and the data it needs in the CEPH data pool, I probably want to avoid putting in OSDs (because otherwise there is no guarantee that a copy of every bit will continue to sit on each of the original three nodes - right?).

Can the fourth node be a CEPH node anyway (i.e. without OSDs)? If this works, I am aware that my network would become a bottle neck when loading apps (data probably not so much, as they are coming from slow disks anyway). But can I do that or is this going to create some unexpected issues down the road?

Bonus questions: A 4/2 rule would probably accomplish the same (but require more OSDs), right? But if I turn on the fourth node only every once in a while, syncing all the missing stuff would probably congest my network...

Any thoughts?

Thanks!
 
Last edited:
Yes you can have ceph nodes without osds without any issues, as long as you have quorum (MON-Service) an Corosync-Quorum.

For SIZE 4 and MIN-SIZE 2 you need 4 Nodes, and every node needs at least a osd (or more). All the nodes need to be online, otherwise PGs will never turn green.
 
Last edited:
Yes, you can add nodes with no OSD but be carefull for NIC configuration to avoid bottlenek. My tip is to configure NIC for node with no OSDs using the same NIC configuration of the other nodes with OSD
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!