Question - Run PVE Ceph & Non-PVE Ceph within the same cluster?

Jan 17, 2025
18
11
3
Netherlands
Hi there,

We're planning on migrating one of our Ceph platforms to new Ceph versions, this currently runs 'regular' Ceph from the official download()ceph()com repository on Ubuntu 22.04.
Now we'd like to change this out for Proxmox VE Ceph, primarily because of the extra bug-fixes & cherry picks that the regular Ceph repository does not include.

Our question at the moment is; 'Can we run these Ceph versions together, probably mismatched between different Reef versions'. Unfortunately we've not got any extra hardware laying around that we could prepare and simply `ceph-volume lvm activate all`, it has to be migrated node-per-node.
Has anyone else done this before, or ran into an issue with this?

Thanks!
 
  • Like
Reactions: xorp
Its POSSIBLE to do what you ask. but I wouldnt, because ceph is pretty sensitive and trying to keep parts of the cluster with one central configuration AND the rest synchronized is quite failure prone.

Unfortunately we've not got any extra hardware laying around that we could prepare and simply `ceph-volume lvm activate all`
This wouldnt work anyway. you need the same monitor data from the old cluster.

Depending on how many nodes you have, how the OSDs are distributed, and how much excess capacity it may be possible to do an in place migration- which is what I would do in your shoes.