Minimally required packages for external ceph cluster

[grin]

Hello there!
I have tried to search the forum but found no definite answer, and the documentation seem only mention "hyper-converged" clusters (where the ceph daemons are on the proxmox nodes, which is - based on exprience - is not a good idea in the long run).
Proxmox see the cluster storage fine (it only requires a few files, basically ceph-common, ceph.conf in /etc/ceph and/or /etc/pve/priv/).

However it cannot show (and definitely not able to manage) the ceph cluster state.

My guess is either some packages it requires or some permissions it lacks (since I use a specific user not client.admin). But I cannot be sure.

Could someone please specify what proxmox needs to be able to see the [external] cluster state? It seems to be a waste (not to mention potantial trouble spot) to install the whole ceph infrastructure on client-only proxmox nodes.

Thank you!

 

[VictorSTS]

The whole point of integrating Ceph with Proxmox is to use an hyper-converged storage network, where all or some of the Ceph nodes are the same servers running Proxmox VMs/LXC. If you are not using the integrated hyper-converged Ceph integration you will have to manage your Ceph cluster with it's own tooling or 3rd party tools and simply publish rbd / cephfs storage space to be used by your Proxmox cluster, as if it were any other kind of external storage like a NAS, SAN, etc.

I'm not aware that Proxmox is able to manage external ceph clusters from the GUI.

Why are you saying that it is not a good idea in the long run?

Obviously this is practical / useful up to some cluster size. If you need 200 servers with 40 OSD each and Petabytes of storage, probably Proxmox isn't the best tool to manage that.

 

[grin]

The whole point of integrating Ceph with Proxmox is to use an hyper-converged storage network, where all or some of the Ceph nodes are the same

Please do not hijack threads. Open one with your question (and ping me there), or use direct messages and I'll answer there.

Here I'd prefer to get answers to my question, thank you.

 

[VictorSTS]

I was:

1.- Giving my answer to your question.
2.- Asking your point of view on this because I'm obviously missing something.

If thats hijacking a thread...

Peace

 

[fiona]

Hi,
as far as I'm aware, Proxmox VE is not designed to manage external clusters. Most of the commands would need to be executed on the node with the OSD/MON/etc. anyways and Proxmox VE doesn't have any system information about those external servers, which it would need to create a new OSD for example. Much could be implemented via ssh, but again, it's not designed for this. Proxmox VE is just a client in this case.

Why not manage the cluster where it is?

 

[grin]

as far as I'm aware, Proxmox VE is not designed to manage external clusters.

I am not sure you're correct. Proxmox UI correctly has all the features required to use an external Ceph cluster, including handling multiple mons and authorization.

I would also guess that Proxmox uses standard Ceph API calls to get information about the clusters, so when ceph status works on the machine Proxmox is supposed to be able to gather all the status information it needs the usual way, possibly no changes necessary.

Most of the commands would need to be executed on the node with the OSD/MON/etc. anyways and Proxmox VE doesn't have any system information about those external servers, which it would need to create a new OSD for example.

You are mixing up two different topics here, which may have caused some confusion:

1. Showing cluster status: it needs that the local machine have read access to the mon/mgr, which Proxmox has, so these should work "out of the box"
2. Changing cluster state: this won't work without efforts and I do not think it is required at all. I don't expect this and this post isn't about it.

So, what I am talking about is showing cluster status in the dashboard and status pages, showing cluster iops/speed, daemon states etc. I do not talk about changing these.

Much could be implemented via ssh, but again, it's not designed for this. Proxmox VE is just a client in this case.

Indeed, and as a client it sees the complete status of the cluster to show to the user.

Why not manage the cluster where it is?

Indeed I do. Please carefully observe my opening post: I was asking about seeing cluster state, not managing it.

I believe the two parts (showing status vs. management) are not separated right now so Proxmox does a much wider check than required: the dashboard and display parts should work when the cluster (mon/mgr) is visible, and the change buttons ("management") shall be inactive only.

If you're curious: I am using Ceph dashoboards to see the stats, but it would be also useful to see all the cluster status in one place on the Proxmox dashboard and ceph status page.

 

[fiona]

I am not sure you're correct. Proxmox UI correctly has all the features required to use an external Ceph cluster, including handling multiple mons and authorization.

I would also guess that Proxmox uses standard Ceph API calls to get information about the clusters, so when ceph status works on the machine Proxmox is supposed to be able to gather all the status information it needs the usual way, possibly no changes necessary.

There's two layers. The Proxmox VE web UI uses the Proxmox VE API to get its information, nothing else. That Proxmox VE API uses Ceph/RADOS commands and host information to compile the information. The existing Proxmox VE API endpoints for Ceph are below /nodes/{node}/ceph and /cluster/ceph, the cluster here refers to the Proxmox VE cluster of course. Since those can't be used, because the external cluster has nothing to do with the Proxmox VE nodes or cluster, it would require adding new endpoints e.g. in /ceph or below the storage endpoint.

It's not impossible to implement of course, but just to re-iterate that it's not currently designed for this.

You are mixing up two different topics here, which may have caused some confusion:

1. Showing cluster status: it needs that the local machine have read access to the mon/mgr, which Proxmox has, so these should work "out of the box"
2. Changing cluster state: this won't work without efforts and I do not think it is required at all. I don't expect this and this post isn't about it.

So, what I am talking about is showing cluster status in the dashboard and status pages, showing cluster iops/speed, daemon states etc. I do not talk about changing these.

Feel free to open up a feature request on the bug tracker, but be aware that it needs quite a bit restructuring and I'm not sure there's enough demand to warrant that. There's already other solutions to monitor an external Ceph cluster in detail.

Indeed I do. Please carefully observe my opening post: I was asking about seeing cluster state, not managing it.

However it cannot show (and definitely not able to manage) the ceph cluster state.

Did sound a bit like you want both features

 

[grin]

There's two layers. The Proxmox VE web UI uses the Proxmox VE API to get its information, nothing else. That Proxmox VE API uses Ceph/RADOS commands and host information to compile the information. The existing Proxmox VE API endpoints for Ceph are below /nodes/{node}/ceph and /cluster/ceph, the cluster here refers to the Proxmox VE cluster of course. Since those can't be used, because the external cluster has nothing to do with the Proxmox VE nodes or cluster, it would require adding new endpoints e.g. in /ceph or below the storage endpoint.

It's not impossible to implement of course, but just to re-iterate that it's not currently designed for this.

Without browsing through the code, here's one example from Ceph.pm (line 645, or the very end):

    code => sub {
        my ($param) = @_;

        PVE::Ceph::Tools::check_ceph_inited();

        my $rados = PVE::RADOS->new();

        my $rules = $rados->mon_command({ prefix => 'osd crush rule ls' });

It calls the mon for osd crush rule ls, which works with ceph-common only and proxmix with properly configured external mons.
I'd say every sub using mon_command would seamlessly work with both local and external cluster.

It seems to be designed for it, and indeed, well done that, separating the logical layers.

Feel free to open up a feature request on the bug tracker, but be aware that it needs quite a bit restructuring

Okay. From a brief look at the code it needs a few places separating viewing from manipulation, probably mostly in the UI. The checks have to be trimmed for viewing purposes (it only needs a configured mon), and the current checks shall be used for any manipulation routines.

I'll talk with Bugzilla.

and I'm not sure there's enough demand to warrant that.

Well since really serious problems aren't fixed for multiple years now (buggy window scrollbars, for example) I have no doubts about what is going to happen.

Did sound a bit like you want both features

Thank you for helping me specifying better!