CEPH multiple MDS on the same node

[encore]

Hi,

I am running a 5-node ceph cluster (Octopus) and when I increased the number of active MDs from 2 to 4 I experienced a gain of performance in my CephFS.
Since I have a lot of client using the cephfs pool I think it might be a good idea to increase the number of MDs even more.

https://docs.ceph.com/en/octopus/cephfs/multimds/
Ceph supports multiple MDs per node since octopus but unfortunately proxmox GUI does not.

Would it work to create multiple MDs as described above trough the CLI even if the PVE GUI does not support it?

Thank you!

 

[dlasher]

Fair question, I'd like to see an answer as well - given pretty solid test data showing significant advantages to multiple MDS's (like this : https://croit.io/blog/ceph-performance-test-and-optimization) I'd love to see support for more than one MDS per server.

... the CPU usage by each active ceph-mds process comes close to 100%. In other words, with only four active MDSs, that’s about as fast as the cluster can go in these metadata-intensive benchmarks.....In fact, we found that even two MDSs per host are not enough to avoid the bottleneck. So we deployed 20 MDSs total - four per host, and made 16 of them active....

 

[fiona]

Hi,
yes, this should be possible in the CLI. Use pveceph mds create --name <name> with different names to create multiple MDS on the same node. Since it's possible to create multiple Ceph filesystems via Proxmox VE GUI since 7.1, this likely makes sense to expose in the GUI too. Feel free to create a feature request for it.

 

[Stephan Becker]

Same here.
The GUI of Version 7.3-6 still does not support this, at least I think so because when I try to create a sescond MDS on my first cluster node (pveNode0) I get:
"MDS 'pveNode0' already referenced in ceph config, abort"
That effectively limits the number of CephFS on a cluster of n nodes with n MDS set up to n-1 filesystems, if one aims to survive at least a single node failure with all defined filesystems still online (One spare MDS is standing by waiting to take over the MDS that dies with the node).
With n filesystems defined you'll loose the filesystem that is served by the MDS on the failing node.
So taking off a node for maintanance or like is not possible with a number of CephFS set up that is equal to the number of nodes in the cluster.

What is the state of development here?
Is the GUI extension to support multiple MDS on an Node already on the way?

Cheers
Stephan

 

[fiona]

Hi,

Same here.
The GUI of Version 7.3-6 still does not support this, at least I think so because when I try to create a sescond MDS on my first cluster node (pveNode0) I get:
"MDS 'pveNode0' already referenced in ceph config, abort"
That effectively limits the number of CephFS on a cluster of n nodes with n MDS set up to n-1 filesystems, if one aims to survive at least a single node failure with all defined filesystems still online (One spare MDS is standing by waiting to take over the MDS that dies with the node).
With n filesystems defined you'll loose the filesystem that is served by the MDS on the failing node.
So taking off a node for maintanance or like is not possible with a number of CephFS set up that is equal to the number of nodes in the cluster.

What is the state of development here?
Is the GUI extension to support multiple MDS on an Node already on the way?

Cheers
Stephan

nobody linked to a feature request yet, so I'd assume that nobody created one yet. Feel free to do so (after checking that it doesn't exist to make sure) on our bugtracker: https://bugzilla.proxmox.com/ Please also mention your use case for so many file systems there.

For now, you'll just have to configure it via CLI.

 

[modeadmin]

Hello,

I also want this feature. I opened a ticket after checking if there was already one: https://bugzilla.proxmox.com/show_bug.cgi?id=4571