CEPH Configuration: CephFS only on a Specific Pool?

[Illydth]

So hopefully this isn't a duplicate of something well asked:

I have 2 pools of disks: The default Ceph pool that I use for VMs (the large pool on the larger disks) and a smaller pool I intend to use as K8s storage. The larger pool I want to keep Exclusively for VM storage, but the smaller pool I want to put CephFS on top of.

And that leads to the basic question: How do I specify the pool that Ceph is going to put data into? I tried creating a "cephfs_data" pool on the secondary area and putting the cephfs_metadata pool on primary, and then using:

ceph fs new cephfs cephfs_metadata cephfs_data

But it returned: "Error EINVAL: poll 'cephfs_data' (id '11') has a non-CephFS application enabled."

I assume because I created it through the normal "pool" creation in the GUI it's giving me this.

Anyone know how to create the CephFS pool on something other than the "Default" Ceph storage?

 

[dlasher]

realize this is a late answer, but ran into this thread as I prepare to re-pool my metadata.

Short answer: Each pool must be "enabled" for specific applications. When you create the pool via the UI or command line, it defaults to RBD. You can see which applications are enabled for your pool like so:

root@pmx2:~# ceph osd pool application get cephfs_metadata
{
    "cephfs": {
        "metadata": "cephfs"
    }
}

root@pmx2:~# ceph osd pool application get cephfs_data
{
    "cephfs": {
        "data": "cephfs"
    }
}

root@pmx2:~# ceph osd pool application get SM32
{
    "rbd": {}
}

Following that model, we'd need to enable the target pool for cephfs:

ceph osd pool application enable {pool-name} {application-name}

or

ceph osd pool application set --pool=poolname --app=string  --key=string --value=string

(will update this thread more after I finish migrating pools)

REFERENCE:
* https://docs.ceph.com/en/quincy/api/mon_command_api/#osd-pool-application-get

 

[FSNaval]

following above steps, I executed the command for the pool i want to add, named vmpool.

when executing the command, i received the following warning:

<<RADOS pool 'vmpool' has another non-CephFS application enabled.>>

Tried to solve it by typing:

<<ceph osd pool application enable vmpool cephfs>>

i received following warning:

<<Are you SURE? Pool 'vmpool' already has an enabled application; pass --yes-i-really-mean-it to proceed anyway>>

If i proceed with the above suggestion, do you know if the vmpool be erased/formatted? This pool already has data stored and i dont want to loose them.

 

[fba]

So you have a rbd pool called vmpool and now you want to put CephFS into that pool? That doesn't work and doesn't make sense, to me at least.
Would you like to elaborate on what you want to achieve?

 

[FSNaval]

i am in the process of moving from docker to docker swarm, in order to better utilize my hardware resources. For this, i have created a swarm cluster and i am investigating my alternatives for shared storage for the swarm cluster.

Currently, on my proxmox cluster i have two pools. One pool is established exclusively from NVME drives, called vmpool and the second pool consists solely on HDDs for bulk storage.

One of the ways for shared storage for docker swarm is cephFS. I want the cephFS to use only vmpool (that has the fast NVME drives) for better performance.

With a small search and asking around, i understand that i have to create a third pool for the swarm cluster, lets name it dockerpool, and on this pool i will have to point the cephFS.

Hope to clear things a little on what i want to do.

 

[readyspace]

following above steps, I executed the command for the pool i want to add, named vmpool.

when executing the command, i received the following warning:

<<RADOS pool 'vmpool' has another non-CephFS application enabled.>>

Tried to solve it by typing:

<<ceph osd pool application enable vmpool cephfs>>

i received following warning:

<<Are you SURE? Pool 'vmpool' already has an enabled application; pass --yes-i-really-mean-it to proceed anyway>>

If i proceed with the above suggestion, do you know if the vmpool be erased/formatted? This pool already has data stored and i dont want to loose them.

Safest option: create a fresh empty pool for CephFS.

 

[FSNaval]

Safest option: create a fresh empty pool for CephFS.

Thank you for your reply.

From what i have read so far, this is what i will do. I will create a new pool (named dockerswarm) and assign to it the OSDs of the NVME drives.


I have read comments that cephFS is not the best solution for a docker swarm cluster, as it was not designed for this. Can you comment on this please?

 

[readyspace]

CephFS works, but it’s metadata-heavy—great for shared POSIX files, not ideal for DB-like/container write-intensive workloads without careful MDS sizing/tuning.

To save the trouble tuning and reduce risk, I'll use RBD volumes (fast, block-level).

 

[FSNaval]

CephFS works, but it’s metadata-heavy—great for shared POSIX files, not ideal for DB-like/container write-intensive workloads without careful MDS sizing/tuning.

To save the trouble tuning and reduce risk, I'll use RBD volumes (fast, block-level).

Noted, thank you. How should i then do your proposal?

The easy way would be to create a shared folder amongst the docker swarm vm's through NFS, but i have read that this also has performance issues.

 

[readyspace]

Hi @FSNaval , i've missed out that you are using NVMe. If you are, using CephFS pool on NVMe will work ok even with metadata heavy. But i will still separate those containers that needs fast disk to use RDB

So... If you need shared files (many containers reading/writing the same folder), then yes — use CephFS on dockerswarm.

If each container just needs its own fast disk (databases, logs, cache, etc.), then use RBD volumes instead — still on dockerswarm, but without CephFS.

 

[FSNaval]

Hi @FSNaval ,
So... If you need shared files (many containers reading/writing the same folder), then yes — use CephFS on dockerswarm.

Thank you!

Above is my use case, since it resembles the current state i currently work, and believe that migrating from single docker node to a docker swarm cluster (keeping the same compose files properly amended for docker swarm will require less changes to be made on them)