Proxmox Ceph Pools

[hac3ru]

Hello,

While I understand that this is not really a Proxmox question, I've found no other place to ask this into so here we go:
I've installed Proxmox with 3 nodes, each having additional disks installed in order to form a Ceph cluster. I did so, having 3 osds (1 on each host) and creating two Pools. Now, as of my understanding, the data from one Pool should not be accessible from another Pool and vice-versa. To my amazement, after creating the 2nd pool and mounting it inside a Linux machine, I was able to browse through the data from the first Pool. What am I missing? Aren't Pools supposed to be like LUNs and the data from one Pool is only accessible inside that Pool?

Thank you in advance.

 

[Alwin Antreich]

What do you mean by pools, Proxmox VE storage or Ceph pool?

In the end it depends on the permissions given. A hyper-converged PVE + Ceph cluster uses on set of credentials.
https://docs.ceph.com/en/octopus/architecture/#about-pools

 

[hac3ru]

What do you mean by pools, Proxmox VE storage or Ceph pool?

In the end it depends on the permissions given. A hyper-converged PVE + Ceph cluster uses on set of credentials.
https://docs.ceph.com/en/octopus/architecture/#about-pools

I mean Ceph pools. From what I know from storage devices, Pools / LUNs do not share data among each other. Now, I created 2 Ceph Pools and from one pool I can see the data from the other one. "The Ceph storage system supports the notion of ‘Pools’, which are logical partitions for storing objects." doesn't this mean that the data among pools are not shared?
I created two pools, I created an additional user, so I have the admin user and another called user1. The two pools are named PoolA and PoolB. Admin obviously has access to both while user1 only has write access to pool B.
I mounted the ceph storage on a Linux machine using
mount -t ceph <ip of mon>:/ /mnt -o name=user1,secretfile=<path to secret> and I can see the content of Pool A but, weirdly enough, I cannot write to the storage. So I either have to specify the pool (I found no info about this on the web), or I am missing some info in understanding how Ceph works.
Coming back to Proxmox, I added both Ceph Pools as storages. They hold the same data, have the same usage, etc.....

The end goal of this is: I want to store VMs on the cluster, to have a highly available Proxmox cluster AND I also want to create a separate Ceph Pool (I guess it'a a pool), to mount that pool to some Linux workers and save some data, because I need that data to be shared across the Linux machines.

 

[Alwin Antreich]

Coming back to proxmox, I added both Ceph Pools as storages. They hold the same data, have the same usage, etc.....

Same usage and same data, suggests that it is in fact the same pool that is configured as two different storage.

mount -t ceph <ip of mon>:/ /mnt -o name=user1,secretfile=<path to secret> and I can see the content of Pool A but, weirdly enough, I cannot write to the storage. So I either have to specify the pool (I found no info about this on the web), or I am missing some info in understanding how Ceph works.

The client capabilities need to be limited to a specific pool.
https://docs.ceph.com/en/octopus/rados/operations/user-management/#authorization-capabilities

The end goal of this is: I want to store VMs on the cluster, to have a highly available Proxmox cluster AND I also want to create a separate Ceph Pool (I guess it'a a pool), to mount that pool to some Linux workers and save some data, because I need that data to be shared across the Linux machines.

If those workers write objects directly to Ceph, then it's a pool. Otherwise you may want to look into CephFS or use a NFS/SMB server living on Ceph.

 

[hac3ru]

Same usage and same data, suggests that it is in fact the same pool that is configured as two different storage.


The client capabilities need to be limited to a specific pool.
https://docs.ceph.com/en/octopus/rados/operations/user-management/#authorization-capabilities


If those workers write objects directly to Ceph, then it's a pool. Otherwise you may want to look into CephFS or use a NFS/SMB server living on Ceph.

Well, I do see different Ceph Pools:

# ceph osd pool stats
pool device_health_metrics id 1
  nothing is going on

pool cephfs_data id 2
  nothing is going on

pool cephfs_metadata id 3
  nothing is going on

pool VM_Pool id 4
  nothing is going on

pool App_Pool id 5
  nothing is going on

pool ISOs id 6
  nothing is going on

pool test id 9
  nothing is going on

pool Test2 id 11
  nothing is going on

#ceph auth get
[client.user1]
        key = AQBfXkNgUXR8EBAAlfDVilUNsh/XTx7n5HXqdw==
        caps mds = "allow r"
        caps mon = "allow r"
        caps osd = "allow rw pool=Test2"

Mounting Ceph's Pools VM_Pool as RDB and ISOs as CephFS inside Proxmox VE Storage.
Mounting Test2 on Linux, using user1 username and key and it has access to the ISOs Pool. What am I doing wrong???
Mounting on Linux using

mount -t ceph <monitor ip>:/ /mnt -o name=user1,secretfile=<path to secret file>,noatime

Thank you for your availability and help.

 

[Alwin Antreich]

Mounting Test2 on Linux, using user1 username and key and it has access to the ISOs Pool. What am I doing wrong???

CephFS has two pools, the cephfs_data & cephfs_metadata. The latter holds the directory structure and file metadata. The capabilities are only limiting one of the pools.

CephFS also has a little more permissions that need to be set.
https://docs.ceph.com/en/octopus/cephfs/client-auth/

 

[hac3ru]

CephFS has two pools, the cephfs_data & cephfs_metadata. The latter holds the directory structure and file metadata. The capabilities are only limiting one of the pools.

CephFS also has a little more permissions that need to be set.
https://docs.ceph.com/en/octopus/cephfs/client-auth/

So you're telling me that I can create a single CephFS type pool actually?
So I should create the CephFS pool (already created) and limit users on a per directory basis? That way I can actually separate the application files from the Proxmox VE VM / backup / whatever files.

 

[Alwin Antreich]

So you're telling me that I can create a single CephFS type pool actually?

No. I meant, that CephFS always consist out of two pools.

So I should create the CephFS pool (already created) and limit users on a per directory basis? That way I can actually separate the application files from the Proxmox VE VM / backup / whatever files.

You can limit access and visibility but they will be stored on the same CephFS. Much like NFS/SMB.

 

[hac3ru]

No. I meant, that CephFS always consist out of two pools.


You can limit access and visibility but they will be stored on the same CephFS. Much like NFS/SMB.

Ok, so CephFS as a filesystem that can be mounted inside an OS needs a data pool and a metadata pool. Fine so far but... how can I create multiple CephFS filesystems...?
As I said, I want the data from one pool / filesystem to be inaccessible from the other pool / filesystem.
https://docs.ceph.com/en/latest/cephfs/multifs/ I guess this helps? Create multiple filesystems so the data will be "separated" (I know it'll be stored in the same Ceph cluster)...?

 

[Alwin Antreich]

I don't believe that's what you want/need (but I could be wrong). The permissions can be set in a way that only sub-paths can be mounted by a specific client.

Multiple CephFS is a feature coming in Pacific, so its not available yet (or production ready). It will use way more resources, since multiple MDS services will be needed in addition. And in my opinion that feature makes most sense for hosted environments, where the tenant is not always the same entity.

 

[hac3ru]

I don't believe that's what you want/need (but I could be wrong). The permissions can be set in a way that only sub-paths can be mounted by a specific client.

Multiple CephFS is a feature coming in Pacific, so its not available yet (or production ready). It will use way more resources, since multiple MDS services will be needed in addition. And in my opinion that feature makes most sense for hosted environments, where the tenant is not always the same entity.

Setting the permissions so that a given user can only access a certain directory fixes my issue. I wanted to have separated data, data from Proxmox VE and data from the app, so a developer / a bug in the app won't delete anything Proxmox VE related and vice-versa.

In my opinion, I was expecting Pools to be a Logical Unit, with separated data in each. Like a LUN in NAS.

 

[Alwin Antreich]

In my opinion, I was expecting Pools to be a Logical Unit, with separated data in each. Like a LUN in NAS.

Well, they kinda are. But its more like spawning multiple NFS/SMB servers.

Setting the permissions so that a given user can only access a certain directory fixes my issue. I wanted to have separated data, data from Proxmox VE and data from the app, so a developer / a bug in the app won't delete anything Proxmox VE related and vice-versa.

Proxmox VE has root privileges to begin with. But you could use a separate user for the CephFS storage. That user has only write access to its folders (eg. dump, iso, templates), but only read access the root path of the CephFS. This way the Proxmox VE process will not be able to write/delete data.

 

[hac3ru]

Well, they kinda are. But its more like spawning multiple NFS/SMB servers.


Proxmox VE has root privileges to begin with. But you could use a separate user for the CephFS storage. That user has only write access to its folders (eg. dump, iso, templates), but only read access the root path of the CephFS. This way the Proxmox VE process will not be able to write/delete data.

I guess that's the way I'll take.

Thank you for your help, mate.