Using external Ceph cluster

[h3imdull]

We have a Ceph cluster running outside of our Proxmox cluster and I wanted to add this storage to our proxmox cluster. I tried to follow the instructions which was add keyring in /etc/pve/priv/ceph/[storageID].keyring and then add your rdb: storagID to /etc/pve/storage.conf. I now see the rbd storage in cluster: storage but when I do anything theres rbd list errors. (like migrate node to thats not even using the rbd, like a iscsi vm to iscsi.. )

So my question is IF we need to install ceph on the proxmox nodes? am I missing something else?

 

[Alwin]

So my question is IF we need to install ceph on the proxmox nodes? am I missing something else?

Depends on how big the version difference is. Nautilus can speak with Octopus (newer) and Luminous (older). And what permissions the user of the keyring has.

 

[h3imdull]

Our external Ceph cluster is Nautilus.

 

[Alwin]

Our external Ceph cluster is Nautilus.

And what's the ceph osd get-require-min-compat-client? By default Debian ships with the Luminous client.

 

[h3imdull]

so it looks like the ceph binaries that are installed on the Proxmox hosts is 12.2 so would the recommended action be to install nautilus from the croit or Proxmox deb repo ?

root@prdopspve02:~# ceph osd get-require-min-compat-client
2020-12-18 09:51:09.288536 7f1535b4e700 -1 Errors while parsing config file!
2020-12-18 09:51:09.288548 7f1535b4e700 -1 parse_file: cannot open /etc/ceph/ceph.conf: (2) No such file or directory
2020-12-18 09:51:09.288550 7f1535b4e700 -1 parse_file: cannot open ~/.ceph/ceph.conf: (2) No such file or directory
2020-12-18 09:51:09.288569 7f1535b4e700 -1 parse_file: cannot open ceph.conf: (2) No such file or directory
Error initializing cluster client: ObjectNotFound('error calling conf_read_file',)

 

[Alwin]

root@prdopspve02:~# ceph osd get-require-min-compat-client

That won't work, since only a RBD/CephFS storage is configured. No ceph.conf is used then.

so it looks like the ceph binaries that are installed on the Proxmox hosts is 12.2

But that doesn't say what the require-min-compat-client of the cluster is.

install nautilus from the croit or Proxmox deb repo

We can only give support on our Ceph packages. A simple pveceph install will take care of it.

 

[h3imdull]

So the biggest issue was that the I had not created the rdb pool on storage side ... duh! after pool was created all I needed was put key ring in /etc/pve/priv/ceph/ and us gui to add rbd. I'm now able to use my external Ceph cluster as an RBD storage backend for VMs.

 

[leihnix]

Dear Community
I have the same Problem, I installed a external Ceph Cluster using cephadm. (https://ralph.blog.imixs.com/2020/04/14/ceph-octopus-running-on-debian-buster/). Then I created a replicated pool (T_Pool) and applied the rbd Application.



I copied the Keys from my ceph cluster to the proxmox node:
scp /etc/ceph/ceph.client.admin.keyring proxmox:/etc/ceph
scp /etc/ceph/ceph.conf proxmox:/etc/ceph

I copied the file on the proxmox node as following:
cp /etc/ceph/ceph.conf /etc/pve/priv/ceph/T_Pool.conf
cp /etc/ceph/ceph.client.admin.keyring /etc/pve/priv/ceph/T_Pool.keyring

In the Proxmox Webinterface I added my pool.



my pool is now visible with a question mark but the size on the pool is recognized.





When I want to create a VM into that pool i get an error: "TASK ERROR: unable to create VM 115 - rbd error: rbd: listing images failed: (2) No such file or directory"

Can someone tell me what is going wrong?

On Proxmox I have
root@px01:/etc/pve/priv/ceph# ceph --version
ceph version 15.2.13 (1f5c7871ec0e36ade641773b9b05b6211c308b9d) octopus (stable)

On Ceph I have
root@v-ceph-01:/# ceph --version
ceph version 15.2.13 (c44bc49e7a57a87d84dfff2a077a2058aa2172e2) octopus (stable)

Kind regards
Mike

 

[leihnix]

I made a seperate thread for my Problem here:

https://forum.proxmox.com/threads/using-external-ceph-cluster.93453/

Kind regards
Mike