Using external Ceph cluster

[leihnix]

Dear Community
I have the same Problem, I installed a external Ceph Cluster using cephadm. (https://ralph.blog.imixs.com/2020/04/14/ceph-octopus-running-on-debian-buster/). Then I created a replicated pool (T_Pool) and applied the rbd Application.



I copied the Keys from my ceph cluster to the proxmox node:
scp /etc/ceph/ceph.client.admin.keyring proxmox:/etc/ceph
scp /etc/ceph/ceph.conf proxmox:/etc/ceph

I copied the file on the proxmox node as following:
cp /etc/ceph/ceph.conf /etc/pve/priv/ceph/T_Pool.conf
cp /etc/ceph/ceph.client.admin.keyring /etc/pve/priv/ceph/T_Pool.keyring

In the Proxmox Webinterface I added my pool.



my pool is now visible with a question mark but the size on the pool is recognized.



When I want to create a VM into that pool i get an error: "TASK ERROR: unable to create VM 115 - rbd error: rbd: listing images failed: (2) No such file or directory"

Can someone tell me what is going wrong?

On Proxmox I have
root@px01:/etc/pve/priv/ceph# ceph --version
ceph version 15.2.13 (1f5c7871ec0e36ade641773b9b05b6211c308b9d) octopus (stable)

On Ceph I have
root@v-ceph-01:/# ceph --version
ceph version 15.2.13 (c44bc49e7a57a87d84dfff2a077a2058aa2172e2) octopus (stable)

Kind regards
Mike

 

[aaron]

What version is the Ceph cluster?

So you have the keyring file located at /etc/pve/priv/ceph/<storage>.keyring? And its contents match if you check against the keyrings in the Ceph cluster? ceph auth ls

Try to remove any ceph.conf file as it is not needed if you just have an external Ceph cluster that is not managed by Proxmox VE.

 

[leihnix]

Dear Aaron

the contents of the keyring matches the one in the "ceph auth ls"



On the Ceph cluster I have_
root@v-ceph-01:/# ceph --version
ceph version 15.2.13 (c44bc49e7a57a87d84dfff2a077a2058aa2172e2) octopus (stable)

On the Proxmox node i have:
root@px01:/etc/pve/priv/ceph# ceph --version
ceph version 15.2.13 (1f5c7871ec0e36ade641773b9b05b6211c308b9d) octopus (stable)

I tried to look for unnessecary ceph.conf files, and found the following files:

root@px01:~# find / -name ceph.conf
find: ‘/proc/2650314’: No such file or directory
1. /etc/ceph/ceph.conf
2. /usr/lib/tmpfiles.d/ceph.conf

the second file seems somewhat irrelevant. Otherwise i dont see any additional ceph.conf files.

Kind Regards Mike

 

[aaron]

Does the following command work?

rbd -p <POOL> -m <MON1>,<MON2> -n client.admin --keyring /etc/pve/priv/ceph/<STORAGE>.keyring ls

 

[leihnix]

Hallo Aaron

the command comes back with a empty result

root@px01:/etc/pve/priv/ceph# /usr/bin/rbd -p T_Pool -m 192.168.10.71,192.168.10.72 -n client.admin --keyring /etc/pve/priv/ceph/T_Pool.keyring ls
root@px01:/etc/pve/priv/ceph#

Kind regards Mike

 

[aaron]

the command comes back with a empty result

As expected, since no disk image is present yet? That means that the connection itself is working.

root@px01:~# find / -name ceph.conf
find: ‘/proc/2650314’: No such file or directory
1. /etc/ceph/ceph.conf
2. /usr/lib/tmpfiles.d/ceph.conf

the second file seems somewhat irrelevant. Otherwise i dont see any additional ceph.conf files.

Try to remove the /etc/ceph/ceph.conf file as well, since all you have is the external cluster and not one managed by PVE itself.
For external Ceph clusters, the RBD command is used as we did it manually and there is no need for the ceph.conf. Depending on what you have in there, it might even confuse PVE.

 

[leihnix]

Hello Aaron

Thank you for your answer!

after hours of trial and error I finally got it working. Unfortunately I cannot remember what the cause was.
The only thing I have now left is the question mark, the unknow status and the usage graph with the 1070 date.





Is there a solution to that?

Regards Mike

 

[aaron]

Try to restart the pvestatd daemon.

 

[leihnix]

Excellent help! It woked after restarting pvestatd.



Thank you very much.