[SOLVED] Ceph RBD datastore status is showing unknown

[nawapon.je]

I have a new 4 nodes cluster with local ceph. after deployed new ceph cluster, I create RBD pool and mount to my proxmox cluster and status show Unknown. but I create cephfs and mount to cluster it work correctly. I need to use ceph rbd

pool I cant create VM in this unknown rbd pool.

 

[aaron]

Can you please show the contents of the /etc/pve/storage.cfg file?

 

[nawapon.je]

Ok.

dir: local
        path /var/lib/vz
        content vztmpl,backup,iso

lvmthin: local-lvm
        thinpool data
        vgname pve
        content rootdir,images

rbd: SRB2-DATASTORE-CEPH
        content rootdir,images
        krbd 1
        pool SRB2-DATASTORE-CEPH

cephfs: SRB2-DATASTORE-CEPH-BACKUP
        path /mnt/pve/SRB2-DATASTORE-CEPH-BACKUP
        content iso,vztmpl,backup

 

[aaron]

First, the pools start with "TST-..." while in the storage.cfg eveything starts with "SRB2-...".
I can understand why this does not work for the RBD pool but I am confused why it works for CephFS. Did you rename the pools at some point?

 

[aaron]

Or it might be even a bit different, but I am even more confused about the TST and SRB2 prefixes.

What does a ceph osd lspools show?

 

[nawapon.je]

sorry, It's my fault. I have 2 cluster first is a TST.... and second is a SRB2. I have a problem in SRB2. Here's a recap SRB2 pic.

 

[nawapon.je]

I tried to mount RBD cross to other site it can use normally but if I use RBD other site to use in the problem site, It cant use same.
and I tried to redeploy SRB2 cluster but nothing different .... What shoud I do?
I only know different between 2 site is TST deploy with image pve6.3.1.iso and SRB2 deploy with image pve6.4.1.iso

 

[ermanishchawla]

It should work like this

For my site A
cat /etc/pve/storage.cfg

rbd: vm-siteA
content images,rootdir
krbd 1
pool vm-siteA

for my siteB

rbd: vm-siteB
content images,rootdir
krbd 1
pool vm-siteB


Now suppose you want to mount cross, it should be like this

for Site A
rbd: vm-siteA
content images,rootdir
krbd 1
pool vm-siteA
rbd: vm-siteB
content images,rootdir
krbd 1
pool vm-siteB


Now once you do that vm-SiteB shall show status as Unknown. Reason is keyring file is missing

Now go to site B

go to path /etc/pve/priv/ceph

you will find a file

vm-siteB.keyring file

copy this file and paste in the site A Cluster in the same path /etc/pve/priv/ceph

Now you should see 2 files in this path

vm-siteA.keying
vm-siteB.keyring

Thats it , you are ready to use external RBD

You may also need to add monitor IP in /etc/pve/storage.cfg file for remote RBD

 

[nawapon.je]

I should have miscommunicated.
I have a problem to in site B.. with RBD-siteB
I cant mount RBD-siteB in siteB but it show status Unknown and I cant use it anymore.
I tested by use RBD-siteB mount into siteA....It work correctly so I tested by use RBD-siteA mount into siteB...it shown status unknown same as a RBD-siteB mount to siteB (It's a local RBD storage...why I cant use it)

I tried to use ceph RBD site B like a site A . but I dont know what I did wrong.
I only know I deploy site A with 6.3.1.iso and site B(problem) is 6.4.1.iso [same ceph octopus version ]

 

[ermanishchawla]

Your keyring file only is messed up. Just check over there.

 

[nawapon.je]

It's seem correctly. Where should I go other to check?

 

[ermanishchawla]

Can you share the following on Site A and Site B individually


ceph df
ceph osd lspools

ls -ltr /etc/pve/priv/ceph/

cat /etc/pve/storage.cfg

I just want to confirm onething thats it, I guess I know the issue

 

[nawapon.je]

Ok, now I solved my problem and found something
if I deploy proxmox 6.4.1 with iso and install ceph without upgrade version proxmox , Its work. but if I deploy proxmox 6.4.1 and upgrade version to 6.4.6 and install ceph thats cause a problem

I dont know what different between 6.4.1 and 6.4.6 after upgrade.

 

[ermanishchawla]

Ok, now I solved my problem and found something
if I deploy proxmox 6.4.1 with iso and install ceph without upgrade version proxmox , Its work. but if I deploy proxmox 6.4.1 and upgrade version to 6.4.6 and install ceph thats cause a problem

I dont know what different between 6.4.1 and 6.4.6 after upgrade.

There is no relationship with this, it is most likely keyring issue. Anyhow if your issue is solved, mark thread as solved

 

[nawapon.je]

There is no relationship with this, it is most likely keyring issue. Anyhow if your issue is solved, mark thread as solved

Thank you for your help @ermanishchawla