Is there any different of RBD(PVE) and RBD externel under proxmox?

[danielc]

Hello,

In my ceph cluster /etc/pve/storage.cfg, i found that:

This works (show as:RBD externel)
rbd: ceph-hdd
content images,rootdir
krbd 1
monhost 10.10.10.1;10.10.10.2;10.10.10.3;10.10.10.4
pool hdd
username admin

This also works (show as: RBD(PVE))
rbd: backup
content images
krbd 0
pool backup

In the ceph tutorial of proxmox, it teachs us using ebd externel.
Why not just use RBD(PVE)? is there any advantages or any problems afterwards???
It is confusing me.
Thanks!

 

[spirit]

rbd(PVE), autoconfigure the rbd storage is installed on proxmox. (no need to specify ip, username,....)

rbd(external), is for define an external cluster (with the monitor ips). But you can also use it with ceph installed on proxmox, no problem.

 

[danielc]

Hello spirit,

Thank you very much for the explaination. So will stick with rbd pve then.

 

[RobFantini]

Hello
I did find a difference between storage rbd pve and external. this may be expected or a bug.

we have a pool created before the option of pve rbd type was created:

rbd: lxc-ceph
        content rootdir
        krbd 1
        monhost 10.11.12.3;10.11.12.8;10.11.12.10
        pool lxc-ceph
        username admin

I tried creating a new rbd type pve . there is an issue with moving vm storage from the old to new:

   TASK ERROR: error with cfs lock 'storage-lxc-ceph-new': rbd error: rbd: list: (95) Operation not supported

that error went away when I deleted the rbd/ceph and used rbd/external.

 

[udo]

Hello
I did find a difference between storage rbd pve and external. this may be expected or a bug.

we have a pool created before the option of pve rbd type was created:

rbd: lxc-ceph
        content rootdir
        krbd 1
        monhost 10.11.12.3;10.11.12.8;10.11.12.10
        pool lxc-ceph
        username admin

I tried creating a new rbd type pve . there is an issue with moving vm storage from the old to new:

   TASK ERROR: error with cfs lock 'storage-lxc-ceph-new': rbd error: rbd: list: (95) Operation not supported

that error went away when I deleted the rbd/ceph and used rbd/external.

Hi Rob,
but your storage aren't named storage-lxc-ceph-new - your example named lxc-ceph! So the error message don't fit to the confg entry - hard to find an error.

Udo

 

[RobFantini]

Hi Rob,
but your storage aren't named storage-lxc-ceph-new - your example named lxc-ceph! So the error message don't fit to the confg entry - hard to find an error.

Udo

I think I messed up the copy and pastes done. I had tried a few different times and may have mixed up the pad i put notes on...

So I'll try recreating the issue and post better debugging data...

 

[RobFantini]

Hi Rob,
but your storage aren't named storage-lxc-ceph-new - your example named lxc-ceph! So the error message don't fit to the confg entry - hard to find an error.

Udo

move disk from

rbd: lxc-ceph
        content rootdir
        krbd 1
        monhost 10.11.12.3;10.11.12.8;10.11.12.10
        pool lxc-ceph
        username admin

to

rbd: lxc-test-pve
        content rootdir
        krbd 1
        pool lxc-test

result

TASK ERROR: error with cfs lock 'storage-lxc-test-pve': rbd error: rbd: list: (95) Operation not supported

 

[RobFantini]

Also on pve screen, the 'lxc-test-pve' the storage icons at each host show a ? and when clicked display:

rbd error: rbd: list: (95) Operation not supported (500)

So it looks like I did something wrong on the set up. I'll check on this tomorrow.

 

[RobFantini]

Hello Udo,

I rechecked the set up and can not figure out why the 'not supported (500)' error occurs. the setup of pool and storage look ok. [ see post #7 above]

/var/log/syslog shows this error :

pvestatd[5655]: rados_connect failed - Operation not supported

any clues to fix this?