Proxmox v4 - Ceph OSD not liking CCISS disks on HP C30000

[mjoconr]

Hi All

I'm trying to enable a disk in proxmox 4.x under the CEPH system. I'm getting an error 'no such block device '/dev/cciss!c0d0' (500)' when trying to create the OSD.
The correct path should be /dev/cciss/c0d0



Thanks

 

[Q-wulf]

I have never worked with Smart Array before.

You could try and zap the Disks beforehand to see what error you will get (if any)


ceph-disk list <-- check what path ceph thinks your accessing the disks
ceph-disk zap /dev/sdX <-- Deletes partition Table - erases all data compare with http://docs.ceph.com/docs/v0.71/rados/deployment/ceph-deploy-osd/

 

[mjoconr]

So I've done as you indicated and it would seem to be a bug in Proxmox.

root@blade7:~# ceph-disk list
/dev/cciss/c0d0 other, unknown
/dev/cciss/c0d1 other, unknown
/dev/cciss/c0d2 other, unknown
/dev/cciss/c0d3 other, unknown
/dev/cciss/c0d4 other, unknown
/dev/cciss/c0d5 other, unknown
/dev/sda :
 /dev/sda1 other, 21686148-6449-6e6f-744e-656564454649
 /dev/sda2 other, vfat
 /dev/sda3 other, LVM2_member
/dev/sdb :
 /dev/sdb1 ceph data, active, cluster ceph, osd.2, journal /dev/sdb2
 /dev/sdb2 ceph journal, for /dev/sdb1

As you can see ceph sees the drives with an extra forward slash but inside the web interface there seen as a !. How do I fix this ?

Mike

 

[Q-wulf]

I am not sure how you'd fix that part in Proxmox.

Might be a good idea to add it to proxmox bugzilla. I asume its a bug in pveceph, not parsing the output of "ceph-disk list" properly or not using it in the first place.
https://bugzilla.proxmox.com/





As a quick work around you could try and add the Disks in Ceph via CLI.
http://docs.ceph.com/docs/master/man/8/ceph-disk/

To know what commands you'd need you basically go to Proxmox -> Ceph -> Disk -> Create OSD. Select one of the "missformated" disks - and execute. It should then tell you what command pveceph is trying to execute. If it does not, double-click the failed "task" in the task view. It should list all you need to manual add the OSD via CLI to the cluster.

 

[manu]

Q-Wulf is right, can you open a bug report for that ?

It seems the issue was discussed on http://forum.proxmox.com/threads/18220-No-such-block-device-Proxmox-3-2-and-Ceph-configuration, but unfortunately the thread was buried with other topics and the initial point was lost ( this is why thread hijacking is nefast )

 

[mjoconr]

I found a bug report for the same issue, https://bugzilla.proxmox.com/show_bug.cgi?id=702. It seem like the DEV's have taken no interested as the status as not been updated or assigned to anyone.

The drive in this case are local and non raided,. the annoying things is that the GUI and the CLI both see the drives but will not activate them.

 

[wolfgang]

The problem is we have not such HW and so we can't fix it.

 

[Q-wulf]

The problem is we have not such HW and so we can't fix it.

How does pveceph determine the disks / disklocation ?

does it parse "ceph-disk list" output? ??

ceph-disk list
/dev/cciss/c0d0 other, unknown
/dev/cciss/c0d1 other, unknown
/dev/cciss/c0d2 other, unknown
/dev/cciss/c0d3 other, unknown
/dev/cciss/c0d4 other, unknown
/dev/cciss/c0d5 other, unknown
/dev/sda :
/dev/sda1 other, 21686148-6449-6e6f-744e-656564454649
/dev/sda2 other, vfat
/dev/sda3 other, LVM2_member
/dev/sdb :
/dev/sdb1 ceph data, active, cluster ceph, osd.2, journal /dev/sdb2
/dev/sdb2 ceph journal, for /dev/sdb1

because "ceph-disk list" will find the proper path, while pveceph seems to be listing them as

/dev/cciss!c0d0

that sure seems like an issue in a parsing script somewhere.

 

[mjoconr]

Yes, from the other forum posts and my look at this issue its clear to me that this is a design and script issue.

 

[wolfgang]

That's true the parse do not support this and this is where the problem starts, when you try to fix something but do not know what you have to parse.
and no it is not only /dev/cciss/x to parse there are dev information that we need.

 

[Q-wulf]

Oh okay, i thought you guys just wrote a parser for pvceph that parses the output of "ceph-disk list". my mistake.

 

[wolfgang]

That would be nice
I will think about it how I can fix this without having this controller.

 

[mjoconr]

The cards which use this controller can be purchased from eBay very cheaply ($50 AU). My 6 drive blade has a standard PCI-E card in it with two MiniSAS connectors which are connected to via standard flexible cables.

The website for the cciss driver give an indication of the cards which are supported.
http://cciss.sourceforge.net/

I would be happy to donate the funds needed to purchase a card for testing. The card should be able to be fitted to your test bench because of its standard connectors.

 

[wolfgang]

Is it not possible to user the hspa driver because I know the identify as /dev/sdx.

 

[KnowVation]

Hi!

In the past i had the same problem, but i fixed it.
The HDDs should have no raid system for ceph. (HP Bios Config)
It works only, if you add the /dev/cciss disk manully, over the GUI it wont work! Its important that you format the disk in proxmox console with "fdisk /dev/cciss..." then "d" and then "w".
After that create the OSDs manually; pveceph createosd /dev/cciss...
I think i have manually changed the config for the osds (because of the ! and / for the HP Server Disks).. but i am not shure. Is there a config file for OSDs?

Best regards, Roman

 

[mjoconr]

Is it not possible to user the hspa driver because I know the identify as /dev/sdx.

As I understand it the hspa driver support the new P4xx cards but not the one used in these storage blades.

 

[pacoabal]

Hello:
I have found the answer for cciss ! problem in another thread in this forum. It works for me.
https://forum.proxmox.com/threads/no-such-block-device-proxmox-3-2-and-ceph-configuration.18220/

 

[mjoconr]

In general that fixes the issue via a patch but the issue its self is not fixed and will come back to be a pain.