Ceph and LSI 3108 with JBOD

[SteveITS]

I’ve read many threads but haven’t quite figured it out… I know one should not use RAID with Ceph. Do drives connected as JBOD/single drives on a RAID controller (3108) still fall into that warning category?

We’re hoping to reuse some older hardware.

 

[alexskysilk]

LSI 3108 is a pure controller and doesnt support IT mode, but it does have jbod mode which might be ok.

 

[SteveITS]

Thanks, yeah that distinction is why I’m wondering. I know it’s fine with Virtuozzo Storage.

 

[alexskysilk]

give it a shot if it doesnt yield acceptable results (or if it does) please update the thread.

 

[SteveITS]

FWIW after a week no known issues on two so far. Using JBOD for the ZFS boot mirror as well as various Ceph OSD drives.

 

[p3ter_b]

Sorry for negrobumping, but: I have (I guess) a similar setup with a Supermicro AVAGO 3108 MegaRAID BIOS.

Which firmware version of the 3108 controller are you using?
(Mine is FW 4.670.00-0021)

Grateful for feedback.

 

[SteveITS]

I do not know sorry. We were reusing an old server at the time so it was probably pretty old. Had no issues with it though, in JBOD mode.

 

[p3ter_b]

@SteveITS: Thanks for your swift reply!
Well. It was worth asking

Yes the server I'm using now is also very old ([SSG-6048R-E1CR36N](https://www.supermicro.com/products/system/4U/6048/SSG-6048R-E1CR36N.cfm)) - around 2016. The hardware is still running fine without issues though, and I'm hoping to get pass-as-JBOD working.
Currently it's grayed out... But that may be, because I have a Virtual Device (VD) assigned for the OS disks.

Thanks again! Gives me hope that it'll be possible.

 

[SteveITS]

In Virtuozzo IIRC it was a hardware RAID1 boot with the remaining drives JBOD. In PVE we used JBOD for a ZFS boot mirror.

 

[p3ter_b]

Oookay: So as I understand you, it was possible to have RAID1 for boot / OS - and JBOD on the same controller at the same time?
Excellent news.

btw: I've read I can use [broadcom's 'storcli'](https://techdocs.broadcom.com/us/en/storage-and-ethernet-connectivity/enterprise-storage-solutions/storcli-12gbs-megaraid-tri-mode/1-0/v11869215/v11673442.html) tool for checking/setting JBOD, like:

Quote from [broadcom's storcli documentation](https://docs.broadcom.com/doc/12352476):

This command sets the drive state to JBOD.
Input example:
`storcli /c1/e56/s3 set jbod`

 

[p3ter_b]

Good news everyone: It works!

In order for my reviving of an old thread to be of use for anyone: I got it working, and can provide details - so anyone else may benefit from my/our time.
(Sorry, I don't have working clipboard from the server to my host - so here are short instructions on Proxmox v9.1)

1) Install broadcom StorCLI tool
* Broadcom StorCLI tool (from 2023-08-24)
* apt install zip, alien
* Download, unzip, then use Debian's "alien" tool to convert the Linux/*.rpm to deb.
* Install the .deb package, using dpkg -i
* Symlink the binary: ln -s /opt/MegaRAID/storcli/storcli64 /usr/local/bin/storcli

2) Check if storcli show works, and lists the controller adapter.
Mine lists: "AVAGO3108MegaRAID" - good!

3) Set all unassigned disks to JBOD to pass them through:
storcli /c0 jbod=on

Then refresh the disks view in PVE, and you should see all remaining disks free to use!

(NOTE: The boot/OS disks are on the same controller assigned as RAID-1 VD. So I can confirm it works to pass-through JBODs, even if Virtual Devices are configured on the controller).


Thanks everyone!

 

[SteveITS]

possible to have RAID1 for boot / OS - and JBOD on the same controller at the same time?

Yes it was. Though as mentioned in Proxmox VE we used ZFS which isn't compatible with hardware RAID. Though other file systems are fine.