Cisco UCS SAS 2008M-8i (LSI 9211-8i, I think?) Supports JBOD - what, if any, benefit would I get from cross-flashing to LSI IT Mode?

[HardyBiscuit]

Very new to ProxMox...in fact, haven't even installed it yet, but wanted to understand a bit more about storage setup and presentation to ProxMox before I started. I understand (though never used) ZFS and the welcomed benefits that seems to have general acceptance. I understand that if you want to use ZFS, it is best practice (not required though) to effectively disable the IR modes and make the card a passthrough.

I am playing with an older, way over-powered for my use, UCS M3 C220 server. I have 3 options for cards - LSI 9270-8i, CISCO UCS SAS 2008M-8i, or on-board software RAID. I have a bunch of Hitachi SAS SSD's and Toshiba high speed SAS HDD's, which is the only real reason I need the SAS controllers. In reality, I do have a bunch of (not the greatest, but still) SATA SSD's, though with far less space available.

Usage intent is pretty low intensity. A few VMs and a few containers. Probably a solid Ubuntu VM, Win 11 VM, and a Plex Container, Home Assistant, and some sort of Own/Nextcloud setup. The storage I am referring to here will really be for the VMs themselves. I have a 90TB QNAP Nas that has a 10g fiber link that I'll be using for most of my important data.

From my research, it is not at all possible to put the LSI-9270-8i into IT mode, so my only choice would be to RAID it out. I am honestly more interested in working outside of hardware RAID anyhow.

From what I have found, I believe the CISCO UCS SAS 2008M-8i is basically an OEM'd and rebadged LSI 9211-8i which can be flashed into IT mode through a bit of a convoluted process, but I am up for it as I like tinkering. That being said, the Cisco card already supports JBOD out of box (the 9270 doesn't). Since I am not an expert on hardware RAID, how much of a difference is running the Cisco card as JBOD than flashing it to IT mode, or is this essentially the exact same thing?

I sometimes overthink these things and wanted some more experienced than I advice. Thanks!

I should add my reasoning for asking - if JBOD mode is fine (and basically passthrough), then why is anyone trying to IT crossflash this card to begin with? I have seen it mentioned several places around the web)

 

[shanreich]

I would actually take a step back and consider not using a RAID controller at all, since it is actually heavily discouraged when using ZFS. Even the use of JBOD is discouraged in this case. Is there any particular reason why you would want to use one?

 

[HardyBiscuit]

In no particular order

1.) I have a bunch of Enterprise SSD and HDD SAS drives.

2.) That is the only reason why. I don't have to. I can use SATA (albeit on half the number of channels).

(Bonus 3, bc I'm a tinkerer and make things harder than they need to be)

I had a lot of trouble with the setup last night (520B size on the drives that I need to reformat but in a catch 22 situation with the server presenting them as SATA pass through devices to Linux in order to do so.) Perhaps it is time to relent and just got with 4 SSDs and move on.

Thanks for the response!

 

[shanreich]

I understand point 3 fully, since it also applies to me

You could always use the controller but then I would not use ZFS as the filesystem, since it needs direct access to the disks in order to function properly. So if you want to use the controller, then you could always opt for using a different FS. Then you have a reason to continue tinkering with your controller

 

[HardyBiscuit]

So I think I found the answer to my original question. It seems as though (at least with this particular card), that JBOD is actually not direct passthrough (IT Mode), but instead JBOD is actually 8 individual RAID0 VD's. Which...works I guess as JBOD, but isn't the same as direct passthrough. I knew there was probably a reason (otherwise, why the difference?). Interesting.

 

[alexskysilk]

520b drives are meant to be used with controllers that have use (checksum) for the extra 8b per sector. LSI cards dont. Its possible that using the raid engine instead of passthrough makes them usable to the OS since it translates disk to host bus, but its sloppy and pointless.

1. depending on firmware, it may be possible to get the controller to do passthrough even if its not using it firmware (
megacli AdpSetProp EnableJBOD 1 -aALL). if this fails you can/should crossflash it mode firmware.
2. once you have direct access, you should be able to use sgdisk to reformat the disks using 512b sectors.

 

[HardyBiscuit]

So, I was actually incorrect about the UCS SAS 2008M-8i JBOD mode being a bunch of single-disk RAID0's. Turns out, there was some old config that was being applied.
I had decided to just forgo any card and connect directly to on-board..until I did and remembered the biggest reason to go with a passthrough card (well 2). It only supported 4 drives (not 8), and (worst of all) it dropped throughput to 3Gbps. Unacceptable, even with SSDs.

So, I did give up the SAS drives though. I can fix them as @alexskysilk mentions, but since I'll be using an iSCSI connection to my QNAP for my most important data, the SAS drives aren't that big of a deal at all. I have 8 SSDs and I connected those to the 2008M-8i mezzanine card. It seems the JBOD mode on this card is indeed IT mode, or at least something very similar. I guess it isn't fully passthrough bc I can still see data about the disks in CIMC. However, I am 100% positive that the HW controller is not influencing anything by way of any disk I/O. It is presenting the drives directly to the OS. Sure, another failure point, but these cards are pretty cheap as they aren't the best HW RAID controllers anyhow, but do offer better speed and more channels than on-board support.