C6320, LSI2008 HBA and disk order

[MMartinez]

Hello,

I've recently set up a C6230 with an LSI2008 Controller and doing some performance tests I've seen that it performs worse than older C6220 with LSI9265-8i controller.

Now we are doing some tests changing some SAS 10k disks by SSD SATA disk and we have problems with the disk order. We have two issues:

1) When we want to replace one disk, for example /dev/sdf, we supose that it is the 6 disk in the bay, but this is not always true, in some cases it isn't the six one.
2) If we insert the drive again, it gets a new device file -> /dev/sdq for example. We are using Ceph and that causes a problem, it makes imposible to start that OSD from the GUI, it simply fails. One quick solution is just restart the system, but it's not a good practice on a production system.

I can identify the bay position in two different ways:

• by using ledmon package (with "ledctl locate=/dev/sdf" and later "ledctl locate_off=/dev/sdf"),
• or using the command "cat /sys/block/sdf/../../../../sas_device/end_device-*/bay_identifier" to identify the bay number

Being able to locate the disks it makes a bit easier to work with this controller but it would be better for me to have the disks in order. I mean the first disk (on the first bay) associated with the /dev/sda device, the second one with /dev/sdb, the third one with /dev/sdc and so on. Also it would we good that after inserting again a disk on a bay it gets the same device file.

Maybe both issues could be resolved by a udev script, but I have no experience on that. Any idea?

Regards,

Manuel Martínez

 

[Tallguy]

Sorry I can't help with your config question but I wondered if you sorted out the performance issues and is the C6320 + LSI2008 performing better than the older system? Are the LSI2008 the mezzanine cards? I am using C6320 and was wondering if the LSI2008 is worthy of consideration and then noticed your post mentioning performance issues.

 

[jdancer]

I do know when flashing a Dell HBA to IT mode, it changes the order of the disks per https://forums.servethehome.com/ind...full-size-to-it-mode.27459/page-2#post-255082 and https://forums.servethehome.com/ind...full-size-to-it-mode.27459/page-3#post-255098

So, when a Dell HBA is running Dell firmware, the firmware somehow translates that drives in the correct drive bay order.

 

[leesteken]

/dev/sd* is indeed unstable in it's drive mapping. Maybe you can use ls -l /dev/disk/by-path/ to find out which /dev/sd* corresponds to which drive? I have used it to find out which port of which SATA or USB controller was used for a drive.

 

[MMartinez]

Sorry I can't help with your config question but I wondered if you sorted out the performance issues and is the C6320 + LSI2008 performing better than the older system? Are the LSI2008 the mezzanine cards? I am using C6320 and was wondering if the LSI2008 is worthy of consideration and then noticed your post mentioning performance issues.

No, I didn't. We decided to change it for a C6420 with a PERC H330. It performs better than C6320 and in most cases better than the C6220. The C6220 still outperforms the C6420 in write cache, as it has a LSI9265 with write cache and the H330 on the C6420 does not have write cache. We solve this problem by activating the write back (unsafe) cache on the VM that requires better performance.
On the C6420 we have no problem with the disk order as we had with the C6320.
Yes, the LSI2008 were mezzazine cards.
Hope it helps.