JBOD management

acron1234 · Jan 6, 2023

What's the best way to identify failed drive in JBOD in Proxmox?

I am using ZFS connect to external JBOD. I could not find a good way to align drives with JBOD slot. (For example, if I have a defective drive I could not see which slot this drive in the JBOD)

What's the best way to get this done?

Thank you!

alexskysilk · Jan 6, 2023

depends on your enclosure/hba combo.

Whats the make model of both?

acron1234 · Jan 6, 2023

HBA is Broadcom 9500-16e.
JBOD is Seagate Exos E 4U106
Server is Dell R740

Thank you

alexskysilk · Jan 6, 2023

in that case, you can use perccli to perform various operations including locating disks.

if the hba is dell branded-
https://www.dell.com/support/kbdoc/...ility-to-manage-your-raid-controller#bs_Linux

if not, it depends on the firmware (IR or IT.) megacli if IR, sas2ircu if IT. you can get em from broadcom's site, or use https://hwraid.le-vert.net/wiki/DebianPackages

acron1234 · Jan 7, 2023

Installed storcli64 from Broadcom. However, when try to run the command below it showed Failed to get enclosure information. Any suggestions?

./storcli64 /c0 /e109 show all
CLI Version = 007.2103.0000.0000 Dec 08, 2021
Operating system = Linux 5.15.74-1-pve
Controller = 0
Status = Failure
Description = Failed to get enclosure information

If I run ./storcli64 /c0 show. It showed drives but not the serial number. Still not able to identify drives when need to replace.

PD LIST :
=======

----------------------------------------------------------------------------
EID:Slt DID State DG Size Intf Med SED PI SeSz Model Sp
----------------------------------------------------------------------------
109:2 62 JBOD - 14.551 TB SATA HDD - - 512B ST16000NM000J-2TW103 -
109:3 63 JBOD - 14.551 TB SATA HDD - - 512B ST16000NM000J-2TW103 -
109:4 64 JBOD - 14.551 TB SATA HDD - - 512B ST16000NM000J-2TW103 -
109:5 65 JBOD - 14.551 TB SATA HDD - - 512B ST16000NM000J-2TW103 -
109:6 66 JBOD - 14.551 TB SATA HDD - - 512B ST16000NM001G-2KK103 -
109:7 67 JBOD - 14.551 TB SATA HDD - - 512B ST16000NM000J-2TW103 -
109:8 68 JBOD - 14.551 TB SATA HDD - - 512B ST16000NM000J-2TW103 -
109:9 69 JBOD - 14.551 TB SATA HDD - - 512B ST16000NM000J-2TW103 -
109:10 70 JBOD - 14.551 TB SATA HDD - - 512B ST16000NM000J-2TW103 -
109:11 71 JBOD - 14.551 TB SATA HDD - - 512B ST16000NM000J-2TW103 -
109:12 72 JBOD - 14.551 TB SATA HDD - - 512B ST16000NM001G-2KK103 -
109:13 73 JBOD - 14.551 TB SATA HDD - - 512B ST16000NM001G-2KK103 -
109:14 74 JBOD - 14.551 TB SATA HDD - - 512B ST16000NM001G-2KK103 -
109:15 75 JBOD - 14.551 TB SATA HDD - - 512B ST16000NM001G-2KK103 -

wolfspyre · Jan 7, 2023

acron1234 said:
What's the best way to identify failed drive in JBOD in Proxmox?

I am using ZFS connect to external JBOD. I could not find a good way to align drives with JBOD slot. (For example, if I have a defective drive I could not see which slot this drive in the JBOD)

What's the best way to get this done?

Thank you!

Howdy!
the most straightforward way I've found that works FAIRLY CONSISTENTLY is to cat the block device's contents to devnull... something like:

code_language.shell:

#!/usr/bin/env bash
DEAD_JIM=sdzzz
SLEEPTIME=1
BLOCKSIZE=4096
COUNT=1000

while true; do
  echo "making blinkylight"
  dd  bs=${BLOCKSIZE} count=${COUNT} if="/dev/${DEAD_JIM}" of=/dev/null
  sleep "${SLEEPTIME}"
done

this gives you a consistent flashy behavior on the disk.... adjust the timing and size such that it works in the env yer in wrt load, etc...

another thing I try to do is identify the disks with labels that make it easy(ier) to identify the specific physical object that the system is referring to by attributes of it.

tools like `lsscsi` really help too

Good luck!

alexskysilk · Jan 8, 2023

acron1234 said:
Installed storcli64 from Broadcom. However, when try to run the command below it showed Failed to get enclosure information. Any suggestions?

./storcli64 /c0 /e109 show all
CLI Version = 007.2103.0000.0000 Dec 08, 2021
Operating system = Linux 5.15.74-1-pve
Controller = 0
Status = Failure
Description = Failed to get enclosure information

If I run ./storcli64 /c0 show. It showed drives but not the serial number. Still not able to identify drives when need to replace.

PD LIST :
=======

----------------------------------------------------------------------------
EID:Slt DID State DG Size Intf Med SED PI SeSz Model Sp
----------------------------------------------------------------------------
109:2 62 JBOD - 14.551 TB SATA HDD - - 512B ST16000NM000J-2TW103 -
109:3 63 JBOD - 14.551 TB SATA HDD - - 512B ST16000NM000J-2TW103 -
109:4 64 JBOD - 14.551 TB SATA HDD - - 512B ST16000NM000J-2TW103 -
109:5 65 JBOD - 14.551 TB SATA HDD - - 512B ST16000NM000J-2TW103 -
109:6 66 JBOD - 14.551 TB SATA HDD - - 512B ST16000NM001G-2KK103 -
109:7 67 JBOD - 14.551 TB SATA HDD - - 512B ST16000NM000J-2TW103 -
109:8 68 JBOD - 14.551 TB SATA HDD - - 512B ST16000NM000J-2TW103 -
109:9 69 JBOD - 14.551 TB SATA HDD - - 512B ST16000NM000J-2TW103 -
109:10 70 JBOD - 14.551 TB SATA HDD - - 512B ST16000NM000J-2TW103 -
109:11 71 JBOD - 14.551 TB SATA HDD - - 512B ST16000NM000J-2TW103 -
109:12 72 JBOD - 14.551 TB SATA HDD - - 512B ST16000NM001G-2KK103 -
109:13 73 JBOD - 14.551 TB SATA HDD - - 512B ST16000NM001G-2KK103 -
109:14 74 JBOD - 14.551 TB SATA HDD - - 512B ST16000NM001G-2KK103 -
109:15 75 JBOD - 14.551 TB SATA HDD - - 512B ST16000NM001G-2KK103 -

I dont normally use storcli, but I believe the context should be
storcli /c0 /eAll /sAll show

following by locate start.

wolfspyre said:
the most straightforward way I've found that works FAIRLY CONSISTENTLY is to cat the block device's contents to devnull... something like:

this only works if the disk is alive...

acron1234 · Jan 8, 2023

alexskysilk said:
I dont normally use storcli, but I believe the context should be
storcli /c0 /eAll /sAll show

following by locate start.

this only works if the disk is alive...

c0 is for 9500, e109 is the Seagate JBOD. There seems issues to get info on the enclosure...

wolfspyre · Jan 8, 2023

`lsscsi` can also get slot info, serial number, etc... jus takes a lil finessing to get the flags right.

the challenge I have the the various broadcomm/lsi tools is their syntax is so obtuse... `s kinda always been that way... it doesn't make sense to my head (personally)

¯\_(ツ)_/¯

alexskysilk · Jan 8, 2023

wolfspyre said:
the challenge I have the the various broadcomm/lsi tools is their syntax is so obtuse... `s kinda always been that way... it doesn't make sense to my head (personally)

you're not kidding. I've been dealing with LSI controllers for 15 years and I still have to google the syntax...

acron1234 · Jan 9, 2023

wolfspyre said:
`lsscsi` can also get slot info, serial number, etc... jus takes a lil finessing to get the flags right.

the challenge I have the the various broadcomm/lsi tools is their syntax is so obtuse... `s kinda always been that way... it doesn't make sense to my head (personally)

¯\_(ツ)_/¯

mind to share syntax on lsscsi?

alexskysilk · Jan 9, 2023

I dont think you can use lsscsi by ITSELF, but it will give you the SCSI bus ID and drive letter. you can match the drive letter to the WWN using ~~blkid~~ /dev/disk/by-id to complete.

LASTLY, the bus id is NOT guaranteed to match the position in the chassis, as many chassis vendors dont match physical and logical slots numbers (eg, ses reports slot #1 but physically its slot #24, etc.)

Search

Search

JBOD management

acron1234

New Member

alexskysilk

Distinguished Member

acron1234

New Member

alexskysilk

Distinguished Member

acron1234

New Member

wolfspyre

New Member

alexskysilk

Distinguished Member

acron1234

New Member

wolfspyre

New Member

alexskysilk

Distinguished Member

acron1234

New Member

alexskysilk

Distinguished Member