Hello,
since Kernel 5.15.64 (including 5.15.74) our FC HBA stopped working and doesn't see any LUNs.
Used hardware:
Logs:
We think this issue might have got fixed in upstream Linux Kernel 5.15.77, but we didn't try it out.
since Kernel 5.15.64 (including 5.15.74) our FC HBA stopped working and doesn't see any LUNs.
Used hardware:
05:00.0 Fibre Channel: Emulex Corporation LPe31000/LPe32000 Series 16Gb/32Gb Fibre Channel Adapter (rev 01)
05:00.1 Fibre Channel: Emulex Corporation LPe31000/LPe32000 Series 16Gb/32Gb Fibre Channel Adapter (rev 01)
Logs:
2022-11-25T12:03:45.584623+01:00 pm02-vh03 kernel: [ 3.815233] lpfc 0000:05:00.0: 0:2574 IO channels: hdwQ 24 IRQ 24 MRQ: 0
2022-11-25T12:03:45.584645+01:00 pm02-vh03 kernel: [ 3.999183] lpfc 0000:05:00.0: 0:6468 Set host date / time: Status x0:
2022-11-25T12:03:45.584649+01:00 pm02-vh03 kernel: [ 4.043420] lpfc 0000:05:00.0: 0:6448 Dual Dump is enabled
2022-11-25T12:03:45.584682+01:00 pm02-vh03 kernel: [ 4.679159] lpfc 0000:05:00.0: 0:3176 Port Name 0 Physical Link is functional
2022-11-25T12:03:45.584686+01:00 pm02-vh03 kernel: [ 4.867224] lpfc 0000:05:00.1: 1:2574 IO channels: hdwQ 24 IRQ 24 MRQ: 0
2022-11-25T12:03:45.584687+01:00 pm02-vh03 kernel: [ 5.051208] lpfc 0000:05:00.1: 1:6468 Set host date / time: Status x0:
2022-11-25T12:03:45.584687+01:00 pm02-vh03 kernel: [ 5.095243] lpfc 0000:05:00.1: 1:6448 Dual Dump is enabled
2022-11-25T12:03:45.584691+01:00 pm02-vh03 kernel: [ 5.708539] lpfc 0000:05:00.0: 0:1303 Link Up Event x1 received Data: x1 x0 x80 x0 x0 x0 0
2022-11-25T12:03:45.584692+01:00 pm02-vh03 kernel: [ 5.709202] lpfc 0000:05:00.0: start 198 end 197 cnt 256
2022-11-25T12:03:45.584699+01:00 pm02-vh03 kernel: [ 5.709204] lpfc 0000:05:00.0: 198: [ 4.088958] 0:(0):0356 Mailbox cmd x9b (x1/xd) Status x0 Data: x1 x64 x0 x0 x0 x10d x0 x4 x4 x7 x20000000 x8000000 x0 CQ: x0 x0 x0 x90000000
2022-11-25T12:03:45.584700+01:00 pm02-vh03 kernel: [ 5.709206] lpfc 0000:05:00.0: 199: [ 4.088959] 0:2584 HBA EQ setup: queue[7]-id=7
2022-11-25T12:03:45.584700+01:00 pm02-vh03 kernel: [ 5.709208] lpfc 0000:05:00.0: 200: [ 4.092977] 0:(0):0356 Mailbox cmd x9b (x1/xd) Status x0 Data: x1 x64 x0 x0 x0 x10d x0 x4 x4 x8 x20000000 x8000000 x0 CQ: x0 x0 x0 x90000000
2022-11-25T12:03:45.584701+01:00 pm02-vh03 kernel: [ 5.709209] lpfc 0000:05:00.0: 201: [ 4.092977] 0:2584 HBA EQ setup: queue[8]-id=8
2022-11-25T12:03:45.584704+01:00 pm02-vh03 kernel: [ 5.709211] lpfc 0000:05:00.0: 202: [ 4.096995] 0:(0):0356 Mailbox cmd x9b (x1/xd) Status x0 Data: x1 x64 x0 x0 x0 x10d x0 x4 x4 x9 x20000000 x8000000 x0 CQ: x0 x0 x0 x90000000
2022-11-25T12:03:45.584704+01:00 pm02-vh03 kernel: [ 5.709213] lpfc 0000:05:00.0: 203: [ 4.096995] 0:2584 HBA EQ setup: queue[9]-id=9
2022-11-25T12:03:45.584705+01:00 pm02-vh03 kernel: [ 5.709214] lpfc 0000:05:00.0: 204: [ 4.101012] 0:(0):0356 Mailbox cmd x9b (x1/xd) Status x0 Data: x1 x64 x0 x0 x0 x10d x0 x4 x4 xa x20000000 x8000000 x0 CQ: x0 x0 x0 x90000000
2022-11-25T12:03:45.584705+01:00 pm02-vh03 kernel: [ 5.709215] lpfc 0000:05:00.0: 205: [ 4.101013] 0:2584 HBA EQ setup: queue[10]-id=10
2022-11-25T12:03:45.584705+01:00 pm02-vh03 kernel: [ 5.709217] lpfc 0000:05:00.0: 206: [ 4.105033] 0:(0):0356 Mailbox cmd x9b (x1/xd) Status x0 Data: x1 x64 x0 x0 x0 x10d x0 x4 x4 xb x20000000 x8000000 x0 CQ: x0 x0 x0 x90000000
2022-11-25T12:03:45.584706+01:00 pm02-vh03 kernel: [ 5.709218] lpfc 0000:05:00.0: 207: [ 4.105034] 0:2584 HBA EQ setup: queue[11]-id=11
2022-11-25T12:03:45.584709+01:00 pm02-vh03 kernel: [ 5.709219] lpfc 0000:05:00.0: 208: [ 4.109052] 0:(0):0356 Mailbox cmd x9b (x1/xd) Status x0 Data: x1 x64 x0 x0 x0 x10d x0 x4 x4 xc x20000000 x8000000 x0 CQ: x0 x0 x0 x90000000
2022-11-25T12:03:45.584709+01:00 pm02-vh03 kernel: [ 5.709220] lpfc 0000:05:00.0: 209: [ 4.109053] 0:2584 HBA EQ setup: queue[12]-id=12
2022-11-25T12:03:45.584710+01:00 pm02-vh03 kernel: [ 5.709221] lpfc 0000:05:00.0: 210: [ 4.113084] 0:(0):0356 Mailbox cmd x9b (x1/xd) Status x0 Data: x1 x64 x0 x0 x0 x10d x0 x4 x4 xd x20000000 x8000000 x0 CQ: x0 x0 x0 x90000000
2022-11-25T12:03:45.584710+01:00 pm02-vh03 kernel: [ 5.709223] lpfc 0000:05:00.0: 211: [ 4.113085] 0:2584 HBA EQ setup: queue[13]-id=13
2022-11-25T12:03:45.584719+01:00 pm02-vh03 kernel: [ 5.709224] lpfc 0000:05:00.0: 212: [ 4.117201] 0:(0):0356 Mailbox cmd x9b (x1/xd) Status x0 Data: x1 x64 x0 x0 x0 x10d x0 x4 x4 xe x20000000 x8000000 x0 CQ: x0 x0 x0 x90000000
2022-11-25T12:03:45.584722+01:00 pm02-vh03 kernel: [ 5.709225] lpfc 0000:05:00.0: 213: [ 4.117202] 0:2584 HBA EQ setup: queue[14]-id=14
2022-11-25T12:03:45.584722+01:00 pm02-vh03 kernel: [ 5.709226] lpfc 0000:05:00.0: 214: [ 4.121221] 0:(0):0356 Mailbox cmd x9b (x1/xd) Status x0 Data: x1 x64 x0 x0 x0 x10d x0 x4 x4 xf x20000000 x8000000 x0 CQ: x0 x0 x0 x90000000
2022-11-25T12:03:45.584722+01:00 pm02-vh03 kernel: [ 5.709227] lpfc 0000:05:00.0: 215: [ 4.121224] 0:2584 HBA EQ setup: queue[15]-id=15
2022-11-25T12:03:45.584723+01:00 pm02-vh03 kernel: [ 5.709229] lpfc 0000:05:00.0: 216: [ 4.125243] 0:(0):0356 Mailbox cmd x9b (x1/xd) Status x0 Data: x1 x64 x0 x0 x0 x10d x0 x4 x4 x10 x20000000 x8000000 x0 CQ: x0 x0 x0 x90000000
2022-11-25T12:03:45.584723+01:00 pm02-vh03 kernel: [ 5.709230] lpfc 0000:05:00.0: 217: [ 4.125243] 0:2584 HBA EQ setup: queue[16]-id=16
2022-11-25T12:03:45.584724+01:00 pm02-vh03 kernel: [ 5.709231] lpfc 0000:05:00.0: 218: [ 4.129503] 0:(0):0356 Mailbox cmd x9b (x1/xd) Status x0 Data: x1 x64 x0 x0 x0 x10d x0 x4 x4 x11 x20000000 x8000000 x0 CQ: x0 x0 x0 x90000000
2022-11-25T12:03:45.584727+01:00 pm02-vh03 kernel: [ 5.709232] lpfc 0000:05:00.0: 219: [ 4.129504] 0:2584 HBA EQ setup: queue[17]-id=17
2022-11-25T12:03:45.584727+01:00 pm02-vh03 kernel: [ 5.709233] lpfc 0000:05:00.0: 220: [ 4.133534] 0:(0):0356 Mailbox cmd x9b (x1/xd) Status x0 Data: x1 x64 x0 x0 x0 x10d x0 x4 x4 x12 x20000000 x8000000 x0 CQ: x0 x0 x0 x90000000
2022-11-25T12:03:45.584728+01:00 pm02-vh03 kernel: [ 5.709235] lpfc 0000:05:00.0: 221: [ 4.133535] 0:2584 HBA EQ setup: queue[18]-id=18
2022-11-25T12:03:45.584728+01:00 pm02-vh03 kernel: [ 5.709236] lpfc 0000:05:00.0: 222: [ 4.137554] 0:(0):0356 Mailbox cmd x9b (x1/xd) Status x0 Data: x1 x64 x0 x0 x0 x10d x0 x4 x4 x13 x20000000 x8000000 x0 CQ: x0 x0 x0 x90000000
2022-11-25T12:03:45.584728+01:00 pm02-vh03 kernel: [ 5.709237] lpfc 0000:05:00.0: 223: [ 4.137554] 0:2584 HBA EQ setup: queue[19]-id=19
2022-11-25T12:03:45.584731+01:00 pm02-vh03 kernel: [ 5.709238] lpfc 0000:05:00.0: 224: [ 4.141572] 0:(0):0356 Mailbox cmd x9b (x1/xd) Status x0 Data: x1 x64 x0 x0 x0 x10d x0 x4 x4 x14 x20000000 x8000000 x0 CQ: x0 x0 x0 x90000000
2022-11-25T12:03:45.584732+01:00 pm02-vh03 kernel: [ 5.709239] lpfc 0000:05:00.0: 225: [ 4.141573] 0:2584 HBA EQ setup: queue[20]-id=20
2022-11-25T12:03:45.584740+01:00 pm02-vh03 kernel: [ 5.709241] lpfc 0000:05:00.0: 226: [ 4.145593] 0:(0):0356 Mailbox cmd x9b (x1/xd) Status x0 Data: x1 x64 x0 x0 x0 x10d x0 x4 x4 x15 x20000000 x8000000 x0 CQ: x0 x0 x0 x90000000
2022-11-25T12:03:45.584740+01:00 pm02-vh03 kernel: [ 5.709242] lpfc 0000:05:00.0: 227: [ 4.145594] 0:2584 HBA EQ setup: queue[21]-id=21
2022-11-25T12:03:45.584741+01:00 pm02-vh03 kernel: [ 5.709244] lpfc 0000:05:00.0: 228: [ 4.149614] 0:(0):0356 Mailbox cmd x9b (x1/xd) Status x0 Data: x1 x64 x0 x0 x0 x10d x0 x4 x4 x16 x20000000 x8000000 x0 CQ: x0 x0 x0 x90000000
2022-11-25T12:03:45.584741+01:00 pm02-vh03 kernel: [ 5.709245] lpfc 0000:05:00.0: 229: [ 4.149615] 0:2584 HBA EQ setup: queue[22]-id=22
2022-11-25T12:03:45.584744+01:00 pm02-vh03 kernel: [ 5.709246] lpfc 0000:05:00.0: 230: [ 4.153643] 0:(0):0356 Mailbox cmd x9b (x1/xd) Status x0 Data: x1 x64 x0 x0 x0 x10d x0 x4 x4 x17 x20000000 x8000000 x0 CQ: x0 x0 x0 x90000000
2022-11-25T12:03:45.584744+01:00 pm02-vh03 kernel: [ 5.709247] lpfc 0000:05:00.0: 231: [ 4.153644] 0:2584 HBA EQ setup: queue[23]-id=23
2022-11-25T12:03:45.584745+01:00 pm02-vh03 kernel: [ 5.709249] lpfc 0000:05:00.0: 232: [ 4.157667] 0:(0):0356 Mailbox cmd x9b (x1/xc) Status x0 Data: x1 x44 x0 x0 x0 x10c x0 x4 x4 x232 xb8000000 x0 x1000 CQ: x0 x0 x0 x90000000
2022-11-25T12:03:45.584745+01:00 pm02-vh03 kernel: [ 5.709250] lpfc 0000:05:00.0: 233: [ 4.157668] 0:6087 CQ setup: cq[0]-id=562, parent eq[0]-id=0
2022-11-25T12:03:45.584746+01:00 pm02-vh03 kernel: [ 5.709251] lpfc 0000:05:00.0: 234: [ 4.161696] 0:(0):0356 Mailbox cmd x9b (xc/x1) Status x0 Data: x1 x54 x0 x0 x0 xc01 x0 x4 x4 x2320000 x4008604 x0 x1d9f8000 CQ: x0 x0 x0 x90000000
2022-11-25T12:03:45.584749+01:00 pm02-vh03 kernel: [ 5.709252] lpfc 0000:05:00.0: 235: [ 4.161697] 0:2593 WQ setup: wq[0]-id=0 assoc=562, cq[0]-id=562
2022-11-25T12:03:45.584749+01:00 pm02-vh03 kernel: [ 5.709253] lpfc 0000:05:00.0: 236: [ 4.165720] 0:(0):0356 Mailbox cmd x9b (x1/xc) Status x0 Data: x1 x44 x0 x0 x0 x10c x0 x4 x4 x233 xb8000000 x1 x1000 CQ: x0 x0 x0 x90000000
2022-11-25T12:03:45.584749+01:00 pm02-vh03 kernel: [ 5.709254] lpfc 0000:05:00.0: 237: [ 4.165721] 0:6087 CQ setup: cq[1]-id=563, parent eq[1]-id=1
2022-11-25T12:03:45.584750+01:00 pm02-vh03 kernel: [ 5.709255] lpfc 0000:05:00.0: 238: [ 4.169748] 0:(0):0356 Mailbox cmd x9b (xc/x1) Status x0 Data: x1 x54 x0 x0 x0 xc01 x0 x4 x4 x2330001 x4008604 x0 x1da28000 CQ: x0 x0 x0 x90000000
2022-11-25T12:03:45.584750+01:00 pm02-vh03 kernel: [ 5.709256] lpfc 0000:05:00.0: 239: [ 4.169750] 0:2593 WQ setup: wq[1]-id=1 assoc=563, cq[1]-id=563
2022-11-25T12:03:45.584760+01:00 pm02-vh03 kernel: [ 5.709257] lpfc 0000:05:00.0: 240: [ 4.173773] 0:(0):0356 Mailbox cmd x9b (x1/xc) Status x0 Data: x1 x44 x0 x0 x0 x10c x0 x4 x4 x234 xb8000000 x2 x1000 CQ: x0 x0 x0 x90000000
2022-11-25T12:03:45.584761+01:00 pm02-vh03 kernel: [ 5.709259] lpfc 0000:05:00.0: 241: [ 4.173773] 0:6087 CQ setup: cq[2]-id=564, parent eq[2]-id=2
2022-11-25T12:03:45.584761+01:00 pm02-vh03 kernel: [ 5.709260] lpfc 0000:05:00.0: 242: [ 4.177800] 0:(0):0356 Mailbox cmd x9b (xc/x1) Status x0 Data: x1 x54 x0 x0 x0 xc01 x0 x4 x4 x2340002 x4008604 x0 x1da58000 CQ: x0 x0 x0 x90000000
2022-11-25T12:03:45.584994+01:00 pm02-vh03 kernel: [ 5.709445] lpfc 0000:05:00.0: 144: [ 4.648640] 0:2830 This device does not support Advanced Error Reporting (AER)
2022-11-25T12:03:45.584994+01:00 pm02-vh03 kernel: [ 5.709446] lpfc 0000:05:00.0: 145: [ 4.648649] 0:(0):2542 Try to issue mailbox command x9b (x1/xb5) synchronously ahead of async mailbox command queue: xc200 x1
2022-11-25T12:03:45.584995+01:00 pm02-vh03 kernel: [ 5.709447] lpfc 0000:05:00.0: 146: [ 4.652666] 0:(0):0356 Mailbox cmd x9b (x1/xb5) Status x0 Data: x1 x80 x0 x0 x0 x1b5 x0 x5c x5c x3 x22c40 x1010108 x1f0 CQ: x0 x0 x0 x90000000
2022-11-25T12:03:45.584995+01:00 pm02-vh03 kernel: [ 5.709448] lpfc 0000:05:00.0: 147: [ 4.652671] 0:(0):2542 Try to issue mailbox command x9b (x1/xbf) synchronously ahead of async mailbox command queue: xc200 x1
2022-11-25T12:03:45.584996+01:00 pm02-vh03 kernel: [ 5.709449] lpfc 0000:05:00.0: 148: [ 4.656686] 0:(0):0356 Mailbox cmd x9b (x1/xbf) Status x1 Data: x1 x24 x0 x0 x0 x1bf x44 x0 x0 x21 x4 x400002 x0 CQ: x44 x0 x0 x90000000
2022-11-25T12:03:45.584999+01:00 pm02-vh03 kernel: [ 5.709450] lpfc 0000:05:00.0: 149: [ 4.656689] 0:(0):2597 Sync Mailbox command x9b (x1/xbf) failure: mqe_sta: x1 mcqe_sta: x44/x0 Data: xe200 x1
2022-11-25T12:03:45.584999+01:00 pm02-vh03 kernel: [ 5.709452] lpfc 0000:05:00.0: 150: [ 4.656689] 0:6245 Enable MI Mailbox x9b (x1/xbf) failed, rc:x10 mi:x2
2022-11-25T12:03:45.584999+01:00 pm02-vh03 kernel: [ 5.709453] lpfc 0000:05:00.0: 151: [ 4.656690] 0:6220 CMF is disabled
2022-11-25T12:03:45.585000+01:00 pm02-vh03 kernel: [ 5.709454] lpfc 0000:05:00.0: 152: [ 4.656690] 0:6470 Setup MI version 2 CMF 0 mode 0
2022-11-25T12:03:45.585000+01:00 pm02-vh03 kernel: [ 5.709455] lpfc 0000:05:00.0: 153: [ 4.656706] 0:(16777215):0354 Mbox cmd issue - Enqueue Data: x5 (x0/x0) x0 xc200 x2
2022-11-25T12:03:45.585001+01:00 pm02-vh03 kernel: [ 5.709455] lpfc 0000:05:00.0: 154: [ 4.656707] 0:0480 Enabled MSI-X interrupt mode.
2022-11-25T12:03:45.585004+01:00 pm02-vh03 kernel: [ 5.709456] lpfc 0000:05:00.0: 155: [ 4.656710] 0:0428 Perform SCSI scan
We think this issue might have got fixed in upstream Linux Kernel 5.15.77, but we didn't try it out.
James Smart (6):
Revert "scsi: lpfc: Resolve some cleanup issues following SLI path refactoring"
Revert "scsi: lpfc: Fix element offset in __lpfc_sli_release_iocbq_s4()"
Revert "scsi: lpfc: Fix locking for lpfc_sli_iocbq_lookup()"
Revert "scsi: lpfc: SLI path split: Refactor SCSI paths"
Revert "scsi: lpfc: SLI path split: Refactor fast and slow paths to native SLI4"
Revert "scsi: lpfc: SLI path split: Refactor lpfc_iocbq"