Proxmox 6.2 LPFC error port type wrong

[Oliver Polterauer]

We did today on one of your test clusters an upgrade from 6.1 (running 5.3.18-3-pve) to 6.2 (5.4.34-1-pve) and after that the port_type for the lpfc drivers is not working anymore. Booting with the old kernel one gets FC Connection back online.


Version 5.3.18-3-pve

modinfo:
description: Emulex LightPulse Fibre Channel SCSI driver 12.2.0.3

port_type = "NPort (fabric via point-to-point)"
port_type = "NPort (fabric via point-to-point)"
Status: Online

Version 5.4.34-1-pve

modinfo:
description: Emulex LightPulse Fibre Channel SCSI driver 12.6.0.4

port_type = "Point-To-Point (direct nport connection)"
port_type = "Point-To-Point (direct nport connection)"
Status: Bypassed

We tried to set the port-selection within FC Bios but did not help.

Firmewareversion of Controller: FV2.10X6

 

[t.lamprecht]

Do you have any suspicious message from the SCSI or LPFC subsystem in the kernel log, dmesg?

What's the topology set to the module: cat /sys/module/lpfc/parameters/lpfc_topology

Also, if you can it could be worth to try out the 5.4.27-1-pve kernel version, as after that one a huge bunch of lpfc changes got into the kernel with an older FW version:
apt install pve-kernel-5.4.27-1-pve

 

[Oliver Polterauer]

Do you have any suspicious message from the SCSI or LPFC subsystem in the kernel log, dmesg?

What's the topology set to the module: cat /sys/module/lpfc/parameters/lpfc_topology

Also, if you can it could be worth to try out the 5.4.27-1-pve kernel version, as after that one a huge bunch of lpfc changes got into the kernel with an older FW version:
apt install pve-kernel-5.4.27-1-pve

these are the errors from log file:

2020-05-12T14:40:44.780036+02:00 hostname kernel: [    8.701385] lpfc 0000:07:00.0: 0:1303 Link Up Event x1 received Data: x1 xf7 x20 x0 x0 x0 0
2020-05-12T14:40:44.780052+02:00 hostname kernel: [   11.177527] lpfc 0000:07:00.1: 1:1303 Link Up Event x1 received Data: x1 xf7 x20 x0 x0 x0 0
2020-05-12T14:40:56.138424+02:00 hostname kernel: [   28.779567] lpfc 0000:07:00.0: 0:(0):2858 FLOGI failure Status:x3/x2 TMO:x10 Data x101000 x0
2020-05-12T14:40:58.618409+02:00 hostname kernel: [   31.259404] lpfc 0000:07:00.1: 1:(0):2858 FLOGI failure Status:x3/x2 TMO:x10 Data x101000 x0
2020-05-12T14:41:16.202437+02:00 hostname kernel: [   48.845746] lpfc 0000:07:00.0: 0:(0):2858 FLOGI failure Status:x3/x2 TMO:x10 Data x101000 x0
2020-05-12T14:41:18.702449+02:00 hostname kernel: [   51.345606] lpfc 0000:07:00.1: 1:(0):2858 FLOGI failure Status:x3/x2 TMO:x10 Data x101000 x0
2020-05-12T14:41:36.266434+02:00 hostname kernel: [   68.910887] lpfc 0000:07:00.0: 0:(0):2858 FLOGI failure Status:x3/x2 TMO:x10 Data x101000 x0
2020-05-12T14:41:38.786384+02:00 hostname kernel: [   71.430776] lpfc 0000:07:00.1: 1:(0):2858 FLOGI failure Status:x3/x2 TMO:x10 Data x101000 x0
2020-05-12T14:41:56.374439+02:00 hostname kernel: [   89.018037] lpfc 0000:07:00.0: 0:(0):2858 FLOGI failure Status:x3/x2 TMO:x10 Data x101000 x0
2020-05-12T14:41:58.870415+02:00 hostname kernel: [   91.513955] lpfc 0000:07:00.1: 1:(0):2858 FLOGI failure Status:x3/x2 TMO:x10 Data x101000 x0
2020-05-12T14:42:16.021077+02:00 hostname kernel: [  109.081209] lpfc 0000:07:00.0: 0:(0):2858 FLOGI failure Status:x3/x2 TMO:x10 Data x101000 x0
2020-05-12T14:42:18.556997+02:00 hostname kernel: [  111.620017] lpfc 0000:07:00.1: 1:(0):2858 FLOGI failure Status:x3/x2 TMO:x10 Data x101000 x0
2020-05-12T14:42:36.081087+02:00 hostname kernel: [  129.141345] lpfc 0000:07:00.0: 0:(0):2858 FLOGI failure Status:x3/x2 TMO:x10 Data x101000 x0
2020-05-12T14:42:38.656995+02:00 hostname kernel: [  131.720169] lpfc 0000:07:00.1: 1:(0):2858 FLOGI failure Status:x3/x2 TMO:x10 Data x101000 x0
2020-05-12T14:42:56.225058+02:00 hostname kernel: [  149.287507] lpfc 0000:07:00.0: 0:(0):2858 FLOGI failure Status:x3/x2 TMO:x10 Data x101000 x0
2020-05-12T14:42:58.725043+02:00 hostname kernel: [  151.789462] lpfc 0000:07:00.1: 1:(0):2858 FLOGI failure Status:x3/x2 TMO:x10 Data x101000 x0
2020-05-12T14:43:16.281043+02:00 hostname kernel: [  169.342679] lpfc 0000:07:00.0: 0:(0):2858 FLOGI failure Status:x3/x2 TMO:x10 Data x101000 x0
2020-05-12T14:43:18.789021+02:00 hostname kernel: [  171.853541] lpfc 0000:07:00.1: 1:(0):2858 FLOGI failure Status:x3/x2 TMO:x10 Data x101000 x0
2020-05-12T14:43:36.329078+02:00 hostname kernel: [  189.391874] lpfc 0000:07:00.0: 0:(0):2858 FLOGI failure Status:x3/x2 TMO:x10 Data x101000 x0
2020-05-12T14:43:38.837062+02:00 hostname kernel: [  191.898749] lpfc 0000:07:00.1: 1:(0):2858 FLOGI failure Status:x3/x2 TMO:x10 Data x101000 x0
2020-05-12T14:43:56.385045+02:00 hostname kernel: [  209.448035] lpfc 0000:07:00.0: 0:(0):2858 FLOGI failure Status:x3/x2 TMO:x10 Data x101000 x0
2020-05-12T14:43:58.901065+02:00 hostname kernel: [  211.962912] lpfc 0000:07:00.1: 1:(0):2858 FLOGI failure Status:x3/x2 TMO:x10 Data x101000 x0
2020-05-12T14:44:16.453033+02:00 hostname kernel: [  229.518206] lpfc 0000:07:00.0: 0:(0):2858 FLOGI failure Status:x3/x2 TMO:x10 Data x101000 x0
2020-05-12T14:44:18.965031+02:00 hostname kernel: [  232.030086] lpfc 0000:07:00.1: 1:(0):2858 FLOGI failure Status:x3/x2 TMO:x10 Data x101000 x0
2020-05-12T14:44:36.525087+02:00 hostname kernel: [  249.588380] lpfc 0000:07:00.0: 0:(0):2858 FLOGI failure Status:x3/x2 TMO:x10 Data x101000 x0
2020-05-12T14:44:39.021014+02:00 hostname kernel: [  252.084245] lpfc 0000:07:00.1: 1:(0):2858 FLOGI failure Status:x3/x2 TMO:x10 Data x101000 x0
2020-05-12T14:44:56.573038+02:00 hostname kernel: [  269.637536] lpfc 0000:07:00.0: 0:(0):2858 FLOGI failure Status:x3/x2 TMO:x10 Data x101000 x0
2020-05-12T14:44:59.101022+02:00 hostname kernel: [  272.167392] lpfc 0000:07:00.1: 1:(0):2858 FLOGI failure Status:x3/x2 TMO:x10 Data x101000 x0
2020-05-12T14:45:16.701075+02:00 hostname kernel: [  289.765704] lpfc 0000:07:00.0: 0:(0):2858 FLOGI failure Status:x3/x2 TMO:x10 Data x101000 x0
2020-05-12T14:45:19.185064+02:00 hostname kernel: [  292.249601] lpfc 0000:07:00.1: 1:(0):2858 FLOGI failure Status:x3/x2 TMO:x10 Data x101000 x0
2020-05-12T14:45:36.740988+02:00 hostname kernel: [  309.806804] lpfc 0000:07:00.0: 0:(0):2858 FLOGI failure Status:x3/x2 TMO:x10 Data x101000 x0
2020-05-12T14:45:39.261020+02:00 hostname kernel: [  312.327781] lpfc 0000:07:00.1: 1:(0):2858 FLOGI failure Status:x3/x2 TMO:x10 Data x101000 x0
2020-05-12T14:45:56.809080+02:00 hostname kernel: [  329.874036] lpfc 0000:07:00.0: 0:(0):2858 FLOGI failure Status:x3/x2 TMO:x10 Data x101000 x0
2020-05-12T14:45:59.325061+02:00 hostname kernel: [  332.389947] lpfc 0000:07:00.1: 1:(0):2858 FLOGI failure Status:x3/x2 TMO:x10 Data x101000 x0
2020-05-12T14:46:16.857050+02:00 hostname kernel: [  349.923182] lpfc 0000:07:00.0: 0:(0):2858 FLOGI failure Status:x3/x2 TMO:x10 Data x101000 x0
2020-05-12T14:46:19.385044+02:00 hostname kernel: [  352.450112] lpfc 0000:07:00.1: 1:(0):2858 FLOGI failure Status:x3/x2 TMO:x10 Data x101000 x0
2020-05-12T14:46:36.945036+02:00 hostname kernel: [  370.011364] lpfc 0000:07:00.0: 0:(0):2858 FLOGI failure Status:x3/x2 TMO:x10 Data x101000 x0
2020-05-12T14:46:39.480988+02:00 hostname kernel: [  372.549235] lpfc 0000:07:00.1: 1:(0):2858 FLOGI failure Status:x3/x2 TMO:x10 Data x101000 x0
2020-05-12T14:46:57.053043+02:00 hostname kernel: [  390.119547] lpfc 0000:07:00.0: 0:(0):2858 FLOGI failure Status:x3/x2 TMO:x10 Data x101000 x0
2020-05-12T14:46:59.641044+02:00 hostname kernel: [  392.707463] lpfc 0000:07:00.1: 1:(0):2858 FLOGI failure Status:x3/x2 TMO:x10 Data x101000 x0
2020-05-12T14:47:17.109054+02:00 hostname kernel: [  410.174721] lpfc 0000:07:00.0: 0:(0):2858 FLOGI failure Status:x3/x2 TMO:x10 Data x101000 x0
2020-05-12T14:47:19.693010+02:00 hostname kernel: [  412.760620] lpfc 0000:07:00.1: 1:(0):2858 FLOGI failure Status:x3/x2 TMO:x10 Data x101000 x0
2020-05-12T14:47:37.253022+02:00 hostname kernel: [  430.318861] lpfc 0000:07:00.0: 0:(0):2858 FLOGI failure Status:x3/x2 TMO:x10 Data x101000 x0
2020-05-12T14:47:39.749040+02:00 hostname kernel: [  432.817806] lpfc 0000:07:00.1: 1:(0):2858 FLOGI failure Status:x3/x2 TMO:x10 Data x101000 x0
2020-05-12T14:47:54.341031+02:00 hostname kernel: [  447.407880] lpfc 0000:07:00.0: 0:(0):0237 Pending Link Event during Discovery: State x7
2020-05-12T14:47:54.341460+02:00 hostname kernel: [  447.409116] lpfc 0000:07:00.0: 0:1305 Link Down Event x2 received Data: x2 x7 x98014 x0 x0
2020-05-12T14:47:54.409045+02:00 hostname kernel: [  447.476795] lpfc 0000:07:00.0: 0:1303 Link Up Event x3 received Data: x3 x0 x20 x0 x0 x0 0
2020-05-12T14:47:59.849052+02:00 hostname kernel: [  452.916982] lpfc 0000:07:00.1: 1:(0):2858 FLOGI failure Status:x3/x2 TMO:x10 Data x101000 x0
2020-05-12T14:48:13.541042+02:00 hostname kernel: [  466.607427] lpfc 0000:07:00.0: 0:(0):2858 FLOGI failure Status:x3/x2 TMO:x10 Data x101000 x0
2020-05-12T14:48:19.945034+02:00 hostname kernel: [  473.014108] lpfc 0000:07:00.1: 1:(0):2858 FLOGI failure Status:x3/x2 TMO:x10 Data x101000 x0
2020-05-12T14:48:33.609047+02:00 hostname kernel: [  486.676600] lpfc 0000:07:00.0: 0:(0):2858 FLOGI failure Status:x3/x2 TMO:x10 Data x101000 x0
2020-05-12T14:48:40.029050+02:00 hostname kernel: [  493.097325] lpfc 0000:07:00.1: 1:(0):2858 FLOGI failure Status:x3/x2 TMO:x10 Data x101000 x0
2020-05-12T14:48:53.709039+02:00 hostname kernel: [  506.777720] lpfc 0000:07:00.0: 0:(0):2858 FLOGI failure Status:x3/x2 TMO:x10 Data x101000 x0
2020-05-12T14:49:00.153049+02:00 hostname kernel: [  513.223454] lpfc 0000:07:00.1: 1:(0):2858 FLOGI failure Status:x3/x2 TMO:x10 Data x101000 x0
2020-05-12T14:49:13.809051+02:00 hostname kernel: [  526.876940] lpfc 0000:07:00.0: 0:(0):2858 FLOGI failure Status:x3/x2 TMO:x10 Data x101000 x0
2020-05-12T14:49:20.280995+02:00 hostname kernel: [  533.350598] lpfc 0000:07:00.1: 1:(0):2858 FLOGI failure Status:x3/x2 TMO:x10 Data x101000 x0

cat /sys/module/lpfc/parameters/lpfc_topology
0

with 5.4.27-1 FC is working

cat /sys/module/lpfc/parameters/lpfc_topology
0

 

[t.lamprecht]

with 5.4.27-1 FC is working

Thanks for confirming my hunch, I'll take a closer look at this.

 

[Oliver Polterauer]

if you need any more testing, we can also do this directly on the server... can contact me with private message....

 

[zeha]

Hi Thomas,

reverting this commit makes lpfc login properly:

commit 77d5805eafdb5c42bdfe78f058ad9c40ee1278b4
Author: James Smart <jsmart2021@gmail.com>
Date:   Mon Jan 27 16:23:03 2020 -0800

    scsi: lpfc: Fix broken Credit Recovery after driver load

 

[t.lamprecht]

Much thanks for that pointer!

 

[t.lamprecht]

Oh, and got the upstream discussion moving: https://lore.kernel.org/linux-scsi/20200512212855.36q2ut2io2cdtagn@zeha.at/
I'll follow that one.

 

[Oliver Polterauer]

in the newest version the kernel LPFC with 8Gbit Cards seam to work! Could not find the fix at kernel.org did you patch this yourself?

 

[t.lamprecht]

in the newest version the kernel LPFC with 8Gbit Cards seam to work! Could not find the fix at kernel.org did you patch this yourself?

Yes, we reverted the patch causing your issue which zeha found so quickly after the upstream discussion went stale. Not the ideal solution, but for now a workaround a bit back to the previous behavior.
https://git.proxmox.com/?p=pve-kernel.git;a=commitdiff;h=161bea4e8377bec4e608b6e2c98d56abbb3a86f0

 

[Oliver Polterauer]

Yes, we reverted the patch causing your issue which zeha found so quickly after the upstream discussion went stale. Not the ideal solution, but for now a workaround a bit back to the previous behavior.
https://git.proxmox.com/?p=pve-kernel.git;a=commitdiff;h=161bea4e8377bec4e608b6e2c98d56abbb3a86f0

thanks If you need somebody to test if patch is available please contact me I can test this anytime on a test cluster! Zeha has also access to this cluster

 

[t.lamprecht]

Ack, will do!

 

[t.lamprecht]

We now got another fix from upstream backported as stable release, namely:
https://lore.kernel.org/linux-scsi/20200911200147.110826-1-james.smart@broadcom.com/

I'll for the next kernel bump drop the workaround revert, as this one gets applied.
Just a heads up, to watch out for any similar/related regression possibly caused.

 

[Oliver Polterauer]

We now got another fix from upstream backported as stable release, namely:
https://lore.kernel.org/linux-scsi/20200911200147.110826-1-james.smart@broadcom.com/

I'll for the next kernel bump drop the workaround revert, as this one gets applied.
Just a heads up, to watch out for any similar/related regression possibly caused.

Thanks for the update. We will watch this in detail!