Proxmox 6.2 LPFC error port type wrong

Oliver Polterauer · May 12, 2020

We did today on one of your test clusters an upgrade from 6.1 (running 5.3.18-3-pve) to 6.2 (5.4.34-1-pve) and after that the port_type for the lpfc drivers is not working anymore. Booting with the old kernel one gets FC Connection back online.

Version 5.3.18-3-pve

modinfo:
description: Emulex LightPulse Fibre Channel SCSI driver 12.2.0.3

Code:

port_type = "NPort (fabric via point-to-point)"
port_type = "NPort (fabric via point-to-point)"
Status: Online

Version 5.4.34-1-pve

modinfo:
description: Emulex LightPulse Fibre Channel SCSI driver 12.6.0.4

Code:

port_type = "Point-To-Point (direct nport connection)"
port_type = "Point-To-Point (direct nport connection)"
Status: Bypassed

We tried to set the port-selection within FC Bios but did not help.

Firmewareversion of Controller: FV2.10X6

t.lamprecht · May 12, 2020

Do you have any suspicious message from the SCSI or LPFC subsystem in the kernel log, dmesg?

What's the topology set to the module: cat /sys/module/lpfc/parameters/lpfc_topology

Also, if you can it could be worth to try out the 5.4.27-1-pve kernel version, as after that one a huge bunch of lpfc changes got into the kernel with an older FW version:
apt install pve-kernel-5.4.27-1-pve

Oliver Polterauer · May 12, 2020

t.lamprecht said:
Do you have any suspicious message from the SCSI or LPFC subsystem in the kernel log, dmesg?

What's the topology set to the module: cat /sys/module/lpfc/parameters/lpfc_topology

Also, if you can it could be worth to try out the 5.4.27-1-pve kernel version, as after that one a huge bunch of lpfc changes got into the kernel with an older FW version:
apt install pve-kernel-5.4.27-1-pve

these are the errors from log file:

Code:

2020-05-12T14:40:44.780036+02:00 hostname kernel: [    8.701385] lpfc 0000:07:00.0: 0:1303 Link Up Event x1 received Data: x1 xf7 x20 x0 x0 x0 0
2020-05-12T14:40:44.780052+02:00 hostname kernel: [   11.177527] lpfc 0000:07:00.1: 1:1303 Link Up Event x1 received Data: x1 xf7 x20 x0 x0 x0 0
2020-05-12T14:40:56.138424+02:00 hostname kernel: [   28.779567] lpfc 0000:07:00.0: 0:(0):2858 FLOGI failure Status:x3/x2 TMO:x10 Data x101000 x0
2020-05-12T14:40:58.618409+02:00 hostname kernel: [   31.259404] lpfc 0000:07:00.1: 1:(0):2858 FLOGI failure Status:x3/x2 TMO:x10 Data x101000 x0
2020-05-12T14:41:16.202437+02:00 hostname kernel: [   48.845746] lpfc 0000:07:00.0: 0:(0):2858 FLOGI failure Status:x3/x2 TMO:x10 Data x101000 x0
2020-05-12T14:41:18.702449+02:00 hostname kernel: [   51.345606] lpfc 0000:07:00.1: 1:(0):2858 FLOGI failure Status:x3/x2 TMO:x10 Data x101000 x0
2020-05-12T14:41:36.266434+02:00 hostname kernel: [   68.910887] lpfc 0000:07:00.0: 0:(0):2858 FLOGI failure Status:x3/x2 TMO:x10 Data x101000 x0
2020-05-12T14:41:38.786384+02:00 hostname kernel: [   71.430776] lpfc 0000:07:00.1: 1:(0):2858 FLOGI failure Status:x3/x2 TMO:x10 Data x101000 x0
2020-05-12T14:41:56.374439+02:00 hostname kernel: [   89.018037] lpfc 0000:07:00.0: 0:(0):2858 FLOGI failure Status:x3/x2 TMO:x10 Data x101000 x0
2020-05-12T14:41:58.870415+02:00 hostname kernel: [   91.513955] lpfc 0000:07:00.1: 1:(0):2858 FLOGI failure Status:x3/x2 TMO:x10 Data x101000 x0
2020-05-12T14:42:16.021077+02:00 hostname kernel: [  109.081209] lpfc 0000:07:00.0: 0:(0):2858 FLOGI failure Status:x3/x2 TMO:x10 Data x101000 x0
2020-05-12T14:42:18.556997+02:00 hostname kernel: [  111.620017] lpfc 0000:07:00.1: 1:(0):2858 FLOGI failure Status:x3/x2 TMO:x10 Data x101000 x0
2020-05-12T14:42:36.081087+02:00 hostname kernel: [  129.141345] lpfc 0000:07:00.0: 0:(0):2858 FLOGI failure Status:x3/x2 TMO:x10 Data x101000 x0
2020-05-12T14:42:38.656995+02:00 hostname kernel: [  131.720169] lpfc 0000:07:00.1: 1:(0):2858 FLOGI failure Status:x3/x2 TMO:x10 Data x101000 x0
2020-05-12T14:42:56.225058+02:00 hostname kernel: [  149.287507] lpfc 0000:07:00.0: 0:(0):2858 FLOGI failure Status:x3/x2 TMO:x10 Data x101000 x0
2020-05-12T14:42:58.725043+02:00 hostname kernel: [  151.789462] lpfc 0000:07:00.1: 1:(0):2858 FLOGI failure Status:x3/x2 TMO:x10 Data x101000 x0
2020-05-12T14:43:16.281043+02:00 hostname kernel: [  169.342679] lpfc 0000:07:00.0: 0:(0):2858 FLOGI failure Status:x3/x2 TMO:x10 Data x101000 x0
2020-05-12T14:43:18.789021+02:00 hostname kernel: [  171.853541] lpfc 0000:07:00.1: 1:(0):2858 FLOGI failure Status:x3/x2 TMO:x10 Data x101000 x0
2020-05-12T14:43:36.329078+02:00 hostname kernel: [  189.391874] lpfc 0000:07:00.0: 0:(0):2858 FLOGI failure Status:x3/x2 TMO:x10 Data x101000 x0
2020-05-12T14:43:38.837062+02:00 hostname kernel: [  191.898749] lpfc 0000:07:00.1: 1:(0):2858 FLOGI failure Status:x3/x2 TMO:x10 Data x101000 x0
2020-05-12T14:43:56.385045+02:00 hostname kernel: [  209.448035] lpfc 0000:07:00.0: 0:(0):2858 FLOGI failure Status:x3/x2 TMO:x10 Data x101000 x0
2020-05-12T14:43:58.901065+02:00 hostname kernel: [  211.962912] lpfc 0000:07:00.1: 1:(0):2858 FLOGI failure Status:x3/x2 TMO:x10 Data x101000 x0
2020-05-12T14:44:16.453033+02:00 hostname kernel: [  229.518206] lpfc 0000:07:00.0: 0:(0):2858 FLOGI failure Status:x3/x2 TMO:x10 Data x101000 x0
2020-05-12T14:44:18.965031+02:00 hostname kernel: [  232.030086] lpfc 0000:07:00.1: 1:(0):2858 FLOGI failure Status:x3/x2 TMO:x10 Data x101000 x0
2020-05-12T14:44:36.525087+02:00 hostname kernel: [  249.588380] lpfc 0000:07:00.0: 0:(0):2858 FLOGI failure Status:x3/x2 TMO:x10 Data x101000 x0
2020-05-12T14:44:39.021014+02:00 hostname kernel: [  252.084245] lpfc 0000:07:00.1: 1:(0):2858 FLOGI failure Status:x3/x2 TMO:x10 Data x101000 x0
2020-05-12T14:44:56.573038+02:00 hostname kernel: [  269.637536] lpfc 0000:07:00.0: 0:(0):2858 FLOGI failure Status:x3/x2 TMO:x10 Data x101000 x0
2020-05-12T14:44:59.101022+02:00 hostname kernel: [  272.167392] lpfc 0000:07:00.1: 1:(0):2858 FLOGI failure Status:x3/x2 TMO:x10 Data x101000 x0
2020-05-12T14:45:16.701075+02:00 hostname kernel: [  289.765704] lpfc 0000:07:00.0: 0:(0):2858 FLOGI failure Status:x3/x2 TMO:x10 Data x101000 x0
2020-05-12T14:45:19.185064+02:00 hostname kernel: [  292.249601] lpfc 0000:07:00.1: 1:(0):2858 FLOGI failure Status:x3/x2 TMO:x10 Data x101000 x0
2020-05-12T14:45:36.740988+02:00 hostname kernel: [  309.806804] lpfc 0000:07:00.0: 0:(0):2858 FLOGI failure Status:x3/x2 TMO:x10 Data x101000 x0
2020-05-12T14:45:39.261020+02:00 hostname kernel: [  312.327781] lpfc 0000:07:00.1: 1:(0):2858 FLOGI failure Status:x3/x2 TMO:x10 Data x101000 x0
2020-05-12T14:45:56.809080+02:00 hostname kernel: [  329.874036] lpfc 0000:07:00.0: 0:(0):2858 FLOGI failure Status:x3/x2 TMO:x10 Data x101000 x0
2020-05-12T14:45:59.325061+02:00 hostname kernel: [  332.389947] lpfc 0000:07:00.1: 1:(0):2858 FLOGI failure Status:x3/x2 TMO:x10 Data x101000 x0
2020-05-12T14:46:16.857050+02:00 hostname kernel: [  349.923182] lpfc 0000:07:00.0: 0:(0):2858 FLOGI failure Status:x3/x2 TMO:x10 Data x101000 x0
2020-05-12T14:46:19.385044+02:00 hostname kernel: [  352.450112] lpfc 0000:07:00.1: 1:(0):2858 FLOGI failure Status:x3/x2 TMO:x10 Data x101000 x0
2020-05-12T14:46:36.945036+02:00 hostname kernel: [  370.011364] lpfc 0000:07:00.0: 0:(0):2858 FLOGI failure Status:x3/x2 TMO:x10 Data x101000 x0
2020-05-12T14:46:39.480988+02:00 hostname kernel: [  372.549235] lpfc 0000:07:00.1: 1:(0):2858 FLOGI failure Status:x3/x2 TMO:x10 Data x101000 x0
2020-05-12T14:46:57.053043+02:00 hostname kernel: [  390.119547] lpfc 0000:07:00.0: 0:(0):2858 FLOGI failure Status:x3/x2 TMO:x10 Data x101000 x0
2020-05-12T14:46:59.641044+02:00 hostname kernel: [  392.707463] lpfc 0000:07:00.1: 1:(0):2858 FLOGI failure Status:x3/x2 TMO:x10 Data x101000 x0
2020-05-12T14:47:17.109054+02:00 hostname kernel: [  410.174721] lpfc 0000:07:00.0: 0:(0):2858 FLOGI failure Status:x3/x2 TMO:x10 Data x101000 x0
2020-05-12T14:47:19.693010+02:00 hostname kernel: [  412.760620] lpfc 0000:07:00.1: 1:(0):2858 FLOGI failure Status:x3/x2 TMO:x10 Data x101000 x0
2020-05-12T14:47:37.253022+02:00 hostname kernel: [  430.318861] lpfc 0000:07:00.0: 0:(0):2858 FLOGI failure Status:x3/x2 TMO:x10 Data x101000 x0
2020-05-12T14:47:39.749040+02:00 hostname kernel: [  432.817806] lpfc 0000:07:00.1: 1:(0):2858 FLOGI failure Status:x3/x2 TMO:x10 Data x101000 x0
2020-05-12T14:47:54.341031+02:00 hostname kernel: [  447.407880] lpfc 0000:07:00.0: 0:(0):0237 Pending Link Event during Discovery: State x7
2020-05-12T14:47:54.341460+02:00 hostname kernel: [  447.409116] lpfc 0000:07:00.0: 0:1305 Link Down Event x2 received Data: x2 x7 x98014 x0 x0
2020-05-12T14:47:54.409045+02:00 hostname kernel: [  447.476795] lpfc 0000:07:00.0: 0:1303 Link Up Event x3 received Data: x3 x0 x20 x0 x0 x0 0
2020-05-12T14:47:59.849052+02:00 hostname kernel: [  452.916982] lpfc 0000:07:00.1: 1:(0):2858 FLOGI failure Status:x3/x2 TMO:x10 Data x101000 x0
2020-05-12T14:48:13.541042+02:00 hostname kernel: [  466.607427] lpfc 0000:07:00.0: 0:(0):2858 FLOGI failure Status:x3/x2 TMO:x10 Data x101000 x0
2020-05-12T14:48:19.945034+02:00 hostname kernel: [  473.014108] lpfc 0000:07:00.1: 1:(0):2858 FLOGI failure Status:x3/x2 TMO:x10 Data x101000 x0
2020-05-12T14:48:33.609047+02:00 hostname kernel: [  486.676600] lpfc 0000:07:00.0: 0:(0):2858 FLOGI failure Status:x3/x2 TMO:x10 Data x101000 x0
2020-05-12T14:48:40.029050+02:00 hostname kernel: [  493.097325] lpfc 0000:07:00.1: 1:(0):2858 FLOGI failure Status:x3/x2 TMO:x10 Data x101000 x0
2020-05-12T14:48:53.709039+02:00 hostname kernel: [  506.777720] lpfc 0000:07:00.0: 0:(0):2858 FLOGI failure Status:x3/x2 TMO:x10 Data x101000 x0
2020-05-12T14:49:00.153049+02:00 hostname kernel: [  513.223454] lpfc 0000:07:00.1: 1:(0):2858 FLOGI failure Status:x3/x2 TMO:x10 Data x101000 x0
2020-05-12T14:49:13.809051+02:00 hostname kernel: [  526.876940] lpfc 0000:07:00.0: 0:(0):2858 FLOGI failure Status:x3/x2 TMO:x10 Data x101000 x0
2020-05-12T14:49:20.280995+02:00 hostname kernel: [  533.350598] lpfc 0000:07:00.1: 1:(0):2858 FLOGI failure Status:x3/x2 TMO:x10 Data x101000 x0

Code:

cat /sys/module/lpfc/parameters/lpfc_topology
0

with 5.4.27-1 FC is working

Code:

cat /sys/module/lpfc/parameters/lpfc_topology
0

t.lamprecht · May 12, 2020

Oliver Polterauer said:
with 5.4.27-1 FC is working

Thanks for confirming my hunch, I'll take a closer look at this.

Oliver Polterauer · May 12, 2020

if you need any more testing, we can also do this directly on the server... can contact me with private message....

zeha · May 12, 2020

Hi Thomas,

reverting this commit makes lpfc login properly:

Code:

commit 77d5805eafdb5c42bdfe78f058ad9c40ee1278b4
Author: James Smart <jsmart2021@gmail.com>
Date:   Mon Jan 27 16:23:03 2020 -0800

    scsi: lpfc: Fix broken Credit Recovery after driver load

t.lamprecht · May 13, 2020

Much thanks for that pointer!

t.lamprecht · May 13, 2020

Oh, and got the upstream discussion moving: https://lore.kernel.org/linux-scsi/20200512212855.36q2ut2io2cdtagn@zeha.at/
I'll follow that one.

Oliver Polterauer · May 27, 2020

in the newest version the kernel LPFC with 8Gbit Cards seam to work! Could not find the fix at kernel.org did you patch this yourself?

t.lamprecht · May 27, 2020

Oliver Polterauer said:
in the newest version the kernel LPFC with 8Gbit Cards seam to work! Could not find the fix at kernel.org did you patch this yourself?

Yes, we reverted the patch causing your issue which zeha found so quickly after the upstream discussion went stale. Not the ideal solution, but for now a workaround a bit back to the previous behavior.
https://git.proxmox.com/?p=pve-kernel.git;a=commitdiff;h=161bea4e8377bec4e608b6e2c98d56abbb3a86f0

Oliver Polterauer · May 27, 2020

t.lamprecht said:
Yes, we reverted the patch causing your issue which zeha found so quickly after the upstream discussion went stale. Not the ideal solution, but for now a workaround a bit back to the previous behavior.
https://git.proxmox.com/?p=pve-kernel.git;a=commitdiff;h=161bea4e8377bec4e608b6e2c98d56abbb3a86f0

thanks If you need somebody to test if patch is available please contact me I can test this anytime on a test cluster! Zeha has also access to this cluster

t.lamprecht · May 27, 2020

Ack, will do!

t.lamprecht · Nov 16, 2020

We now got another fix from upstream backported as stable release, namely:
https://lore.kernel.org/linux-scsi/20200911200147.110826-1-james.smart@broadcom.com/

I'll for the next kernel bump drop the workaround revert, as this one gets applied.
Just a heads up, to watch out for any similar/related regression possibly caused.

Oliver Polterauer · Nov 16, 2020

t.lamprecht said:
We now got another fix from upstream backported as stable release, namely:
https://lore.kernel.org/linux-scsi/20200911200147.110826-1-james.smart@broadcom.com/

I'll for the next kernel bump drop the workaround revert, as this one gets applied.
Just a heads up, to watch out for any similar/related regression possibly caused.

Thanks for the update. We will watch this in detail!

Search

Search

Proxmox 6.2 LPFC error port type wrong

Oliver Polterauer

Member

t.lamprecht

Proxmox Staff Member

Oliver Polterauer

Member

t.lamprecht

Proxmox Staff Member

Oliver Polterauer

Member

zeha

Renowned Member

t.lamprecht

Proxmox Staff Member

t.lamprecht

Proxmox Staff Member

Oliver Polterauer

Member

t.lamprecht

Proxmox Staff Member

Oliver Polterauer

Member

t.lamprecht

Proxmox Staff Member

t.lamprecht

Proxmox Staff Member

Oliver Polterauer

Member