Is there a general proxmox problem with qlogic fiber channel cards?

[peanut42]

Hello!

This is the second (2462 and 2532) FC I have used from qlogic and I am running into problems when scanning for the SAN to be recognized by the system:

scsi 3:0:0:0: lun4194304 has a LUN larger than allowed by the host adapter
scsi 3:0:0:0: lun4194560 has a LUN larger than allowed by the host adapter

It really seems like proxmox is allergic to qlogic cards since I don't have these problems in Red Hat or Fedora.

Is there a way to control the generated LUN from the proxmox side? I've been looking at various posts online and can't seem to get ahead.

These two articles (https://microdevsys.com/wp/lun4194304-has-a-lun-larger-than-allowed-by-the-host-adapter/ + https://puerto53.com/linux/errores/lun4194304-has-a-lun-larger-than-allowed-by-the-host-adapter/) suggest to use the "dracut -f" command but when attempting to install this utility the OS warns me that this will completely murder everything related to proxmox.

W: (pve-apt-hook) !! WARNING !!
W: (pve-apt-hook) You are attempting to remove the meta-package 'proxmox-ve'!
W: (pve-apt-hook)
W: (pve-apt-hook) If you really want to permanently remove 'proxmox-ve' from your system, run the following command
W: (pve-apt-hook)       touch '/please-remove-proxmox-ve'
W: (pve-apt-hook) run apt purge proxmox-ve to remove the meta-package
W: (pve-apt-hook) and repeat your apt invocation.
W: (pve-apt-hook)
W: (pve-apt-hook) If you are unsure why 'proxmox-ve' would be removed, please verify
W: (pve-apt-hook)       - your APT repository settings
W: (pve-apt-hook)       - that you are using 'apt full-upgrade' to upgrade your system
E: Sub-process /usr/share/proxmox-ve/pve-apt-hook returned an error code (1)
E: Failure running script /usr/share/proxmox-ve/pve-apt-hook

Is there a way to use these FCs with proxmox or am I just out of luck here?

 

[Stoiko Ivanov]

As far as I can see this is not a general problem with the qlogic cards and the PVE kernel, but you need to provide a parameter to the kernel module if you want to access LUNs > 65k

Depending on which module (=driver) is needed for your particular HBA you need to create a file in /etc/modprobe.d.

Assuming the module is 'qla2xxx.ko' try adding the file: /etc/modprobe.d/qla2xxx.conf with the following content:

options qla2xxxx  ql2xmaxlun=4294967296

(Since I neither have a qlogic card nor a FC-storage at hand I cannot check whether this works)

These two articles (https://microdevsys.com/wp/lun4194304-has-a-lun-larger-than-allowed-by-the-host-adapter/ + https://puerto53.com/linux/errores/lun4194304-has-a-lun-larger-than-allowed-by-the-host-adapter/) suggest to use the "dracut -f" command but when attempting to install this utility the OS warns me that this will completely murder everything related to proxmox.

Proxmox does not use dracut (a package for gnerating initrd's for the kernel) but initramfs-tools - in that case the invocation probably should be:

update-initramfs -k all -u

afterwards reboot and see if it works

I hope this helps!

 

[peanut42]

Thank you so much Stoiko!

Your suggestions helped a lot! Just to clarify, I needed a smaller LUN, therefore I set the max to a relative small number and after the reboot I could connect to the SAN successfully.

 

[Stoiko Ivanov]

Glad you found a solution!

 

[peanut42]

Well, I spoke too soon

When rescanning the Lun I get the exact same message as in the first post... the LUN are exactly the same :'(

echo "- - -" > /sys/class/scsi_host/host3/scan

Is there any other way to manipulate this huge LUN?

 

[Stoiko Ivanov]

* did you adapt the ql2xxx config?
* is ql2xxx the correct module ? lsmod|grep ql

 

[peanut42]

Well, I needed to created the qla2xxx.conf file from scratch.

cat /etc/modprobe.d/qla2xxx.conf
options qla2xxxx  ql2xmaxlun=254

AND

lsmod | grep ql
qla2xxx               749568  0
nvme_fc                40960  1 qla2xxx
scsi_transport_fc      57344  1 qla2xxx

Should I maybe try "qla2xmaxlun" or "qla2maxlun" instead? The thought just came up while typing this....

 

[Stoiko Ivanov]

sorry - i mistyped - the line should be:

options qla2xxx  ql2xmaxlun=254

the line is 'options <modname> <parameter'

(see `man modprobe.d`)

I hope this helps!

 

[peanut42]

Unfortunately, no luck. The LUN is still 4194304 and 4194560 respectively. I give up

 

[Stoiko Ivanov]

Ok - just to be clear - the maxlun parameter needs to be set higher than your highest LUN (I sadly mistyped again) - please set it to 4294967296 (and update-initramfs and reboot)

 

[peanut42]

Same result, the LUNs didn't change.

 

[Stoiko Ivanov]

hmm - could you post the journal from the time when the problem occurs (and the dmesg) - this might help in narrowing this down further...
As said - don't have a fc-storage or HBA to reproduce this here locally

 

[peanut42]

I replaced the Fiberchannel card with something newer since I did not want to fight the qlogic card anymore with:

0e:00.0 Fibre Channel: Emulex Corporation Saturn-X: LightPulse Fibre Channel Host Adapter (rev 03)
0e:00.1 Fibre Channel: Emulex Corporation Saturn-X: LightPulse Fibre Channel Host Adapter (rev 03)

... and also flashed it with the newest firmware (January 2021) and things worked up until today. I had to rebuild the logical volume down to the partition and on my SAN (Infortrend DS 1016GE) and also used the opportunity to update and reboot the proxmox node:

Kernel Version Linux 5.4.157-1-pve #1 SMP PVE 5.4.157-1 (Mon, 29 Nov 2021 12:01:44 +0100)

PVE Manager Version pve-manager/6.4-13/9f411e79

Now however I'm back to the original problem. I run the following to scan for changes:


for host in `ls /sys/class/scsi_host/`;do echo "- - -" >/sys/class/scsi_host/${host}/scan; done


... and receive the following dmesg:


lpfc 0000:0e:00.0: 0:1305 Link Down Event x2 received Data: x2 x20 x80011 x0 x0
rport-0:0-0: blocked FC remote port time out: removing target and saving binding
lpfc 0000:0e:00.0: 00):0203 Devloss timeout on WWPN 21:00:00:d0:23:02:bb:91 NPort x000001 Data: x0 x8 x0
lpfc 0000:0e:00.0: 0:1303 Link Up Event x3 received Data: x3 x0 x20 x0 x0 x0 0
scsi 0:0:0:0: Enclosure IFT DS 1000 Series 556C PQ: 0 ANSI: 5
ses 0:0:0:0: Attached Enclosure device
ses 0:0:0:0: Attached scsi generic sg2 type 13
scsi 0:0:0:1: Enclosure IFT DS 1000 Series 556C PQ: 0 ANSI: 5
ses 0:0:0:1: Attached Enclosure device
ses 0:0:0:1: Attached scsi generic sg3 type 13
scsi 1:0:0:0: lun4194304 has a LUN larger than allowed by the host adapter
scsi 1:0:0:0: lun4194304 has a LUN larger than allowed by the host adapter

As far as I can see (which does not mean anything) there is no maxlun set:

modinfo lpfc |grep -i lun

parm: lpfc_lun_queue_depth:Max number of FCP commands we can queue to a specific LUN (uint)
parm: lpfc_max_luns:Maximum allowed LUN ID (ullong)


What am I doing wrong? I need help! I losing all faith in fiber channel cards and SANs.

 

[peanut42]

WELP, out of desperation I tried unloading and reloading the lpfc module:

modprobe -r lpfc

modprobe lpfc

... and it seems to be working (I know I'm talking to myself, but maybe it will help someone else).

Wish me luck