[SOLVED] Problems installing 8.1 and iscsi/multipath on root filesystem on HPE BL660c blade with HPE 650FLB hba/nic

[softworx]

Hello,

I'm working on a poc to install proxmox on an HPE BL660c blade connected to an iSCSI san using be2iscsi based controller/nics.

I have it mostly installed and working, though I've run into a problem with iscsi. I've been reading through the forums and googling for both proxmox and debian related topics, but so far the solution, or even the root issue, remains unsolved.

Everything works with Ubuntu 22.04 and AlmaLinux 9, so it's isolated to something to do with proxmox.

The SAN network is isolated from the data network, with SAN configuration saved to firmware via BIOS settings and available to the OS via iBFT. As such, the only way to talk to the san is by specifying the be2iscsi controller's interface to iscsiadm.

I'm installing via ipxe boot and using the remote console on the blade to run the text version of the installer.

The debug installer is needed to install the iscsi and multipath packages and then configure things at post install in order for the system to boot and bring everything up. At that point it seems things are working as expected : iscsi sees 2 sessions, and multipath has the two connections in /dev/mapper.

The problem is with iscsi when attempting to add additional iscsi sessions. Any attempt to tell iscsiadm to do anything that requires communication to the san fails with timeout errors. It's acting as though the iscsiadm commands are trying to use the network interface to connect to the san, and not the controller's interface (be2iscsi), even when specifying the interfaces on the command.

My installation method is below in case someone sees something I missed, or did wrong.

I should add I attempted to use the debian netinstall to install debian, without success, as even though I can drop to shell and see /dev/sda, sdb, sdc, and sdd (2 luns, both with 2 connections) the debian installer refuses to recognize these as drives. In this regard, the proxmox debian base works better than debian itself. This has prevented me from installing debian first and then adding the proxmox packages on top of debian as mentioned is needed for certain custom installation scenarios.

Any one have any ideas?

Thanks in advance,
Greg

Installation Method:

1. ipxe is setup to look for custom ipxe file based on mac address
2. ipxe file for proxmox host:
#!ipxe

# Figure out if client is 64-bit capable
cpuid --ext 29 && set arch x86_64 || set arch x86

echo "Booting ProxMox 8.1 (${arch})
# base-url set earlier in boot process -- http://{server}/path/ipxe/ <- trailing slash required
set boot-url ${base-url}proxmox/
set kernel-url ${boot-url}linux26
set initrd-url ${boot-url}initrd
set iso-url ${boot-url}proxmox-ve_8.1-1.iso
imgfree
kernel ${kernel-url} vga=normal video=vesafb:ywrap,mtrr ramdisk_size=16777216 rw initrd=initrd.magic splash=verbose proxtui iscsi_auto debug proxdebug install disk-detect/multipath/enable=true
initrd ${initrd-url}
initrd ${iso-url} /proxmox.iso
boot

3. boot via pxe
4. start install
5. at first pause with prompt, press Ctrl-D to resume
6. do the following steps at the second prompt:
cd /cdrom/debian/proxmox/packages/
dpkg -i libisns0_0.101-0.2+b1_amd64.deb libopeniscsiusr_2.1.8-1_amd64.deb open-iscsi_2.1.8-1_amd64.deb
iscsistart -f
iscsistart -b
# the above uses iBFT to pick up the iSCSI parameters configured via bios in the controller
press Ctrl-D to resume install
7. install proxmox normally, selecting to use desired boot drive ONLY (/dev/sda in my case)
8. when install finishes, at final prompt, do the following steps:
# choose either /target or /mnt to work in
mount /dev/pve/root /mnt
mount -o bind /dev /mnt/dev
mount -o bind /proc /mnt/proc
mount -o bind /sys /mnt/sys
chroot /mnt/ /bin/bash
cd /etc/apt/sources.list.d/
sed -i -e 's/^/#/' ceph.list
sed -i -e 's/^/#/' pve-enterprise.list
cd ..
echo 'deb https://download.proxmox.com/debian/pve bookworm pve-no-subscription' >> sources.list
apt update

# iscsi.initramfs
echo 'ISCSI_AUTO=true' > /etc/iscsi/iscsi.initramfs

# multipath.conf (kludgy, but echo always works)
echo "defaults {" >> /etc/multipath.com
echo " find_multipaths yes" >> /etc/multipath.com
echo "}" >> /etc/multipath.com
echo "" >> /etc/multipath.com
echo "blacklist {" >> /etc/multipath.com
echo "}" >> /etc/multipath.com

apt install multipath-tools multipath-tools-boot
multipath -a /dev/sda
multipath -a /dev/sdb

update-initramfs -u

Press Ctrl-D to exit
9. Reboot
10. When system is up, log in to shell
11. Do the following:
# verify multipath is working as expected
multipath -ll

# verify controller interfaces are listed
iscsiadm -m iface

# verify iscsi sessions are in place on expected interfaces
iscsiadm -m session

# check mounting
mount
# the root file system should be on a /dev/mapper/* device,
# and the /dev/mapper/* should point towards multipath device
# not a single disk (i.e. /dev/sda)



Up to this point it appears everything is working. It is possible to use the system.
The problem is iscsiadm commands to show targets, discover targets, connect sessions, etc all result in timeouts.
iscsiadm --mode discoverydb --type sendtargets --portal {ip of san} -I {see -m iface output for interface} --discover

No new connections can be made, no changes made, which means no adding luns for VMs, and more critically, being unable to add the additional sessions for redundancy (each iscsi interface connects to 4 ports, for a total of 8 connections under multipath).

 

[softworx]

To follow up my own post, with some more information in case it is helpful and someone can spot what is going on.

I went through the installation process again. This time I attempted to create the additional connections from the installation environment before the final reboot.

It's acting as something is blocking the iscsiadm tool from talking to the be2iscsi interfaces. The only parts working are the two initial targets set up in bios on the HPE650flb.

In Alma and Ubuntu, the iscsiadm command is able to use the be2iscsi interfaces (-I argument on iscsiadm) to add additional connections.

So far Debian/ProxMox does not seem to work.

I'm still trying things, but so far no joy.

Thanks
Greg

 

[softworx]

No replies to this thread. But there's not been a lot of time since my original post; or maybe this is such a niche there's not enough people to be able to help.

In either event, I did get the installation fully working, full iscsi and multipath setup. I'll post the process below in case it helps someone else.

Note: The installer has a couple of issues that makes this more difficult. I'll try to find out where to post bugs so hopefully that can be remedied. For one, disk selection is limited to only /dev/sd*, it will not let you pick /dev/mapper/mpatha nor will it honor a symlink from /dev/sdq to /dev/mapper/mpatha. Also, if multipath is enabled at the second pause, the installer would go through part of the installation to /dev/sda, then LVM would detect the LVM volumes also on /dev/mapper/mpatha, and the installation would halt. If the installer would handle these better, then everything could be done in the second debug pause and this process would be easier.

This process assumes you have ipxe set up and have the necessary bits in place for all of that. This also requires the use of the debug installation, which pauses at three places and gives you a shell prompt to set some things up.

Multipath, iscsi, proxmox 8.1 Installation process:

1. ipxe boot

#!ipxe

# assumes base-url is set earlier in the ipxe boot process
#set base-url http://{server}/path/
# trailing slash is required

# Figure out if client is 64-bit capable
cpuid --ext 29 && set arch x86_64 || set arch x86

echo "Booting ProxMox 8.1 (${arch})
set boot-url ${base-url}proxmox/
set kernel-url ${boot-url}linux26
set initrd-url ${boot-url}initrd
set iso-url ${boot-url}proxmox-ve_8.1-1.iso
set display-opts vga=normal video=vesafb:ywrap,mtrr
set boot-opts ramdisk_size=16777216 rw splash=verbose iscsi_auto disk-detect/multipath/enable=true
set initrd-opts initrd=initrd.magic
set proxmox-opts proxtui debug proxdebug install
imgfree
# quiet
kernel ${kernel-url} ${display-opts} ${boot-opts} ${initrd-opts} ${proxmox-opts}
initrd ${initrd-url}
initrd ${iso-url} /proxmox.iso
boot

2. boot machine
3. start install process
4. at first shell prompt, press Ctrl-D to continue, nothing to do here
5. at second shell prompt, do the following

# set up your primary NIC, below assumes dhcp
dhclient -v eno49

# iscsiadm needs this
mkdir /run/lock

# install iscsi
cd /cdrom/debian/proxmox/packages/
dpkg -i libisns0_0.101-0.2+b1_amd64.deb libopeniscsiusr_2.1.8-1_amd64.deb open-iscsi_2.1.8-1_amd64.deb
cd -

# set basic initiator name
echo "InitiatorName={this server initiator}" > /etc/iscsi/initiatorname.iscsi

# start iscsid
iscsid

# use iscsistart to look at iBFT details
iscsistart -f

# use iscsiadm to set up everything else
# get interfaces
iscsiadm -m iface
iface1=$(iscsiadm -m iface | awk '{print $1}' | grep 'ipv4' | head -n 1)
iface2=$(iscsiadm -m iface | awk '{print $1}' | grep 'ipv4' | tail -n 1)
if [ "${iface1}" = "" -o "${iface2}" = "" ]; then
    echo "ERROR: Unable to determine both iscsi interfaces"
    echo "stop here"
fi
# save interfaces if you want
#rm -f ifaces.sh
#echo "iface1=$iface1" >> ifaces.sh
#echo "iface2=$iface2" >> ifaces.sh

# connect everything
iscsiadm -m discovery -I $iface1 -I $iface2 --portal {portal1 ip addr} --type sendtargets
iscsiadm -m discovery -I $iface1 -I $iface2 --portal {portal2 ip addr} --type sendtargets
iscsiadm -m node --op=update -n node.conn[0].startup -v automatic
iscsiadm -m node --op=update -n node.startup -v automatic
iscsiadm -m node --loginall=automatic

echo "Pausing for 5 seconds for iscsi"
for T in 1 2 3 4 5 ; do echo -n "." ; sleep 1 ; done ; echo "done"
iscsiadm -m session
# you should see all your iscsi connections

## if the installer didn't have a bug with LVM detecting multipath volumes,
## or not letting multipath drives be selected for installation, multipath
## could be set up here. until those are addressed, don't do it, the install
## will not be successful.

Press ctrl-d to resume the installation process

6. Install as desired
7. At the third (post-install) shell prompt, before the reboot, do the following:

mount /dev/pve/root /mnt
mount -o bind /dev /mnt/dev
mount -o bind /proc /mnt/proc
mount -o bind /sys /mnt/sys
cp -a /etc/iscsi /mnt/root/iscsi
chroot /mnt/ /bin/bash

# these steps are now being run in the chroot environment
cp -a /root/iscsi/ifaces /etc/iscsi/ifaces
cp -a /root/iscsi/nodes /etc/iscsi/nodes
cp -a /root/iscsi/send_targets /etc/iscsi/send_targets
cp -a /etc/iscsi/initiatorname.iscsi /etc/iscsi/initiatorname.iscsi.backup
cp -a /root/iscsi/initiatorname.iscsi /etc/iscsi/initiatorname.iscsi

# setup multipath with minimal config
MPATHCONF=/etc/multipath.conf
if [ -e ${MPATHCONF} ]; then
    mv ${MPATHCONF} ${MPATHCONF}.backup
fi
echo "defaults {" > ${MPATHCONF}
echo "        user_friendly_names yes" >> ${MPATHCONF}
echo "        find_multipaths yes" >> ${MPATHCONF}
echo "}" >> ${MPATHCONF}
echo "" >> ${MPATHCONF}
echo "blacklist {" >> ${MPATHCONF}
echo "}" >> ${MPATHCONF}

# this is only needed if running no-subscription
cd /etc/apt/sources.list.d/
sed -i -e 's/^/#/' ceph.list
sed -i -e 's/^/#/' pve-enterprise.list
cd -
cd /etc/apt/
echo 'deb https://download.proxmox.com/debian/pve bookworm pve-no-subscription' >> sources.list
cd -

# enable iscsi in initrd
echo "ISCSI_AUTO=true" > /etc/iscsi/iscsi.initramfs

# install multipath
apt update
apt install -y multipath-tools multipath-tools-boot

# start multipath
multipathd

# add your multipath devices, in my case 8
multipath -a /dev/sda
multipath -a /dev/sdb
multipath -a /dev/sdc
multipath -a /dev/sdd
multipath -a /dev/sde
multipath -a /dev/sdf
multipath -a /dev/sdg
multipath -a /dev/sdh

# normally not needed, but every so often it seems to help
echo "pausing for 15 seconds for multipath"
for T in 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 ; do echo -n "." ; sleep 1 ; done ; echo "done"

# update initramfs with the new setup
update-initramfs -u

# enable multipathd and iscsid for the next boot, if they aren't already
systemctl enable multipathd
systemctl enable iscsid

That should be it.
Press ctrl-d to exit chroot
Press ctrl-d to exit shell and resume installation process.

8. At this point, the installation is finished and after pressing ctrl-d the last time in the previous step, the host should reboot

9. If all went well, the host should boot into proxmox.
* log in to the shell as root
* iscsiadm -m session should let all of your iscsi connections (my setup expects 8)
* multipath -ll should list multipath connections (my setup expects 8 total per lun)

 

[softworx]

Bug reported as #5111
Feature enhancements reported as #5112 and #5113