Setting Up single Shared FC LUN - multipath -ll shows both Shared and boot lun

[marowens]

I'm setting up a test proxmox lab. It's been a minute but what was new this time was I'm using block level LUNs from a Pure Flash Array. My UCS blades do not have local storage so my service profile was setup properly to access the Pure Array. All configurations were done on the Pure and booting and installing to SAN was successful. Once I made it to creating the Shared LUN I was met with multipathing and what needs to successfully configure each node to get them set to see the LUN.

Most articles were for iSCSI whereas I'm using FC. Finding and understanding the /etc/multipath.conf file is a challenge and I got both from Pure's doc (i think ) When I thought I had completed the steps I ran #multipath -ll and got nothing. I then edited the file with my first find from Pure- restarted multipathd and i got my output. Next was to try 'pvecreate /dev/mapper/mpatha - it failed. I then found the other multipath.conf from Pure and replaced my file with this one. Now my output shows both the SAN Boot LUN and the Shared LUN. Same results trying to run the pvcreate.

reading more and trying to see what this is doing I realized the second multipath file created an exception for the pure array whereas the other didn't. I think this is why I'm seeing both LUNS on each node. I replaced the multipath.conf file with the original and rebooted all 4 nodes (BTW if I didn't state it the cluster was created and all 4 nodes added prior to starting this configuration). I also found on Pure's site to run pvcreate with the Pure's device ID. I did this and successfully created the LUN then refreshed each node and all 4 successfully see the 10T shared LUN.

My issue - well when I run multipath - ll, on each node I see the nodes boot lun and the shared lun. I don't know if it's normal as all output I've seen assumes local boot and a shared iscisi or fc LUN.


screenshots of outputs below

File from Pure to create /etc/multipath.conf Here, I missed the BOLD TOP instructions to use the device ID when doing the pvcreate (rookie mistake)




First output of the above multipath.conf configuration...



Second try at the /etc/multipath.conf (again from PURE)

I can't find the file now but I know it created a blacklist exception for the VENDOR PURE. after reloading multipathd i get this...



Keep in mind the multipath.conf was restored to the original and I still see both LUNs



The ultimate question (where the answer isn't '42') is this normal or should that boot LUN not be seen?

Thanks, in advance

 

[bbgeek17]

Hi @marowens , welcome to the forum.

First thing - do everyone a favor and do not use screenshots, or avoid them if at all possible. Pasting basic text and using CODE tags from the edit box menu is much easier to read.

Second, did you install pure Debian with PVE packages on top, or did you use PVE ISO?
Does your boot LUN have multiple paths? If so you should use them! PVE ISO currently does not provide a way to take advantage of boot time multipath, so you should use Debian install ISO and follow the procedure to use multipath for boot. Followed by installing PVE packages on top.

Even if your boot LUN only has single path (why would it?), you can still make use of multipath. This will cover you if you add second path later.

There is nothing inherently wrong with having MP control your boot LUN, as long as you use correct device names everywhere. If it bothers you - exclude it from MP config.

None of this is PVE specific. You should decide on the path (pun intended) you want to take, and then follow the Pure/Debian install and configuration guides.

Cheers

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[marowens]

I'll take note of the screenshot issue. Thanks

I installed PVE 9.2.1 on bare metal - SAN Boot obviously.

When you're flying in the dark it's about seeing output change after creating the first /etc/multipath.conf file. Essentially, not understanding that when I did the pvcreate and it failed that replacing the mpatha with the device ID I would have never modified the multipath.conf file from the original I created. I did back it up first prior to replacing it and after i restarted services and running the multipath -ll afterwards I now see both.

For me if originally I see only my shared LUN, then change the file and now see both boot and shared LUNs the question is WHY?

If you're saying that seeing both is fine, okay, But seeing only one, the Shared LUN, leaves zero room for error when running the pvcreate /dev/mapper/<device-id>

 

[bbgeek17]

When a system has gone through multiple configuration changes, it can be difficult to reconstruct exactly how it transitioned from one state to another. Changes made along the way, configuration files that may have been overwritten or not picked up, parsing errors, and services that haven't fully reconciled their state can all complicate the picture.

Going back and forth between screenshots and trying to compare individual details is also not particularly productive. There simply isn't enough information to establish what the system looked like before the changes or how it arrived at its current state.

Keep in mind - you lived through the setup, we did not. It may seem to you that your explanation is clear, but we don't have the background. I've caught myself doing it many times.

I'd suggest rebooting the node(s), making no further configuration changes, and collecting the following outputs from the resulting clean boot:

lsblk -e7 -o NAME,KNAME,PATH,TYPE,SIZE,RO,PKNAME,MAJ:MIN,FSTYPE,UUID,WWN,SERIAL,HCTL,TRAN,MOUNTPOINTS
blkid
findmnt
cat /proc/cmdline

lsscsi -g -u -t

multipath -ll
multipath -d -v3
multipathd show maps
multipathd show paths
multipathd show topology

cat /etc/multipath.conf
multipath -t
multipathd show config

dmsetup ls --tree
pvs -o+devices
vgs
lvs -a -o+devices

journalctl -b -u multipathd --no-pager

Please provide the output as plain text rather than screenshots, retaining the WWIDs and device identifiers.

This is ultimately a Linux administration and troubleshooting exercise. Having a complete picture of the current device discovery, multipath configuration, and LVM layout should give us a much better basis for understanding what's happening. Feel free to post the results here; perhaps someone in the community will spot something useful.

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[marowens]

Thanks - the screenshot thing again

Yeah, I've done all those things you've suggested when my multipath -ll output changed.

The first response to this thread states it's okay to see your SAN boot lun in the multipath output. If that's not an issue then my issue is resolved.

To your point what happened when the file was changed on a single node in the cluster and then the multipath -ll output changed to show both LUNs is difficult to even guess what happened but for me it seemed worthy to discuss.

In vmware when you have exposed multiple LUNs that are in a host group on the server side it sees every LUN including your boot LUN. Multipathing being new I just wasn't sure.

Thanks for those commands I've already tried some and getting a broader idea of what I'm seeing.