Proxmox Configuring HP Nimble Multipath and ISCSI

D

dave10x

New Member
Proxmox Subscriber
Oct 14, 2024
25
2
3
I'm getting the below error when trying to connect to a HPE Nimble via ISCSI.

I can ping the nimble and there is no firewall on the physically separated storage network. I'm not using any CHAP auth.

multipath -ll
nimble-storage (2404cc47e5b15031a6c9ce900ee763fd6) dm-6 Nimble,Server
size=1.0T features='1 queue_if_no_path' hwhandler='1 alua' wp=rw
`-+- policy='round-robin 0' prio=1 status=enabled
|- 2:0:0:0 sdc 8:32 failed ghost running
`- 3:0:0:0 sdd 8:48 failed ghost running

multipath.conf


blacklist {
wwid .*
}

blacklist_exceptions {
wwid "2404cc47e5b15031a6c9ce900ee763fd6"
}


multipaths {
multipath {
wwid "2404cc47e5b15031a6c9ce900ee763fd6"
alias mpath0
}
}

defaults {
polling_interval 2
path_selector "round-robin 0"
path_grouping_policy multibus
uid_attribute ID_SERIAL
rr_min_io 100
failback immediate
no_path_retry queue
user_friendly_names yes
}

devices {
device {
vendor "Nimble"
product "Server"
path_grouping_policy group_by_prio
prio "alua"
hardware_handler "1 alua"
path_selector "service-time 0"
path_checker tur
no_path_retry 30
failback immediate
fast_io_fail_tmo 5
dev_loss_tmo infinity
rr_min_io_rq 1
rr_weight uniform
}
}

# Multipath wwids, Version : 1.0
# NOTE: This file is automatically maintained by multipath and multipathd.
# You should not need to edit this file in normal circumstances.
#
# Valid WWIDs:
/2404cc47e5b15031a6c9ce900ee763fd6/

/lib/udev/scsi_id -g -u -d /dev/sdc
2404cc47e5b15031a6c9ce900ee763fd6

/lib/udev/scsi_id -g -u -d /dev/sdd
2404cc47e5b15031a6c9ce900ee763fd6
 
Last edited:
Morning, I have a proxmox cluster and a nimble, are you happy to share your iscsi configuration also?

Also, have you rebooted since you set up the multipath? You "shouldn't" have to, but strangely, sometimes I found that to be the best way to resolve new multipath issues,

The other oddity is when adding a new volume I do need to restart iscsid and multipathd in that order even after the luns show up to get it working.
 
Hi sure, here it is below, I have rebooted MANY times.

cat /etc/iscsi/iscsid.conf
#
# Open-iSCSI default configuration.
# Could be located at /etc/iscsi/iscsid.conf or ~/.iscsid.conf
#
# Note: To set any of these values for a specific node/session run
# the iscsiadm --mode node --op command for the value. See the README
# and man page for iscsiadm for details on the --op command.
#

######################
# iscsid daemon config
######################
#
# If you want iscsid to start the first time an iscsi tool
# needs to access it, instead of starting it when the init
# scripts run, set the iscsid startup command here. This
# should normally only need to be done by distro package
# maintainers. If you leave the iscsid daemon running all
# the time then leave this attribute commented out.
#
# Default for Fedora and RHEL. Uncomment to activate.
# iscsid.startup = /bin/systemctl start iscsid.socket iscsiuio.socket
#
# Default for Debian and Ubuntu. Uncomment to activate.
#iscsid.startup = /bin/systemctl start iscsid.socket
#
# Default if you are not using systemd. Uncomment to activate.
# iscsid.startup = /usr/bin/service start iscsid

# Check for active mounts on devices reachable through a session
# and refuse to logout if there are any. Defaults to "No".
# iscsid.safe_logout = Yes

# Only require UID auth for MGMT IPCs, and not username.
# Useful if you want to run iscsid in a constrained environment.
# Note: Only do this if you are aware of the security implications.
# Defaults to "No".
# iscsid.ipc_auth_uid = Yes

#############################
# NIC/HBA and driver settings
#############################
# open-iscsi can create a session and bind it to a NIC/HBA.
# To set this up see the example iface config file.

#*****************
# Startup settings
#*****************

# To request that the iscsi service scripts startup a session, use "automatic":
# node.startup = automatic
#
# To manually startup the session, use "manual". The default is manual.
node.startup = automatic

# For "automatic" startup nodes, setting this to "Yes" will try logins on each
# available iface until one succeeds, and then stop. The default "No" will try
# logins on all available ifaces simultaneously.
node.leading_login = No

# *************
# CHAP Settings
# *************

# To enable CHAP authentication set node.session.auth.authmethod
# to CHAP. The default is None.
#node.session.auth.authmethod = CHAP

# To configure which CHAP algorithms to enable, set
# node.session.auth.chap_algs to a comma separated list.
# The algorithms should be listed in order of decreasing
# preference — in particular, with the most preferred algorithm first.
# Valid values are MD5, SHA1, SHA256, and SHA3-256.
# The default is MD5.
#node.session.auth.chap_algs = SHA3-256,SHA256,SHA1,MD5

# To set a CHAP username and password for initiator
# authentication by the target(s), uncomment the following lines:
#node.session.auth.username = bob
#node.session.auth.password = Magp1e4545!!

# To set a CHAP username and password for target(s)
# authentication by the initiator, uncomment the following lines:
#node.session.auth.username_in = bob
#node.session.auth.password_in = Magp1e4545!!

# To enable CHAP authentication for a discovery session to the target,
# set discovery.sendtargets.auth.authmethod to CHAP. The default is None.
#discovery.sendtargets.auth.authmethod = CHAP

# To set a discovery session CHAP username and password for the initiator
# authentication by the target(s), uncomment the following lines:
#discovery.sendtargets.auth.username = username
#discovery.sendtargets.auth.password = password

# To set a discovery session CHAP username and password for target(s)
# authentication by the initiator, uncomment the following lines:
#discovery.sendtargets.auth.username_in = username_in
#discovery.sendtargets.auth.password_in = password_in

# ********
# Timeouts
# ********
#
# See the iSCSI README's Advanced Configuration section for tips
# on setting timeouts when using multipath or doing root over iSCSI.
#
# To specify the length of time to wait for session re-establishment
# before failing SCSI commands back to the application when running
# the Linux SCSI Layer error handler, edit the line.
# The value is in seconds and the default is 120 seconds.
# Special values:
# - If the value is 0, IO will be failed immediately.
# - If the value is less than 0, IO will remain queued until the session
# is logged back in, or until the user runs the logout command.
node.session.timeo.replacement_timeout = 15

# To specify the time to wait for login to complete, edit the line.
# The value is in seconds and the default is 15 seconds.
node.conn[0].timeo.login_timeout = 15

# To specify the time to wait for logout to complete, edit the line.
# The value is in seconds and the default is 15 seconds.
node.conn[0].timeo.logout_timeout = 15

# Time interval to wait for on connection before sending a ping.
# The value is in seconds and the default is 5 seconds.
node.conn[0].timeo.noop_out_interval = 5

# To specify the time to wait for a Nop-out response before failing
# the connection, edit this line. Failing the connection will
# cause IO to be failed back to the SCSI layer. If using dm-multipath
# this will cause the IO to be failed to the multipath layer.
# The value is in seconds and the default is 5 seconds.
node.conn[0].timeo.noop_out_timeout = 5

# To specify the time to wait for an abort response before
# failing the operation and trying a logical unit reset, edit the line.
# The value is in seconds and the default is 15 seconds.
node.session.err_timeo.abort_timeout = 15

# To specify the time to wait for a logical unit response
# before failing the operation and trying session re-establishment,
# edit the line.
# The value is in seconds and the default is 30 seconds.
node.session.err_timeo.lu_reset_timeout = 30

# To specify the time to wait for a target response
# before failing the operation and trying session re-establishment,
# edit the line.
# The value is in seconds and the default is 30 seconds.
node.session.err_timeo.tgt_reset_timeout = 30

# The value is in seconds and the default is 60 seconds.
node.session.err_timeo.host_reset_timeout = 60


#******
# Retry
#******

# To specify the number of times iscsid should retry a login
# if the login attempt fails due to the node.conn[0].timeo.login_timeout
# expiring, modify the following line. Note that if the login fails
# quickly (before node.conn[0].timeo.login_timeout fires) because the network
# layer or the target returns an error, iscsid may retry the login more than
# node.session.initial_login_retry_max times.
#
# This retry count along with node.conn[0].timeo.login_timeout
# determines the maximum amount of time iscsid will try to
# establish the initial login. node.session.initial_login_retry_max is
# multiplied by the node.conn[0].timeo.login_timeout to determine the
# maximum amount.
#
# The default node.session.initial_login_retry_max is 8 and
# node.conn[0].timeo.login_timeout is 15 so we have:
#
# node.conn[0].timeo.login_timeout * node.session.initial_login_retry_max = 120s
#
# Valid values are any integer value. This only
# affects the initial login. Setting it to a high value can slow
# down the iscsi service startup. Setting it to a low value can
# cause a session to not get logged into, if there are distuptions
# during startup or if the network is not ready at that time.
node.session.initial_login_retry_max = 8

################################
# session and device queue depth
################################

# To control how many commands the session will queue, set
# node.session.cmds_max to an integer between 2 and 2048 that is also
# a power of 2. The default is 128.
node.session.cmds_max = 128

# To control the device's queue depth, set node.session.queue_depth
# to a value between 1 and 1024. The default is 32.
node.session.queue_depth = 32

##################################
# MISC SYSTEM PERFORMANCE SETTINGS
##################################

# For software iscsi (iscsi_tcp) and iser (ib_iser), each session
# has a thread used to transmit or queue data to the hardware. For
# cxgb3i, you will get a thread per host.
#
# Setting the thread's priority to a lower value can lead to higher throughput
# and lower latencies. The lowest value is -20. Setting the priority to
# a higher value, can lead to reduced IO performance, but if you are seeing
# the iscsi or scsi threads dominate the use of the CPU then you may want
# to set this value higher.
#
# Note: For cxgb3i, you must set all sessions to the same value.
# Otherwise the behavior is not defined.
#
# The default value is -20. The setting must be between -20 and 20.
node.session.xmit_thread_priority = -20


#***************
# iSCSI settings
#***************

# To enable R2T flow control (i.e., the initiator must wait for an R2T
# command before sending any data), uncomment the following line:
#
#node.session.iscsi.InitialR2T = Yes
#
# To disable R2T flow control (i.e., the initiator has an implied
# initial R2T of "FirstBurstLength" at offset 0), uncomment the following line:
#
# The defaults is No.
node.session.iscsi.InitialR2T = No

#
# To disable immediate data (i.e., the initiator does not send
# unsolicited data with the iSCSI command PDU), uncomment the following line:
#
#node.session.iscsi.ImmediateData = No
#
# To enable immediate data (i.e., the initiator sends unsolicited data
# with the iSCSI command packet), uncomment the following line:
#
# The default is Yes.
node.session.iscsi.ImmediateData = Yes

# To specify the maximum number of unsolicited data bytes the initiator
# can send in an iSCSI PDU to a target, edit the following line.
#
# The value is the number of bytes in the range of 512 to (2^24-1) and
# the default is 262144.
node.session.iscsi.FirstBurstLength = 262144

# To specify the maximum SCSI payload that the initiator will negotiate
# with the target for, edit the following line.
#
# The value is the number of bytes in the range of 512 to (2^24-1) and
# the defauls it 16776192.
node.session.iscsi.MaxBurstLength = 16776192

# To specify the maximum number of data bytes the initiator can receive
# in an iSCSI PDU from a target, edit the following line.
#
# The value is the number of bytes in the range of 512 to (2^24-1) and
# the default is 262144.
node.conn[0].iscsi.MaxRecvDataSegmentLength = 262144

# To specify the maximum number of data bytes the initiator will send
# in an iSCSI PDU to the target, edit the following line.
#
# The value is the number of bytes in the range of 512 to (2^24-1).
# Zero is a special case. If set to zero, the initiator will use
# the target's MaxRecvDataSegmentLength for the MaxXmitDataSegmentLength.
# The default is 0.
node.conn[0].iscsi.MaxXmitDataSegmentLength = 0

# To specify the maximum number of data bytes the initiator can receive
# in an iSCSI PDU from a target during a discovery session, edit the
# following line.
#
# The value is the number of bytes in the range of 512 to (2^24-1) and
# the default is 32768.
discovery.sendtargets.iscsi.MaxRecvDataSegmentLength = 32768

# To allow the targets to control the setting of the digest checking,
# with the initiator requesting a preference of enabling the checking,
# uncomment one or both of the following lines:
#node.conn[0].iscsi.HeaderDigest = CRC32C,None
#node.conn[0].iscsi.DataDigest = CRC32C,None
#
# To allow the targets to control the setting of the digest checking,
# with the initiator requesting a preference of disabling the checking,
# uncomment one or both of the following lines:
#node.conn[0].iscsi.HeaderDigest = None,CRC32C
#node.conn[0].iscsi.DataDigest = None,CRC32C
#
# To enable CRC32C digest checking for the header and/or data part of
# iSCSI PDUs, uncomment one or both of the following lines:
#node.conn[0].iscsi.HeaderDigest = CRC32C
#node.conn[0].iscsi.DataDigest = CRC32C
#
# To disable digest checking for the header and/or data part of
# iSCSI PDUs, uncomment one or both of the following lines:
#node.conn[0].iscsi.HeaderDigest = None
#node.conn[0].iscsi.DataDigest = None
#
# The default is to never use DataDigests or HeaderDigests.
#

# For multipath configurations, you may want more than one session to be
# created on each iface record. If node.session.nr_sessions is greater
# than 1, performing a 'login' for that node will ensure that the
# appropriate number of sessions is created.
node.session.nr_sessions = 1

# When iscsid starts up, it recovers existing sessions (if possible).
# If the target for a session has gone away when this occurs, the
# iscsid daemon normally tries to reestablish each session,
# in succession, in the background, by trying again every two
# seconds until all sessions are restored. This configuration
# variable can limits the number of retries for each session.
# For example, setting reopen_max=150 would mean that each session
# recovery was limited to about five minutes.
node.session.reopen_max = 0

#************
# Workarounds
#************

# Some targets like IET prefer that an initiator does not respond to PDUs like
# R2Ts after it has sent a task management function like an ABORT TASK or a
# LOGICAL UNIT RESET. To adopt this behavior, uncomment the following line.
# The default is Yes.
node.session.iscsi.FastAbort = Yes

# Some targets like Equalogic prefer that an initiator continue to respond to
# R2Ts after it has sent a task management function like an ABORT TASK or a
# LOGICAL UNIT RESET. To adopt this behavior, uncomment the following line.
# node.session.iscsi.FastAbort = No

# To prevent doing automatic scans that would add unwanted luns to the system,
# we can disable them and have sessions only do manually requested scans.
# Automatic scans are performed on startup, on login, and on AEN/AER reception
# on devices supporting it. For HW drivers, all sessions will use the value
# defined in the configuration file. This configuration option is independent
# of the scsi_mod.scan parameter. The default is auto.
node.session.scan = auto
root@pmx6:~#
 
root@pmx6:~# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS
sda 8:0 1 0B 0 disk
sdb 8:16 0 558.9G 0 disk
├─sdb1 8:17 0 1007K 0 part
├─sdb2 8:18 0 1G 0 part /boot/efi
└─sdb3 8:19 0 557.9G 0 part
├─pve-swap 252:0 0 8G 0 lvm [SWAP]
├─pve-root 252:1 0 96G 0 lvm /
├─pve-data_tmeta 252:2 0 4.4G 0 lvm
│ └─pve-data-tpool 252:4 0 429.1G 0 lvm
│ └─pve-data 252:5 0 429.1G 1 lvm
└─pve-data_tdata 252:3 0 429.1G 0 lvm
└─pve-data-tpool 252:4 0 429.1G 0 lvm
└─pve-data 252:5 0 429.1G 1 lvm
sdc 8:32 0 1T 0 disk
└─mpath0 252:6 0 1T 0 mpath
sdd 8:48 0 1T 0 disk
└─mpath0 252:6 0 1T 0 mpath
 
ok, can you also send the output of multipath -ll -v4

On the nimble if you goto monitor/connections can you confirm that you see all the connections you expect to see, also are you vlaning at all for the iscsi network? if so can you show your proxmox network config, also what is in /etc/iscsi/nodes/iqn.xxxxxx/ipaddr/

do you have multiple iscsi ifaces setup in /etc/iscsi/ifaces
 
Sure thanks for this,
root@pmx6:~# multipath -ll -v4
9879.873657 | set open fds limit to 1048576/1048576
9879.873695 | loading /lib/multipath/libchecktur.so checker
9879.873786 | checker tur: message table size = 3
9879.873793 | loading /lib/multipath/libprioconst.so prioritizer
9879.873860 | _init_foreign: found libforeign-nvme.so
9879.873868 | _init_foreign: foreign library "nvme" is not enabled
9879.876473 | Discover device /sys/devices/pci0000:00/0000:00:14.0/usb3/3-4/3-4:1.0/host0/target0:0:0/0:0:0:0/block/sda
9879.876525 | 8:0: dev_t not found in pathvec
9879.876546 | sda: mask = 0x27
9879.876555 | sda: dev_t = 8:0
9879.876557 | open '/sys/devices/pci0000:00/0000:00:14.0/usb3/3-4/3-4:1.0/host0/target0:0:0/0:0:0:0/block/sda/size'
9879.876566 | sda: size = 0
9879.876622 | sda: vendor = Generic-
9879.876633 | sda: product = SD/MMC CRW
9879.876649 | sda: rev = 1.00
9879.877081 | find_hwe: found 0 hwtable matches for Generic-:SD/MMC CRW:1.00
9879.877085 | sda: h:b:t:l = 0:0:0:0
9879.877159 | sda: tgt_node_name = 3-4:1.0
9879.877163 | sda: skip USB device 3-4:1.0
9879.877168 | Discover device /sys/devices/pci0000:5b/0000:5b:00.0/0000:5c:00.0/host1/target1:1:0/1:1:0:0/block/sdb
9879.877200 | 8:16: dev_t not found in pathvec
9879.877213 | sdb: mask = 0x27
9879.877216 | sdb: dev_t = 8:16
9879.877220 | open '/sys/devices/pci0000:5b/0000:5b:00.0/0000:5c:00.0/host1/target1:1:0/1:1:0:0/block/sdb/size'
9879.877229 | sdb: size = 1172058032
9879.877276 | sdb: vendor = HPE
9879.877286 | sdb: product = LOGICAL VOLUME
9879.877298 | sdb: rev = 2.65
9879.877675 | find_hwe: found match /HP:LOGICAL VOLUME:(null)/ for 'HPE:LOGICAL VOLUME:2.65'
9879.877750 | find_hwe: found 1 hwtable matches for HPE:LOGICAL VOLUME:2.65
9879.877753 | sdb: h:b:t:l = 1:1:0:0
9879.877853 | sdb: tgt_node_name =
9879.877857 | sdb: uid_attribute = ID_SERIAL (setting: multipath.conf defaults/devices section)
9879.877860 | sdb: recheck_wwid = 1 (setting: multipath.conf defaults/devices section)
9879.877947 | sdb: udev property ID_WWN whitelisted
9879.877962 | open '/sys/devices/pci0000:5b/0000:5b:00.0/0000:5c:00.0/host1/target1:1:0/1:1:0:0/state'
9879.877971 | sdb: path state = running
9879.877984 | sdb: 65535 cyl, 255 heads, 63 sectors/track, start at 0
9879.877988 | sdb: vpd_vendor_id = 0 "undef" (setting: multipath internal)
9879.877995 | open '/sys/devices/pci0000:5b/0000:5b:00.0/0000:5c:00.0/host1/target1:1:0/1:1:0:0/vpd_pg80'
9879.878009 | sdb: serial = PWXKV0ARHDHFTK
9879.878013 | sdb: detect_checker = yes (setting: multipath internal)
9879.878117 | open '/sys/devices/pci0000:5b/0000:5b:00.0/0000:5c:00.0/host1/target1:1:0/1:1:0:0/inquiry'
9879.878126 | sdb: path_checker = tur (setting: multipath internal)
9879.878129 | sdb: checker timeout = 30 s (setting: kernel sysfs)
9879.878155 | sdb: tur state = up
9879.878159 | Discover device /sys/devices/pci0000:5b/0000:5b:00.0/0000:5c:00.0/host1/target1:1:0/1:1:0:0/block/sdb/sdb1
9879.878202 | Discover device /sys/devices/pci0000:5b/0000:5b:00.0/0000:5c:00.0/host1/target1:1:0/1:1:0:0/block/sdb/sdb2
9879.878235 | Discover device /sys/devices/pci0000:5b/0000:5b:00.0/0000:5c:00.0/host1/target1:1:0/1:1:0:0/block/sdb/sdb3
9879.878266 | Discover device /sys/devices/platform/host2/session1/target2:0:0/2:0:0:0/block/sdc
9879.878295 | 8:32: dev_t not found in pathvec
9879.878307 | sdc: mask = 0x27
9879.878311 | sdc: dev_t = 8:32
9879.878314 | open '/sys/devices/platform/host2/session1/target2:0:0/2:0:0:0/block/sdc/size'
9879.878324 | sdc: size = 2147483648
9879.878371 | sdc: vendor = Nimble
9879.878381 | sdc: product = Server
9879.878390 | sdc: rev = 1.0
9879.878412 | find_hwe: found match /Nimble:Server:(null)/ for 'Nimble:Server:1.0'
9879.878834 | find_hwe: found 1 hwtable matches for Nimble:Server:1.0
9879.878838 | sdc: h:b:t:l = 2:0:0:0
9879.878948 | sdc: tgt_node_name = iqn.2007-11.com.nimblestorage:gcnimble-g69551a7ebf6a6c90
9879.878951 | sdc: uid_attribute = ID_SERIAL (setting: multipath.conf defaults/devices section)
9879.878954 | sdc: recheck_wwid = 1 (setting: multipath.conf defaults/devices section)
9879.879058 | sdc: udev property SCSI_IDENT_LUN_EUI64 whitelisted
9879.879067 | open '/sys/devices/platform/host2/session1/target2:0:0/2:0:0:0/state'
9879.879075 | sdc: path state = running
9879.879114 | sdc: 65535 cyl, 255 heads, 63 sectors/track, start at 0
9879.879118 | sdc: vpd_vendor_id = 0 "undef" (setting: multipath internal)
9879.879125 | open '/sys/devices/platform/host2/session1/target2:0:0/2:0:0:0/vpd_pg80'
9879.879133 | sdc: serial = 404cc47e5b15031a6c9ce900ee763fd6
9879.879137 | sdc: detect_checker = yes (setting: multipath internal)
9879.879152 | open '/sys/devices/platform/host2/session1/target2:0:0/2:0:0:0/inquiry'
9879.879160 | open '/sys/devices/platform/host2/session1/target2:0:0/2:0:0:0/vpd_pg83'
9879.879462 | sdc: path_checker = tur (setting: storage device autodetected)
9879.879466 | sdc: checker timeout = 30 s (setting: kernel sysfs)
9879.879572 | sdc: tur state = ghost
9879.879575 | Discover device /sys/devices/platform/host3/session2/target3:0:0/3:0:0:0/block/sdd
9879.879615 | 8:48: dev_t not found in pathvec
9879.879627 | sdd: mask = 0x27
9879.879630 | sdd: dev_t = 8:48
9879.879635 | open '/sys/devices/platform/host3/session2/target3:0:0/3:0:0:0/block/sdd/size'
9879.879643 | sdd: size = 2147483648
9879.879689 | sdd: vendor = Nimble
9879.879699 | sdd: product = Server
9879.879709 | sdd: rev = 1.0
9879.879738 | find_hwe: found match /Nimble:Server:(null)/ for 'Nimble:Server:1.0'
9879.880208 | find_hwe: found 1 hwtable matches for Nimble:Server:1.0
9879.880213 | sdd: h:b:t:l = 3:0:0:0
9879.880327 | sdd: tgt_node_name = iqn.2007-11.com.nimblestorage:gcnimble-g69551a7ebf6a6c90
9879.880331 | sdd: uid_attribute = ID_SERIAL (setting: multipath.conf defaults/devices section)
9879.880335 | sdd: recheck_wwid = 1 (setting: multipath.conf defaults/devices section)
9879.880426 | sdd: udev property SCSI_IDENT_LUN_EUI64 whitelisted
9879.880433 | open '/sys/devices/platform/host3/session2/target3:0:0/3:0:0:0/state'
9879.880440 | sdd: path state = running
9879.880469 | sdd: 65535 cyl, 255 heads, 63 sectors/track, start at 0
9879.880474 | sdd: vpd_vendor_id = 0 "undef" (setting: multipath internal)
9879.880480 | open '/sys/devices/platform/host3/session2/target3:0:0/3:0:0:0/vpd_pg80'
9879.880488 | sdd: serial = 404cc47e5b15031a6c9ce900ee763fd6
9879.880491 | sdd: detect_checker = yes (setting: multipath internal)
9879.880505 | open '/sys/devices/platform/host3/session2/target3:0:0/3:0:0:0/inquiry'
9879.880513 | open '/sys/devices/platform/host3/session2/target3:0:0/3:0:0:0/vpd_pg83'
9879.880816 | sdd: path_checker = tur (setting: storage device autodetected)
9879.880820 | sdd: checker timeout = 30 s (setting: kernel sysfs)
9879.880916 | sdd: tur state = ghost
9879.880919 | Discover device /sys/devices/virtual/block/loop0
9879.880946 | 7:0: dev_t not found in pathvec
9879.880959 | loop0: device node name blacklisted
9879.880964 | Discover device /sys/devices/virtual/block/loop1
9879.880987 | 7:1: dev_t not found in pathvec
9879.880997 | loop1: device node name blacklisted
9879.881010 | Discover device /sys/devices/virtual/block/loop2
9879.881032 | 7:2: dev_t not found in pathvec
9879.881043 | loop2: device node name blacklisted
9879.881047 | Discover device /sys/devices/virtual/block/loop3
9879.881068 | 7:3: dev_t not found in pathvec
9879.881078 | loop3: device node name blacklisted
9879.881082 | Discover device /sys/devices/virtual/block/loop4
9879.881102 | 7:4: dev_t not found in pathvec
9879.881112 | loop4: device node name blacklisted
9879.881115 | Discover device /sys/devices/virtual/block/loop5
9879.881135 | 7:5: dev_t not found in pathvec
9879.881144 | loop5: device node name blacklisted
9879.881148 | Discover device /sys/devices/virtual/block/loop6
9879.881169 | 7:6: dev_t not found in pathvec
9879.881178 | loop6: device node name blacklisted
9879.881182 | Discover device /sys/devices/virtual/block/loop7
9879.881204 | 7:7: dev_t not found in pathvec
9879.881214 | loop7: device node name blacklisted
9879.881218 | Discover device /sys/devices/virtual/block/dm-0
9879.881241 | 252:0: dev_t not found in pathvec
9879.881252 | dm-0: device node name blacklisted
9879.881256 | Discover device /sys/devices/virtual/block/dm-1
9879.881282 | 252:1: dev_t not found in pathvec
9879.881292 | dm-1: device node name blacklisted
9879.881296 | Discover device /sys/devices/virtual/block/dm-2
9879.881318 | 252:2: dev_t not found in pathvec
9879.881328 | dm-2: device node name blacklisted
9879.881332 | Discover device /sys/devices/virtual/block/dm-3
9879.881352 | 252:3: dev_t not found in pathvec
9879.881362 | dm-3: device node name blacklisted
9879.881366 | Discover device /sys/devices/virtual/block/dm-4
9879.881387 | 252:4: dev_t not found in pathvec
9879.881397 | dm-4: device node name blacklisted
9879.881400 | Discover device /sys/devices/virtual/block/dm-5
9879.881421 | 252:5: dev_t not found in pathvec
9879.881431 | dm-5: device node name blacklisted
9879.881435 | Discover device /sys/devices/virtual/block/dm-6
9879.881455 | 252:6: dev_t not found in pathvec
9879.881465 | dm-6: device node name blacklisted
9879.881469 | Discovered 3/19 paths
===== paths list =====
uuid hcil dev dev_t pri dm_st chk_st vend/prod/rev dev_st
1:1:0:0 sdb 8:16 -1 undef undef HPE,LOGICAL VOLUME unknown
2:0:0:0 sdc 8:32 -1 undef undef Nimble,Server unknown
3:0:0:0 sdd 8:48 -1 undef undef Nimble,Server unknown
9879.881783 | multipath-tools v0.9.4 (12/19, 2022)
9879.881792 | libdevmapper version 1.02.185
9879.881862 | kernel device mapper v4.48.0
9879.881873 | DM multipath kernel driver v1.14.0
9879.881953 | mpath0: disassemble map [1 queue_if_no_path 1 alua 1 1 service-time 0 2 2 8:32 1 1 8:48 1 1 ]
9879.881966 | mpath0: disassemble status [2 0 1 0 1 1 E 0 2 2 8:32 F 1813 0 1 8:48 F 4316 0 1 ]
9879.881973 | sdc: mask = 0x1f
9879.881975 | sdc: dev_t = 8:32
9879.881978 | open '/sys/devices/platform/host2/session1/target2:0:0/2:0:0:0/block/sdc/size'
9879.881990 | sdc: size = 2147483648
9879.881995 | sdc: vendor = Nimble
9879.881999 | sdc: product = Server
9879.882010 | sdc: rev = 1.0
9879.882024 | find_hwe: found match /Nimble:Server:(null)/ for 'Nimble:Server:1.0'
9879.882447 | find_hwe: found 1 hwtable matches for Nimble:Server:1.0
9879.882451 | sdc: h:b:t:l = 2:0:0:0
9879.882496 | sdc: tgt_node_name = iqn.2007-11.com.nimblestorage:gcnimble-g69551a7ebf6a6c90
9879.882500 | open '/sys/devices/platform/host2/session1/target2:0:0/2:0:0:0/state'
9879.882508 | sdc: path state = running
9879.882536 | sdc: 65535 cyl, 255 heads, 63 sectors/track, start at 0
9879.882540 | sdc: vpd_vendor_id = 0 "undef" (setting: multipath internal)
9879.882545 | open '/sys/devices/platform/host2/session1/target2:0:0/2:0:0:0/vpd_pg80'
9879.882553 | sdc: serial = 404cc47e5b15031a6c9ce900ee763fd6
9879.882758 | sdc: tur state = ghost
9879.882763 | sdc: uid = 2404cc47e5b15031a6c9ce900ee763fd6 (udev)
9879.882767 | sdc: detect_prio = yes (setting: multipath internal)
9879.882774 | open '/sys/devices/platform/host2/session1/target2:0:0/2:0:0:0/access_state'
9879.882785 | open '/sys/devices/platform/host2/session1/target2:0:0/2:0:0:0/preferred_path'
9879.882796 | loading /lib/multipath/libpriosysfs.so prioritizer
9879.882876 | sdc: prio = sysfs (setting: storage device autodetected)
9879.882879 | sdc: prio args = "" (setting: storage device autodetected)
9879.882885 | open '/sys/devices/platform/host2/session1/target2:0:0/2:0:0:0/access_state'
9879.882893 | open '/sys/devices/platform/host2/session1/target2:0:0/2:0:0:0/preferred_path'
9879.882900 | sdc: sysfs prio = 1
9879.882905 | sdd: mask = 0x1f
9879.882908 | sdd: dev_t = 8:48
9879.882912 | open '/sys/devices/platform/host3/session2/target3:0:0/3:0:0:0/block/sdd/size'
9879.882924 | sdd: size = 2147483648
9879.882929 | sdd: vendor = Nimble
9879.882932 | sdd: product = Server
9879.882936 | sdd: rev = 1.0
9879.882950 | find_hwe: found match /Nimble:Server:(null)/ for 'Nimble:Server:1.0'
9879.883373 | find_hwe: found 1 hwtable matches for Nimble:Server:1.0
9879.883377 | sdd: h:b:t:l = 3:0:0:0
9879.883424 | sdd: tgt_node_name = iqn.2007-11.com.nimblestorage:gcnimble-g69551a7ebf6a6c90
9879.883428 | open '/sys/devices/platform/host3/session2/target3:0:0/3:0:0:0/state'
9879.883436 | sdd: path state = running
9879.883458 | sdd: 65535 cyl, 255 heads, 63 sectors/track, start at 0
9879.883462 | sdd: vpd_vendor_id = 0 "undef" (setting: multipath internal)
9879.883467 | open '/sys/devices/platform/host3/session2/target3:0:0/3:0:0:0/vpd_pg80'
9879.883474 | sdd: serial = 404cc47e5b15031a6c9ce900ee763fd6
9879.883636 | sdd: tur state = ghost
9879.883639 | sdd: uid = 2404cc47e5b15031a6c9ce900ee763fd6 (udev)
9879.883644 | sdd: detect_prio = yes (setting: multipath internal)
9879.883648 | open '/sys/devices/platform/host3/session2/target3:0:0/3:0:0:0/access_state'
9879.883656 | open '/sys/devices/platform/host3/session2/target3:0:0/3:0:0:0/preferred_path'
9879.883664 | sdd: prio = sysfs (setting: storage device autodetected)
9879.883668 | sdd: prio args = "" (setting: storage device autodetected)
9879.883671 | open '/sys/devices/platform/host3/session2/target3:0:0/3:0:0:0/access_state'
9879.883680 | open '/sys/devices/platform/host3/session2/target3:0:0/3:0:0:0/preferred_path'
9879.883686 | sdd: sysfs prio = 1
mpath0 (2404cc47e5b15031a6c9ce900ee763fd6) dm-6 Nimble,Server
size=1.0T features='1 queue_if_no_path' hwhandler='1 alua' wp=rw
`-+- policy='service-time 0' prio=1 status=enabled
|- 2:0:0:0 sdc 8:32 failed ghost running
`- 3:0:0:0 sdd 8:48 failed ghost running
9879.884098 | tur checker refcount 3
9879.884193 | tur checker refcount 2
9879.884200 | sysfs prioritizer refcount 2
9879.884262 | tur checker refcount 1
9879.884266 | sysfs prioritizer refcount 1
9879.884436 | unloading tur checker
9879.884480 | unloading sysfs prioritizer
9879.884499 | unloading const prioritizer
root@pmx6:~#
 
can you explain your network topology for iscsi, I asked about vlans and iscsi ifaces before also, just trying to build a picture of what you have, so far I see nothing wrong expect the failure of both paths,
 
Ok sorry, missed that part.

root@pmx6:~# cd /etc/iscsi/nodes/
root@pmx6:/etc/iscsi/nodes# ls
iqn.2007-11.com.nimblestorage:gcnimble-g69551a7ebf6a6c90


So the Nimbles Data nics are plugged directly into a separate switch not connected to the main network with a single vlan configured "30" iirc.

There are two Nimbles one in one building and one in another, I am only connecting to one as i cant connect to the other side as its physically not able yet.

I'm now wondering which is set as primary currently and if that's why it can t see the connection. I will check this.

My proxmox network config is:


auto lo
iface lo inet loopback

iface eno5np0 inet manual

iface eno6np1 inet manual

auto ens1f0np0
iface ens1f0np0 inet manual

auto ens1f1np1
iface ens1f1np1 inet manual

auto ens3f0np0
iface ens3f0np0 inet manual

auto ens3f1np1
iface ens3f1np1 inet manual

auto ens2f0np0
iface ens2f0np0 inet static
address 192.168.30.145/24 <<-- nimble data ports

auto ens2f1np1
iface ens2f1np1 inet static
address 192.168.30.144/24 <<-- nimble data ports

iface usb0 inet manual

auto vmbr0
iface vmbr0 inet static
address 10.x.x.x/24
gateway 10.x.x.254
bridge-ports eno5np0
bridge-stp off
bridge-fd 0

source /etc/network/interfaces.d/*


Thanks.
 
Last edited:
We have a winner, I've recreated the LUN but removed protection so its only running on one NIMBLE. doing an FDISK-l I can see the disk now. It must be due to not being able to see both the primary and secondary nimble. I will have the storage network connected via fiber to the other building later this week and try it with a fully redundant setup.

Thanks for the help really appreciated.
 
one step forward....

May have spoken too soon.

the device shows in fdisk -l and the LUN is present in the proxmox gui but when i try to start a VM on the LUN it givers this error.

iscsiadm: default: 1 session requested, but 1 already present.
iscsiadm: default: 1 session requested, but 1 already present.
iscsiadm: Could not login to [iface: default, target: iqn.2007-11.com.nimblestorage:gcnimble-g69551a7ebf6a6c90, portal: 192.168.30.13,3260].
iscsiadm: initiator reported error (15 - session exists)
iscsiadm: Could not login to [iface: default, target: iqn.2007-11.com.nimblestorage:gcnimble-g69551a7ebf6a6c90, portal: 192.168.30.14,3260].
iscsiadm: initiator reported error (15 - session exists)
iscsiadm: Could not log into all portals
Logging in to [iface: default, target: iqn.2007-11.com.nimblestorage:gcnimble-g69551a7ebf6a6c90, portal: 192.168.30.13,3260]
Logging in to [iface: default, target: iqn.2007-11.com.nimblestorage:gcnimble-g69551a7ebf6a6c90, portal: 192.168.30.14,3260]
command '/usr/bin/iscsiadm --mode node --targetname iqn.2007-11.com.nimblestorage:gcnimble-g69551a7ebf6a6c90 --login' failed: exit code 15
/bin/swtpm exit with status 256:
TASK ERROR: start failed: command 'swtpm_setup --tpmstate file:///dev/disk/by-id/scsi-SNimble_Server_2614f0c861be04846c9ce900ee763fd6 --createek --create-ek-cert --create-platform-cert --lock-nvram --config /etc/swtpm_setup.conf --runas 0 --not-overwrite --tpm2 --ecc' failed: exit code 1
 
Hi @dave10x,

It does make sense that if you had an option enabled in Nimble that "if replication is not active then primary site should limit IO". Its a bit extreme but some business cases call for this.

As a matter of hygiene, I recommend using the top menu in "post" editor that provides for CODE and SPOILER tags for copy/paste configuration/output. It will make your posts much more readable.
A trick I always use for configuration files with a lot of comments and defaults: grep -Ev "^#|$^" iscsid.conf . This will only output options that are actively enabled.

The next step in your troubleshooting is to :
a) present how you are configuring iSCSI (cat /etc/pve/storage.cfg)
b) present your VM configuration (qm config [vmid] )
c) present your current iscsi db status: iscsiadm -m node ; iscsiadm -m session

Cheers

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox
 
Not using ZFS on this. Would like to get a working disk 1st before making this even more complex :D.

I've not created an LVM on it, as i cant see the disk in the GUI and thought it may break if i do it via CMD.
 
you should see the volume in the GUI, and then create an LVM on it at the cluster level, before you try to put a VM on it,

also, see the question from bbgeek17, he brings up a valid couple of questions, when you connect to the iscsi LUN are you doing that through the GUI at the cluster level?
 
@ bbgeek17

root@pmx6:~# cat /etc/pve/storage.cfg
dir: local
path /var/lib/vz
content iso,backup,vztmpl

lvmthin: local-lvm
thinpool data
vgname pve
content rootdir,images

iscsi: Nimble
portal 192.168.30.10
target iqn.2007-11.com.nimblestorage:gcnimble-g69551a7ebf6a6c90
content none

I deleted the VM.
root@pmx6:~# iscsiadm -m node
192.168.30.11:3260,2461 iqn.2007-11.com.nimblestorage:gcnimble-g69551a7ebf6a6c90
192.168.30.12:3260,2461 iqn.2007-11.com.nimblestorage:gcnimble-g69551a7ebf6a6c90
192.168.30.13:3260,2464 iqn.2007-11.com.nimblestorage:gcnimble-g69551a7ebf6a6c90
192.168.30.14:3260,2464 iqn.2007-11.com.nimblestorage:gcnimble-g69551a7ebf6a6c90


root@pmx6:~# iscsiadm -m session
tcp: [1] 192.168.30.11:3260,2461 iqn.2007-11.com.nimblestorage:gcnimble-g69551a7ebf6a6c90 (non-flash)
tcp: [2] 192.168.30.12:3260,2461 iqn.2007-11.com.nimblestorage:gcnimble-g69551a7ebf6a6c90 (non-flash)
 
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom