Issue with Duplicate Disks Displayed in Proxmox

[xcientist]

I am experiencing an issue in my Proxmox environment where I see four instances of the same disk, all showing the same WWN. I am utilizing a SAN storage system and allocating disk space using a pathing method.

Could you help me understand why the same disk is appearing multiple times? Is this a common issue when using SAN storage, or could it be related to the configuration of the HBA or storage itself?

what kind of solution should I follow? wwn numbers are all the same, I can't get blacklisted.

Thank you for your assistance!

 

[waltar]

I don't know if the webui is able to show multipath devices (/dev/mapper/...) instead of only /dev/sd* ones but did you setup multipath ?
What is "multipath -ll -v3" show you ?

 

[xcientist]

How can I configure it correctly? I'm still new, do I need to set up "multipath"? I'm not sure; I didn't configure anything, it came like this in the initial installation.

root@PVE01:~# multipath -ll -v3
1372.326717 | set open fds limit to 1048576/1048576
1372.326768 | loading /lib/multipath/libchecktur.so checker
1372.326850 | checker tur: message table size = 3
1372.326859 | loading /lib/multipath/libprioconst.so prioritizer
1372.326927 | _init_foreign: foreign library "nvme" is not enabled
1372.329858 | sda: size = 1874329600
1372.329914 | sda: vendor = BROADCOM
1372.329927 | sda: product = MR9560-16i
1372.329939 | sda: rev = 5.22
1372.330462 | sda: h:b:t:l = 0:3:111:0
1372.330560 | sda: tgt_node_name =
1372.330569 | sda: uid_attribute = ID_SERIAL (setting: multipath internal)
1372.330576 | sda: recheck_wwid = 1 (setting: multipath.conf defaults/devices section)
1372.330702 | sda: 51135 cyl, 255 heads, 63 sectors/track, start at 0
1372.330707 | sda: vpd_vendor_id = 0 "undef" (setting: multipath internal)
1372.330717 | sda: serial = 00c7765a57ab114a2dc0333115b26200
1372.330721 | sda: detect_checker = yes (setting: multipath internal)
1372.330821 | sda: path_checker = tur (setting: multipath internal)
1372.330826 | sda: checker timeout = 90 s (setting: kernel sysfs)
1372.330868 | sda: tur state = up
1372.331013 | sdb: size = 8589934592
1372.331066 | sdb: vendor = HUAWEI
1372.331078 | sdb: product = S6800T
1372.331088 | sdb: rev = 6000
1372.331577 | sdb: h:b:t:l = 22:0:2:1
1372.331730 | sdb: tgt_node_name = 0x2100c469f05ec050
1372.331735 | sdb: uid_attribute = ID_SERIAL (setting: multipath internal)
1372.331737 | sdb: recheck_wwid = 1 (setting: multipath.conf defaults/devices section)
1372.332150 | sdb: 0 cyl, 64 heads, 32 sectors/track, start at 0
1372.332157 | sdb: vpd_vendor_id = 0 "undef" (setting: multipath internal)
1372.332168 | sdb: serial = 2102353SYSFSNA1000070025
1372.332173 | sdb: detect_checker = yes (setting: multipath internal)
1372.332306 | loading /lib/multipath/libcheckrdac.so checker
1372.332395 | checker rdac: message table size = 9
1372.332401 | sdb: path_checker = rdac (setting: storage device autodetected)
1372.332415 | sdb: checker timeout = 30 s (setting: kernel sysfs)
1372.332529 | rdac checker failed to set TAS bit
1372.332589 | sdb: rdac state = up
1372.332643 | sdc: size = 8589934592
1372.332698 | sdc: vendor = HUAWEI
1372.332710 | sdc: product = S6800T
1372.332721 | sdc: rev = 6000
1372.333300 | sdc: h:b:t:l = 22:0:3:1
1372.333459 | sdc: tgt_node_name = 0x2100c469f05ec050
1372.333465 | sdc: uid_attribute = ID_SERIAL (setting: multipath internal)
1372.333467 | sdc: recheck_wwid = 1 (setting: multipath.conf defaults/devices section)
1372.333672 | sdc: 0 cyl, 64 heads, 32 sectors/track, start at 0
1372.333680 | sdc: vpd_vendor_id = 0 "undef" (setting: multipath internal)
1372.333693 | sdc: serial = 2102353SYSFSNA1000070025
1372.333697 | sdc: detect_checker = yes (setting: multipath internal)
1372.333830 | sdc: path_checker = rdac (setting: storage device autodetected)
1372.333851 | sdc: checker timeout = 30 s (setting: kernel sysfs)
1372.333965 | rdac checker failed to set TAS bit
1372.334038 | sdc: rdac state = up
1372.334100 | sdd: size = 8589934592
1372.334156 | sdd: vendor = HUAWEI
1372.334169 | sdd: product = S6800T
1372.334180 | sdd: rev = 6000
1372.334677 | sdd: h:b:t:l = 24:0:2:1
1372.334838 | sdd: tgt_node_name = 0x2100c469f05ec050
1372.334844 | sdd: uid_attribute = ID_SERIAL (setting: multipath internal)
1372.334845 | sdd: recheck_wwid = 1 (setting: multipath.conf defaults/devices section)
1372.335085 | sdd: 0 cyl, 64 heads, 32 sectors/track, start at 0
1372.335095 | sdd: vpd_vendor_id = 0 "undef" (setting: multipath internal)
1372.335107 | sdd: serial = 2102353SYSFSNA1000070025
1372.335114 | sdd: detect_checker = yes (setting: multipath internal)
1372.335255 | sdd: path_checker = rdac (setting: storage device autodetected)
1372.335277 | sdd: checker timeout = 30 s (setting: kernel sysfs)
1372.335379 | rdac checker failed to set TAS bit
1372.335438 | sdd: rdac state = up
1372.335494 | sde: size = 8589934592
1372.335551 | sde: vendor = HUAWEI
1372.335563 | sde: product = S6800T
1372.335574 | sde: rev = 6000
1372.336078 | sde: h:b:t:l = 24:0:3:1
1372.336230 | sde: tgt_node_name = 0x2100c469f05ec050
1372.336236 | sde: uid_attribute = ID_SERIAL (setting: multipath internal)
1372.336238 | sde: recheck_wwid = 1 (setting: multipath.conf defaults/devices section)
1372.336415 | sde: 0 cyl, 64 heads, 32 sectors/track, start at 0
1372.336425 | sde: vpd_vendor_id = 0 "undef" (setting: multipath internal)
1372.336435 | sde: serial = 2102353SYSFSNA1000070025
1372.336439 | sde: detect_checker = yes (setting: multipath internal)
1372.336575 | sde: path_checker = rdac (setting: storage device autodetected)
1372.336596 | sde: checker timeout = 30 s (setting: kernel sysfs)
1372.336693 | rdac checker failed to set TAS bit
1372.336747 | sde: rdac state = up
1372.336787 | loop0: device node name blacklisted
1372.336822 | loop1: device node name blacklisted
1372.336854 | loop2: device node name blacklisted
1372.336886 | loop3: device node name blacklisted
1372.336915 | loop4: device node name blacklisted
1372.336944 | loop5: device node name blacklisted
1372.336974 | loop6: device node name blacklisted
1372.337012 | loop7: device node name blacklisted
1372.337046 | dm-0: device node name blacklisted
1372.337078 | dm-1: device node name blacklisted
1372.337107 | dm-2: device node name blacklisted
1372.337136 | dm-3: device node name blacklisted
1372.337167 | dm-4: device node name blacklisted
===== paths list =====
uuid hcil dev dev_t pri dm_st chk_st vend/prod/rev dev_st
0:3:111:0 sda 8:0 -1 undef undef BROADCOM,MR9560-16i unknown
22:0:2:1 sdb 8:16 -1 undef undef HUAWEI,S6800T unknown
22:0:3:1 sdc 8:32 -1 undef undef HUAWEI,S6800T unknown
24:0:2:1 sdd 8:48 -1 undef undef HUAWEI,S6800T unknown
24:0:3:1 sde 8:64 -1 undef undef HUAWEI,S6800T unknown
1372.338780 | multipath-tools v0.9.4 (12/19, 2022)
1372.338793 | libdevmapper version 1.02.185
1372.338873 | kernel device mapper v4.48.0
1372.338888 | DM multipath kernel driver v1.14.0
1372.339153 | unloading rdac checker
1372.339183 | unloading tur checker
1372.339198 | unloading const prioritizer
root@PVE01:~#

 

[waltar]

Try this file and start multipathd, what does "multipath -ll" say ? Do you see a "ls -l /dev/mapper/Huawei-vdisk1" ?

cat /etc/multipath.conf
# device-mapper-multipath configuration file

# For a complete list of the default configuration values, run either:
# # multipath -t
# or
# # multipathd show config

# For a list of configuration options with descriptions, see the
# multipath.conf man page.

defaults {
      user_friendly_names yes
      find_multipaths yes
      enable_foreign "^$"
}

blacklist_exceptions {
property "(SCSI_IDENT_|ID_WWN)"
}

blacklist {
}

# multipaths: settings for individual multipath devices, eg individual MSA volumes
multipaths {
      multipath {
wwid 6c469f01005ec050627d23c600000019
            alias  Huawei-vdisk1
      }
}

# devices: settings for individual storage controllers, eg MSA2050 all volumes
#devices {
#}

# overrides: settings for all devices
#overrides {
#}

 

[xcientist]

First of all, thank you for your response. When I run multipath -ll, nothing shows up, and multipath wasn't working. I started it using multipathd and tried again. When I checked with cat /etc/multipath.conf, it says that there is no such file at this location. Inside the /etc/ directory, there is only a multipath/ folder, and within that folder, there are only two items."


When I created the file, I still see 4 instances, but I understand they are all the same disk. The other channels are backups, but they are reciprocal, and I am only using one of them

 

[waltar]

And in datacenter under disks you see /dev/mapper/mpatha as one 4.4TB disk now ?
I think unfortunately sdb, sdc, sdd and sde will stay further in that gui view as it's some kind of lsblk output and will not hide the devices when a multipath device is active.

 

[xcientist]

The broadcast coming from the SAN or storage is visible across all devices; I can see the real channel, but I can still see the others on the Proxmox side, even if multipath is disabled. The issue doesn't seem to be with Proxmox, I guess.

 

[waltar]

It's not a problem and multipath devices should all show up on a host but perhaps it's a cosmetic issue that they list directly as like different disks (confusing the pve user) and not in submenue lines from a /dev/mapper/<mpath> line like eg somethink like sda with the partitions here.

 

[xcientist]

I have one more question. When I add a SAN storage, I can't see the newly added disk in the GUI after hitting the reload button. It only appears after I restart the server. What are the commands to scan the SAN WWN numbers or SAN disks and make them visible in the GUI without restarting the server?

 

[waltar]

Just type "rescan-scsi-bus.sh" after that new virtual-hotplugged devices shown up by eg. parted -l, lsblk, ls /dev/sd* ...

 

[xcientist]

The rescan-scsi-bus.sh script has been very helpful for me. By disabling the backup channels on my SAN device, I was able to reduce the number of duplicate disks. However, I want to keep the other channels open for redundancy. Since the WWN numbers are the same, it causes clutter with the other channels, and I need a way to hide them.

By the way, I am doing the same operation for VMware, and it sees all SAN channels open as a single disk. It's an interesting situation.
Thank you for your help! I would also like to hear any new ideas or suggestions from others reading this.