Disks showing up twice / multipath / ceph

[Saxophone]

This is my first time working with Proxmox

I am trying to build a 4 node cluster with proxmox and install ceph. I want to have a single location for my windows machines to access plex, files, photos, music.
After 4 day I am still having problems.

I have 4 used supermicro servers
2x 240GB SSD in raid 1 for boot
2 x 1TB nvme on motherboard (not being used at the moment)
with 2 x 10g nic's, 2 x 1g nic's, 1 x ipmi
1 x hba 9500i plugged into the hard drive backplane 12 slots only 2 filled for setup

I have installed proxmox over and over again and I might be starting to getting good at it.
I set up four nodes pve1, pve2, pve3, pve4

I have clustered the nodes together. I kept having problems and error messages but got it running on 1 of the 1g nic's. I'm hoping once everything is running I will be able to get the 10g nic's working for frontend and backend ceph networks.

I have installed Ceph on the 4 nodes added mon's and mgr's

All pve's are having the same problems. When trying to Create Osd's it fails with the message (command 'ceph-volume lvm create --crush-device-class hdd --data /dev/sdc' failed: exit code 1)

I think the problem is that the hard drives are showing up twice as sdc and sde

I did a test and was able to make an osd out of 1 of the nvme drives just fine.

I've looked into multipath and making a config file but it might not be correct and multipath doesn't seem to be working

Please help I am getting very frustrated with Proxmox at this point.



Here is my /etc/multipath.conf

defaults {
polling_interval 2
path_selector "round-robin 0"
path_grouping_policy multibus
uid_attribute ID_SERIAL
rr_min_io 100
failback immediate
no_path_retry queue
user_friendly_names yes
}

blacklist {
wwid .*
}

blacklist_exceptions {
wwid "35000c500e8d8ab61"
wwid "350014ee20b402e3e"
}

multipaths {
multipath {
wwid "35000c500e8d8ab61"
alias A1
}
multipath {
wwid "350014ee20b402e3e"
alias B1
}
}



multipath -ll returns no output



multipath -v3 (output)

root@pve1:~# multipath -v3
5102.402741 | set open fds limit to 1048576/1048576
5102.402802 | loading /lib/multipath/libchecktur.so checker
5102.402897 | checker tur: message table size = 3
5102.402912 | loading /lib/multipath/libprioconst.so prioritizer
5102.403017 | _init_foreign: foreign library "nvme" is not enabled
5102.411229 | nvme0n1: size = 2000409264
5102.411386 | nvme0n1: vendor = NVME
5102.411392 | nvme0n1: product = SPCC M.2 PCIe SSD
5102.411397 | nvme0n1: serial = NF12401221700158
5102.411401 | nvme0n1: rev = V1221B0
5102.411915 | nvme0n1: uid_attribute = ID_WWN (setting: storage device configuration)
5102.411921 | nvme0n1: recheck_wwid = 1 (setting: multipath.conf defaults/devices section)
5102.412037 | nvme0n1: 59258 cyl, 64 heads, 32 sectors/track, start at 0
5102.412044 | nvme0n1: detect_checker = yes (setting: multipath internal)
5102.412049 | nvme0n1: path_checker = none (setting: storage device configuration)
5102.412052 | nvme0n1: checker timeout = 30 s (setting: multipath internal)
5102.412056 | nvme0n1: none state = up
5102.412060 | nvme0n1: uid = eui.00000000000000014868340000000001 (udev)
5102.412071 | nvme0n1: wwid eui.00000000000000014868340000000001 blacklisted
5102.412190 | nvme1n1: size = 1953525168
5102.412340 | nvme1n1: vendor = NVME
5102.412346 | nvme1n1: product = CT1000P3PSSD8
5102.412350 | nvme1n1: serial = 2417489109E3
5102.412355 | nvme1n1: rev = P9CR413
5102.412872 | nvme1n1: uid_attribute = ID_WWN (setting: storage device configuration)
5102.412878 | nvme1n1: recheck_wwid = 1 (setting: multipath.conf defaults/devices section)
5102.412966 | nvme1n1: 36365 cyl, 64 heads, 32 sectors/track, start at 0
5102.412971 | nvme1n1: detect_checker = yes (setting: multipath internal)
5102.412975 | nvme1n1: path_checker = none (setting: storage device configuration)
5102.412979 | nvme1n1: checker timeout = 30 s (setting: multipath internal)
5102.412982 | nvme1n1: none state = up
5102.412986 | nvme1n1: uid = eui.000000000000000100a07524489109e3 (udev)
5102.412992 | nvme1n1: wwid eui.000000000000000100a07524489109e3 blacklisted
5102.413129 | sda: size = 468862128
5102.413260 | sda: vendor = ATA
5102.413283 | sda: product = KINGSTON SA400S3
5102.413311 | sda: rev = 66.5
5102.413762 | sda: h:b:t:l = 2:0:0:0
5102.414035 | sda: tgt_node_name = ata-3.00
5102.414042 | sda: uid_attribute = ID_SERIAL (setting: multipath.conf defaults/devices section)
5102.414047 | sda: recheck_wwid = 1 (setting: multipath.conf defaults/devices section)
5102.414192 | sda: 29185 cyl, 255 heads, 63 sectors/track, start at 0
5102.414198 | sda: vpd_vendor_id = 0 "undef" (setting: multipath internal)
5102.414216 | sda: serial = 50026B77857177A3
5102.414222 | sda: detect_checker = yes (setting: multipath internal)
5102.414291 | sda: path_checker = tur (setting: multipath internal)
5102.414297 | sda: checker timeout = 30 s (setting: kernel sysfs)
5102.414314 | sda: tur state = up
5102.414320 | sda: uid = KINGSTON_SA400S37240G_50026B77857177A3 (udev)
5102.414325 | sda: wwid KINGSTON_SA400S37240G_50026B77857177A3 blacklisted
5102.414684 | sdb: size = 468862128
5102.414813 | sdb: vendor = ATA
5102.414836 | sdb: product = KINGSTON SA400S3
5102.414861 | sdb: rev = T1A3
5102.415311 | sdb: h:b:t:l = 3:0:0:0
5102.415576 | sdb: tgt_node_name = ata-4.00
5102.415583 | sdb: uid_attribute = ID_SERIAL (setting: multipath.conf defaults/devices section)
5102.415588 | sdb: recheck_wwid = 1 (setting: multipath.conf defaults/devices section)
5102.415722 | sdb: 29185 cyl, 255 heads, 63 sectors/track, start at 0
5102.415728 | sdb: vpd_vendor_id = 0 "undef" (setting: multipath internal)
5102.415743 | sdb: serial = 50026B7785905806
5102.415750 | sdb: detect_checker = yes (setting: multipath internal)
5102.415820 | sdb: path_checker = tur (setting: multipath internal)
5102.415826 | sdb: checker timeout = 30 s (setting: kernel sysfs)
5102.415841 | sdb: tur state = up
5102.415858 | sdb: uid = KINGSTON_SA400S37240G_50026B7785905806 (udev)
5102.415867 | sdb: wwid KINGSTON_SA400S37240G_50026B7785905806 blacklisted
5102.416253 | sdc: size = 15628053168
5102.416413 | sdc: vendor = ATA
5102.416438 | sdc: product = ST8000DM004-2U91
5102.416462 | sdc: rev = 0001
5102.416900 | sdc: h:b:t:l = 18:0:0:0
5102.417059 | sdc: tgt_node_name = 0x500304801eee03c2
5102.417065 | sdc: uid_attribute = ID_SERIAL (setting: multipath.conf defaults/devices section)
5102.417068 | sdc: recheck_wwid = 1 (setting: multipath.conf defaults/devices section)
5102.417189 | sdc: 55297 cyl, 255 heads, 63 sectors/track, start at 0
5102.417195 | sdc: vpd_vendor_id = 0 "undef" (setting: multipath internal)
5102.417211 | sdc: serial = ZR15TYJJ
5102.417217 | sdc: detect_checker = yes (setting: multipath internal)
5102.419846 | sdc: path_checker = tur (setting: multipath internal)
5102.419853 | sdc: checker timeout = 30 s (setting: kernel sysfs)
5102.420026 | sdc: tur state = up
5102.420031 | sdc: uid = 35000c500e8d8ab61 (udev)
5102.420061 | sdc: wwid 35000c500e8d8ab61 whitelisted
5102.420067 | sdc: detect_prio = yes (setting: multipath internal)
5102.420076 | sdc: prio = const (setting: multipath internal)
5102.420081 | sdc: prio args = "" (setting: multipath internal)
5102.420088 | sdc: const prio = 1
5102.420220 | sdd: size = 7813971633
5102.420371 | sdd: vendor = ATA
5102.420396 | sdd: product = WDC WD40EZRX-00S
5102.420421 | sdd: rev = 0A80
5102.420871 | sdd: h:b:t:l = 18:0:1:0
5102.421034 | sdd: tgt_node_name = 0x500304801eee03c5
5102.421040 | sdd: uid_attribute = ID_SERIAL (setting: multipath.conf defaults/devices section)
5102.421043 | sdd: recheck_wwid = 1 (setting: multipath.conf defaults/devices section)
5102.421167 | sdd: 27645 cyl, 255 heads, 63 sectors/track, start at 0
5102.421173 | sdd: vpd_vendor_id = 0 "undef" (setting: multipath internal)
5102.421189 | sdd: serial = WD-WCC4E7HHCCUD
5102.421196 | sdd: detect_checker = yes (setting: multipath internal)
5102.421618 | sdd: path_checker = tur (setting: multipath internal)
5102.421624 | sdd: checker timeout = 30 s (setting: kernel sysfs)
5102.421711 | sdd: tur state = up
5102.421717 | sdd: uid = 350014ee20b402e3e (udev)
5102.421763 | sdd: wwid 350014ee20b402e3e whitelisted
5102.421769 | sdd: detect_prio = yes (setting: multipath internal)
5102.421774 | sdd: prio = const (setting: multipath internal)
5102.421779 | sdd: prio args = "" (setting: multipath internal)
5102.421785 | sdd: const prio = 1
5102.421916 | sde: size = 15628053168
5102.422080 | sde: vendor = ATA
5102.422109 | sde: product = ST8000DM004-2U91
5102.422135 | sde: rev = 0001
5102.422611 | sde: h:b:t:l = 18:0:3:0
5102.422753 | sde: tgt_node_name = 0x500304801eee03c2
5102.422759 | sde: uid_attribute = ID_SERIAL (setting: multipath.conf defaults/devices section)
5102.422764 | sde: recheck_wwid = 1 (setting: multipath.conf defaults/devices section)
5102.422887 | sde: 55297 cyl, 255 heads, 63 sectors/track, start at 0
5102.422893 | sde: vpd_vendor_id = 0 "undef" (setting: multipath internal)
5102.422909 | sde: serial = ZR15TYJJ
5102.422916 | sde: detect_checker = yes (setting: multipath internal)
5102.425164 | sde: path_checker = tur (setting: multipath internal)
5102.425170 | sde: checker timeout = 30 s (setting: kernel sysfs)
5102.425305 | sde: tur state = up
5102.425311 | sde: uid = 35000c500e8d8ab61 (udev)
5102.425315 | sde: wwid 35000c500e8d8ab61 whitelisted
5102.425319 | sde: detect_prio = yes (setting: multipath internal)
5102.425322 | sde: prio = const (setting: multipath internal)
5102.425325 | sde: prio args = "" (setting: multipath internal)
5102.425329 | sde: const prio = 1
5102.425451 | sdf: size = 7813971633
5102.425606 | sdf: vendor = ATA
5102.425630 | sdf: product = WDC WD40EZRX-00S
5102.425653 | sdf: rev = 0A80
5102.426134 | sdf: h:b:t:l = 18:0:4:0
5102.426273 | sdf: tgt_node_name = 0x500304801eee03c5
5102.426278 | sdf: uid_attribute = ID_SERIAL (setting: multipath.conf defaults/devices section)
5102.426282 | sdf: recheck_wwid = 1 (setting: multipath.conf defaults/devices section)
5102.426395 | sdf: 27645 cyl, 255 heads, 63 sectors/track, start at 0
5102.426400 | sdf: vpd_vendor_id = 0 "undef" (setting: multipath internal)
5102.426417 | sdf: serial = WD-WCC4E7HHCCUD
5102.426420 | sdf: detect_checker = yes (setting: multipath internal)
5102.426842 | sdf: path_checker = tur (setting: multipath internal)
5102.426848 | sdf: checker timeout = 30 s (setting: kernel sysfs)
5102.426932 | sdf: tur state = up
5102.426937 | sdf: uid = 350014ee20b402e3e (udev)
5102.426942 | sdf: wwid 350014ee20b402e3e whitelisted
5102.426946 | sdf: detect_prio = yes (setting: multipath internal)
5102.426949 | sdf: prio = const (setting: multipath internal)
5102.426954 | sdf: prio args = "" (setting: multipath internal)
5102.426960 | sdf: const prio = 1
5102.427042 | loop0: device node name blacklisted
5102.427107 | loop1: device node name blacklisted
5102.427168 | loop2: device node name blacklisted
5102.427227 | loop3: device node name blacklisted
5102.427286 | loop4: device node name blacklisted
5102.427346 | loop5: device node name blacklisted
5102.427405 | loop6: device node name blacklisted
5102.427464 | loop7: device node name blacklisted
===== paths list =====
uuid hcil dev dev_t pri dm_st chk_st vend/prod/rev dev_st
35000c500e8d8ab61 18:0:0:0 sdc 8:32 1 undef undef ATA,ST8000DM004-2U91 unknown
350014ee20b402e3e 18:0:1:0 sdd 8:48 1 undef undef ATA,WDC WD40EZRX-00S unknown
35000c500e8d8ab61 18:0:3:0 sde 8:64 1 undef undef ATA,ST8000DM004-2U91 unknown
350014ee20b402e3e 18:0:4:0 sdf 8:80 1 undef undef ATA,WDC WD40EZRX-00S unknown
5102.429336 | multipath-tools v0.9.4 (12/19, 2022)
5102.429348 | libdevmapper version 1.02.185
5102.429525 | kernel device mapper v4.48.0
5102.429540 | DM multipath kernel driver v1.14.0
5102.429561 | sdc: udev property ID_WWN whitelisted
5102.429568 | sdc: wwid 35000c500e8d8ab61 whitelisted
5102.429603 | wwid 35000c500e8d8ab61 not in wwids file, skipping sdc
5102.429609 | sdc: orphan path, only one path
5102.429622 | sdd: udev property ID_WWN whitelisted
5102.429632 | sdd: wwid 350014ee20b402e3e whitelisted
5102.429660 | wwid 350014ee20b402e3e not in wwids file, skipping sdd
5102.429666 | sdd: orphan path, only one path
5102.429677 | sde: udev property ID_WWN whitelisted
5102.429685 | sde: wwid 35000c500e8d8ab61 whitelisted
5102.429712 | wwid 35000c500e8d8ab61 not in wwids file, skipping sde
5102.429718 | sde: orphan path, only one path
5102.429729 | sdf: udev property ID_WWN whitelisted
5102.429737 | sdf: wwid 350014ee20b402e3e whitelisted
5102.429763 | wwid 350014ee20b402e3e not in wwids file, skipping sdf
5102.429768 | sdf: orphan path, only one path
5102.429871 | unloading none checker
5102.429876 | unloading tur checker
5102.429901 | unloading const prioritize

 

[Saxophone]

Update: I solved my own problem. I never did get multipath do work but while continuing my research I came upon a solution. My 950016i HBA has 2 x SFF-8654 ports. The port is supposed to handle 8 hard drives and there are to for a total of 16 drives. With break out cables the took the 2x SFF-8654 ports to 4x SFF-8643 ports. There are 4x 8643 ports on the back of the. I made a logical assumption that if I had 12 hard drives I would need to plug in at least 3 of the connectors. I plugged in all 4 which wasn't necessarily the whole problem. I turns out that the backplanes that I have are meant to be daisy changed together. The bottom 2 are incoming from the HBA and the upper 2 are meant to be plugged into the bottom of the next backplane if you have one. With only one you are only supposed to use the bottom 2 ports and leave the upper 2 open. I would think this would mean that you would only get to use 8 hard drives but it turns out that the server sees all 12. I don't know if I will get the full bandwidth of the HBA but if not hopefully its not too bad. At least now I can access all 12 drives from Ceph inside of Proxmox with no more errors.