Your missing the keyring
See https://forum.proxmox.com/threads/u...-found-error-connecting-to-the-cluster.78961/
Ceph with multipath
- Thread starter christian.g
- Start date
See https://forum.proxmox.com/threads/u...-found-error-connecting-to-the-cluster.78961/
Btw, i changed my script a little bit. My multipath devices are named "mpathXX". If yours are named different, you need to change the script.
Bash:
#!/bin/bash
DRYRUN=0
if [[ "$1" == "--dryrun" ]]
then
DRYRUN=1
fi
readarray -t cdevs < <(ceph-volume lvm list | grep -o -E "/dev/mapper/mpath.*" | sed 's/\/dev\/mapper\///g')
declare -a skipped
for mdev in $(multipath -l)
do
if [[ " ${cdevs[*]} " =~ " ${mdev} " ]]
then
skipped+=("${mdev}")
elif [[ $DRYRUN -eq 1 ]]
then
echo "> Would create OSD from ${mdev}"
else
echo "> Creating OSD from ${mdev}"
pvcreate --metadatasize 250k -y -ff /dev/mapper/${mdev}
vgcreate vg${mdev} /dev/mapper/${mdev}
lvcreate -n lv0 -l 100%FREE vg${mdev}
ceph-volume lvm prepare --data /dev/vg${mdev}/lv0
fi
done
echo "> You need to activate the new OSDs with \"ceph-volume lvm activate --all\""
if [[ "${#skipped[@]}" -gt 0 ]]
then
echo -e "\n> Following devices are already in OSD map and have been skipped.\n"
for d in "${skipped[@]}"
do
echo ">> ${d}"
done
fi
Last edited:
And in case someone needs a script to list OSDs with corresponding devices (incl. multipath and their member disks) as well as not used multipath devices.
Example output:
Bash:
#!/bin/bash
data=$(ceph-volume lvm list --format json)
ar=($(echo $data | jq -r 'to_entries | sort_by(.key | tonumber) | map(.key + ":" + .value[0].lv_size + ":" + (.value[0].devices | join( ",")))' | tr -d '[],"'))
mapped=($(ls /dev/mapper/mpath*))
echo -e "\n------------------------------"
echo -e "OSD count: ${#ar[@]}\n"
for f in "${ar[@]}"
do
values=(${f//:/ })
device=${values[2]}
osd_id=${values[0]}
size=$(numfmt --to iec ${values[1]})
if [[ $device == *"/dev/mapper/mpath"* ]]; then
mpath=$(multipath -ll -v2 $device | grep -o -E "sd[a-z]+" | tr '\n' ' ')
serial=$(multipath -ll -v2 $device | grep -E -o "mpath[a-z]+ \([a-z0-9]+" | sed -E "s/mpath[a-z]+ \(//g")
dev=$(echo $mpath | cut -d ' ' -f 2)
serial2=$(smartctl -a "/dev/${dev}" | grep -i "serial number:" | tr -d ' ' | cut -d ':' -f 2)
echo "OSD: ${osd_id} (${size}) / Device: ${device} / Multi: ${mpath} (Serial: $serial / $serial2)"
for i in "${!mapped[@]}"; do
if [[ ${mapped[i]} = $device ]]; then
unset 'mapped[i]'
fi
done
else
echo "OSD: ${osd_id} (${size}) / Device: ${device}"
fi
done
echo -e "\n------------------------------"
echo "Unmapped device mapper devices: ${#mapped[@]}"
for f in "${mapped[@]}"
do
mpath=$(multipath -ll -v2 $f | grep -o -E "sd[a-z]+" | tr '\n' ' ')
serial=$(multipath -ll -v2 $f | grep -E -o "mpath[a-z]+ \([a-z0-9]+" | sed -E "s/mpath[a-z]+ \(//g")
dev=$(echo $mpath | cut -d ' ' -f 2)
serial2=$(smartctl -a "/dev/${dev}" | grep -i "serial number:" | tr -d ' ' | cut -d ':' -f 2)
echo "Device: ${f} / Multi: ${mpath} (Serial: $serial / $serial2)"
doneExample output:
do your machines boot ok after mapping up the multipath OSDs as my test machine hangs on until I remove the was links then I plug them in and it works. but its boots until I activate the multipath as a OSD.
On my side everything is working flawless.
Make sure your LVM config and HBA Drivers are setup right and included in the initramfs.
Make sure your LVM config and HBA Drivers are setup right and included in the initramfs.
Last edited: