[SOLVED] I broke LVM on iSCSI in a cluster. Can it be fixed?

[Oldsocks]

I think I'm in a bit of trouble here.

We have a san and it took a lot of effort to get working, but eventually it did through some miracle. I added LVM on top and the first two memebs of the cluster saw it without trouble.

Then I realised the other members of the cluster hadn't been set up to see the storage network. So I made the required changes, reboots all round, but they couldn't connect tot he storage. I figured I'd step back and re-create the storage so the setting would propagate through the cluster properly. I deleted the LVM storage that was on top of the iSCSI, rebooted, and tried to recreate it.

Now it doesn't work. The fundamental iscsi storage is unchanged, but when I try to add LVM I select the iscsi storage in base storage, but the base volume doesn't expand - there's nothing there.


iscsiadmin session query returns

iscsiadm -m session -o show

tcp: [1] 10.10.250.120:3260,3 iqn.2015-11.com.hpe:storage.msa2050.112345226d (non-flash)

fdisk -l returns

Disk /dev/sdc: 5.5 TiB, 5999998009344 bytes, 11718746112 sectors
Disk model: MSA 2050 SAN
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 1048576 bytes

I am prety lost here; any help gratefully received.

 

[xPakrikx]

Then just create new one

pvcreate /dev/sdc
vgcreate new_vol_group /dev/sdc

And then use "Existing volume groups" and select created VG.

But yes this looks like bug from Proxmox 6.0. In 5.4 you can see volume and you can create new VG from webGUI. But you can try only "pvcreate /dev/sdc" and then add new LVM over webGui.

also check LVM configuration with commands pvs and vgs.

 

[Oldsocks]

Thank you for the reply; it steered me in the right direction so far! I removed everything, detached all the iscsi sessions, created a new storage entity, and while I couldn't create a new LVM storage in the gui I was able to using that command. However, it looks like the new storage hasn't propagated throughout the cluster - it appears as an option on all the other members but only works on the first server, where I created it. I guess the next and fnial question is how do I make the other cluster members bring the base storage online? I think I'll start with pvesm - scan and go from there.

 

[xPakrikx]

• Check if you have access to volume from every node in cluster (ACLs on storage).
• Check disk over fdisk -l on every node
• Check lvm on every node with pvs and vgs
• Check if you have selected all nodes in iSCSI/LVM proxmox webui storage settings.

 

[Oldsocks]

After removing the lvm and recreating it, it seems to be prety much ok now. One member has intermittent connetivity issues, could be something weird with the config, could be the switch is garbage. My next step is to look into MPIO which might sort the problem if it is the switch. Thank you so much for the responses.