LVM over ISCSI Issues

[Dombok]

Good Day,

We are running Proxmox VE 7.1-10 and make use of LVM over ISCSI. The LVM randomly stoppped working and showed as Unkown / Inactive and no disks was showing. We managed to fix the issue that caused the LVM to stop working but now the disks is not seen as bootable disk and when doing a fsck it gives a bad superblock error.

The superblock could not be read or does not describe a valid ext2/ext3/ext4
filesystem. If the device is valid and it really contains an ext2/ext3/ext4
filesystem (and not swap or ufs or something else), then the superblock
is corrupt, and you might try running e2fsck with an alternate superblock:
e2fsck -b 8193 <device>
or
e2fsck -b 32768 <device>

Any advise on how to resolve this? Not sure what info should be provided so I will post any further info required.

Thanks in advance for any assistance.

 

[bbgeek17]

The LVM randomly stoppped working and showed as Unkown / Inactive and no disks was showing. We managed to fix the issue that caused the LVM to stop working

This is an important part of the story, can you expand on what exactly broke and how you fixed it?

the disks is not seen as bootable disk

Do you mean within the VM? Have you checked that your "pvesm status" is working without errors? Are you sure VM sees the disk and that boot settings are set correctly?

when doing a fsck it gives a bad superblock error.

Where are you running fsck from and against what device?

What is your storage configuration? "cat /etc/pve/storage.cfg"
What is your VM configuration? "qm config [vmid]"
What is your storage status? "pvesm status"
What is your LVM status? "lsblk", "lvs"
What is the output of "fdisk -l [dm device]" of the VM boot disk?
Are there any error when starting VM : terminal1 "journalctl -f" , terminal2 "qm start [vmid]"

---

Blockbridge: Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[Dombok]

This is an important part of the story, can you expand on what exactly broke and how you fixed it?

We found a partition that has been created and kept getting an error that /dev/sde can't be used as it contains a partition. We removed the partition and did a pvck /dev/sde and vgrestore "LV NAME". After this was done all the disks was showing and LVM showed as active.

Do you mean within the VM? Have you checked that your "pvesm status" is working without errors? Are you sure VM sees the disk and that boot settings are set correctly?

Yes, within the VM. Boot settings is correct but when booting VM it shows No bootable disk found.

Where are you running fsck from and against what device?

Was running it from the Proxmox host on the VM disk for example /dev/mapper/"LV-NAME"/VM-1XX-Disc-0

What is your storage configuration? "cat /etc/pve/storage.cfg"

Everything there looks fine accept the LVM has no BASE anymore. Posting the config of the ISCSI / LVM that the issue
iscsi: ZAGT-TCDC-QNAS
portal 10.111.253.131
target iqn.2004-04.com.qnap:ts-1886xu-rp:iscsi.proxmox.66730e
content images

lvm: ZAGT_TCDC_QNAS_NL_PROX_LUN01
vgname ZAGT_TCDC_QNAS_NL_PROX_LUN01
content images
nodes pve01
shared 0

What is your VM configuration? "qm config [vmid]"
What is your storage status? "pvesm status"
What is your LVM status? "lsblk", "lvs"
What is the output of "fdisk -l [dm device]" of the VM boot disk?
Are there any error when starting VM : terminal1 "journalctl -f" , terminal2 "qm start [vmid]"

Will provide updates on these as the LVM is inactive again for some reason.

 

[bbgeek17]

On first read, it sounds like things are being manipulated directly, behind the PVE/QEMU, and generally are not stable.
I'd recommend stabilizing the system (iSCSI/LVM) and restoring the latest known good backup.

Another option is to create a new bootable disk/VM and then move the corrupted one there, it might allow you to recover the data.

---

Blockbridge: Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox