Failed to initialize logical volume. LVM on top iscsi multipath

Oct 8, 2024
6
0
1
the issue appear when migrating vm disk from nfs to lvm on top iscsi. then we test to manually create lv from cli
Code:
root@pve02:~# lvcreate -L 100G -n test_vol Storage-QSAN
  Error reading device /dev/mapper/mpathqsan at 26388279001088 length 4.
  Error reading device /dev/mapper/mpathqsan at 26388279058432 length 4.
  Error reading device /dev/mapper/mpathqsan at 26388279065600 length 24.
  Error reading device /dev/mapper/mpathqsan at 26388279066112 length 512.
  Error writing device value /dev/Storage-QSAN/test_vol at 0 length 4096.
  Failed to initialize logical volume Storage-QSAN/test_vol with value 0 and size 4.00 KiB.
  Aborting. Failed to wipe start of new LV.

Code:
copying volume 'StandardStorage:209/vm-209-disk-1.qcow2' from current storage 'StandardStorage' to target storage 'Standart-QSAN'
create full clone of drive scsi1 (StandardStorage:209/vm-209-disk-1.qcow2)
  Rounding up size to full physical extent 50.01 GiB
  Logical volume "vm-209-disk-1.qcow2" created.
Formatting '/dev/Storage-QSAN/vm-209-disk-1.qcow2', fmt=qcow2 cluster_size=65536 extended_l2=off preallocation=metadata compression_type=zlib size=53687091200 lazy_refcounts=off refcount_bits=16
qemu-img: Failed to flush the L2 table cache: Input/output error
qemu-img: Failed to flush the refcount block cache: Input/output error
  Logical volume "vm-209-disk-1.qcow2" successfully removed.
TASK ERROR: storage migration failed: unable to create image: qemu-img: /dev/Storage-QSAN/vm-209-disk-1.qcow2: Could not resize image: Allocating clusters failed: Input/output error

Status of the multipath
Code:
root@pve02:~# multipath -ll
mpathqsan (3201a0013780ffa00) dm-54 Qsan,XS3212
size=24T features='1 queue_if_no_path' hwhandler='1 alua' wp=rw
`-+- policy='round-robin 0' prio=50 status=active
  |- 10:0:0:0 sdf 8:80 active ready running
  `- 9:0:0:0  sde 8:64 active ready running

vgs result, there is still free remaining
Code:
root@pve02:~# vgs
  Error reading device /dev/mapper/mpathqsan at 26388279001088 length 4.
  Error reading device /dev/mapper/mpathqsan at 26388279058432 length 4.
  Error reading device /dev/mapper/mpathqsan at 26388279065600 length 24.
  Error reading device /dev/mapper/mpathqsan at 26388279066112 length 512.
  VG                                        #PV #LV #SN Attr   VSize    VFree 
  Storage-QSAN                                1  60   0 wz--n-  <24.00t <21.22t
  ceph-2757eb11-5a6d-476d-97e8-e59fd8de01ae   1   1   0 wz--n-   <4.55t      0
  pve                                         1   2   0 wz--n- <930.00g      0

We used qsan storage with iscsi protocol and thin pool lun. and checked on the qsan dashboard there are no issued regarding space or locked lun.

the logs on systemd
Code:
Sep 15 15:04:48 pve01 kernel: sd 9:0:0:0: alua: supports implicit TPGS
Sep 15 15:04:48 pve01 kernel: sd 9:0:0:0: alua: No device descriptors found
Sep 15 15:04:48 pve01 kernel: sd 9:0:0:0: alua: port group 0 rel port 1
Sep 15 15:04:48 pve01 kernel: sd 9:0:0:0: [sde] Very big device. Trying to use READ CAPACITY(16).
Sep 15 15:04:48 pve01 kernel: sd 9:0:0:0: alua: transition timeout set to 60 seconds
Sep 15 15:04:49 pve01 kernel: sd 8:0:0:0: [sdd] tag#69 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_OK cmd_age=0s
Sep 15 15:04:49 pve01 kernel: sd 8:0:0:0: [sdd] tag#69 Sense Key : Data Protect [current]
Sep 15 15:04:49 pve01 kernel: sd 8:0:0:0: [sdd] tag#69 Add. Sense: Space allocation failed write protect
Sep 15 15:04:49 pve01 kernel: sd 8:0:0:0: [sdd] tag#69 CDB: Read(16) 88 00 00 00 00 0b ff ff ff 00 00 00 01 00 00 00
Sep 15 15:04:49 pve01 kernel: critical space allocation error, dev sdd, sector 51539607296 op 0x0:(READ) flags 0x4200 phys_seg 32 prio class 2
Sep 15 15:04:49 pve01 kernel: critical space allocation error, dev dm-23, sector 51539607296 op 0x0:(READ) flags 0x0 phys_seg 32 prio class 2
Sep 15 15:04:49 pve01 kernel: sd 9:0:0:0: [sde] tag#63 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_OK cmd_age=0s
Sep 15 15:04:49 pve01 kernel: sd 9:0:0:0: [sde] tag#63 Sense Key : Data Protect [current]
Sep 15 15:04:49 pve01 kernel: sd 9:0:0:0: [sde] tag#63 Add. Sense: Space allocation failed write protect
Sep 15 15:04:49 pve01 kernel: sd 9:0:0:0: [sde] tag#63 CDB: Read(16) 88 00 00 00 00 0b ff ff ff 00 00 00 01 00 00 00
Sep 15 15:04:49 pve01 kernel: critical space allocation error, dev sde, sector 51539607296 op 0x0:(READ) flags 0x4200 phys_seg 32 prio class 2
Sep 15 15:04:49 pve01 kernel: critical space allocation error, dev dm-23, sector 51539607296 op 0x0:(READ) flags 0x0 phys_seg 32 prio class 2
Sep 15 15:04:49 pve01 kernel: sd 8:0:0:0: [sdd] tag#26 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_OK cmd_age=0s
Sep 15 15:04:49 pve01 kernel: sd 8:0:0:0: [sdd] tag#26 Sense Key : Data Protect [current]
Sep 15 15:04:49 pve01 kernel: sd 8:0:0:0: [sdd] tag#26 Add. Sense: Space allocation failed write protect
Sep 15 15:04:49 pve01 kernel: sd 8:0:0:0: [sdd] tag#26 CDB: Read(16) 88 00 00 00 00 0b ff ff ff 00 00 00 01 00 00 00
Sep 15 15:04:49 pve01 kernel: critical space allocation error, dev sdd, sector 51539607296 op 0x0:(READ) flags 0x4200 phys_seg 32 prio class 2
Sep 15 15:04:49 pve01 kernel: critical space allocation error, dev dm-23, sector 51539607296 op 0x0:(READ) flags 0x0 phys_seg 32 prio class 2
Sep 15 15:04:49 pve01 kernel: sd 9:0:0:0: [sde] tag#110 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_OK cmd_age=0s
Sep 15 15:04:49 pve01 kernel: sd 9:0:0:0: [sde] tag#110 Sense Key : Data Protect [current]
Sep 15 15:04:49 pve01 kernel: sd 9:0:0:0: [sde] tag#110 Add. Sense: Space allocation failed write protect
Sep 15 15:04:49 pve01 kernel: sd 9:0:0:0: [sde] tag#110 CDB: Read(16) 88 00 00 00 00 0b ff ff ff 00 00 00 01 00 00 00
Sep 15 15:04:49 pve01 kernel: critical space allocation error, dev sde, sector 51539607296 op 0x0:(READ) flags 0x4200 phys_seg 32 prio class 2
Sep 15 15:04:49 pve01 kernel: critical space allocation error, dev dm-23, sector 51539607296 op 0x0:(READ) flags 0x0 phys_seg 32 prio class 2
Sep 15 15:04:49 pve01 pveproxy[2501214]: got inotify poll request in wrong process - disabling inotify
Sep 15 15:04:49 pve01 kernel: sd 8:0:0:0: [sdd] tag#2 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_OK cmd_age=0s
Sep 15 15:04:49 pve01 kernel: sd 8:0:0:0: [sdd] tag#2 Sense Key : Data Protect [current]
Sep 15 15:04:49 pve01 kernel: sd 8:0:0:0: [sdd] tag#2 Add. Sense: Space allocation failed write protect
Sep 15 15:04:49 pve01 kernel: sd 8:0:0:0: [sdd] tag#2 CDB: Read(16) 88 00 00 00 00 0b ff ff ff 00 00 00 01 00 00 00
Sep 15 15:04:49 pve01 kernel: critical space allocation error, dev sdd, sector 51539607296 op 0x0:(READ) flags 0x4200 phys_seg 32 prio class 2
Sep 15 15:04:49 pve01 kernel: critical space allocation error, dev dm-23, sector 51539607296 op 0x0:(READ) flags 0x0 phys_seg 32 prio class 2
Sep 15 15:04:49 pve01 kernel: sd 9:0:0:0: [sde] tag#77 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_OK cmd_age=0s
Sep 15 15:04:49 pve01 kernel: sd 9:0:0:0: [sde] tag#77 Sense Key : Data Protect [current]
Sep 15 15:04:49 pve01 kernel: sd 9:0:0:0: [sde] tag#77 Add. Sense: Space allocation failed write protect
Sep 15 15:04:49 pve01 kernel: sd 9:0:0:0: [sde] tag#77 CDB: Read(16) 88 00 00 00 00 0b ff ff ff 00 00 00 01 00 00 00
Sep 15 15:04:49 pve01 kernel: sd 8:0:0:0: [sdd] tag#3 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_OK cmd_age=0s
Sep 15 15:04:49 pve01 kernel: sd 8:0:0:0: [sdd] tag#3 Sense Key : Data Protect [current]
Sep 15 15:04:49 pve01 kernel: sd 8:0:0:0: [sdd] tag#3 Add. Sense: Space allocation failed write protect
Sep 15 15:04:49 pve01 kernel: sd 8:0:0:0: [sdd] tag#3 CDB: Read(16) 88 00 00 00 00 0b ff ff ff 00 00 00 01 00 00 00
Sep 15 15:04:49 pve01 kernel: sd 9:0:0:0: [sde] tag#78 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_OK cmd_age=0s
Sep 15 15:04:49 pve01 kernel: sd 9:0:0:0: [sde] tag#78 Sense Key : Data Protect [current]
Sep 15 15:04:49 pve01 kernel: sd 9:0:0:0: [sde] tag#78 Add. Sense: Space allocation failed write protect
Sep 15 15:04:49 pve01 kernel: sd 9:0:0:0: [sde] tag#78 CDB: Read(16) 88 00 00 00 00 0b ff ff ff 00 00 00 01 00 00 00
 
We used qsan storage with iscsi protocol and thin pool lun.

Ah, seems like using an LVM thin pool might be at fault here.

In order to be able to use an LVM thin pool on top of iSCSI, you need to ensure to not share it across nodes, meaning that only a single node can use / access the LUN. You cannot put a thin pool on a volume group that is shared across nodes, since that can lead to data loss.

Looking at your logs, I can see Sense Key : Data Protect and Add. Sense: Space allocation failed write protect showing up. My guess is that this means that your thin-pool has overcommitted to your storage, which means that you probably ran out of disk space that the thin-pool can use. If that is really the case, then you'll have to add more disks and (most likely) reboot, so that your LUN is taken out of write-protect mode.

Is there perhaps any way to check how much storage you are currently using?



All that being said, I recommend setting up "regular" LVM instead. See our Multipath Guide for more information, if you haven't already. Regular LVM now also supports Snapshots as Volume-Chain, but note that it's still a tech preview (as of writing).
 
Coincidentally we just finished and published on article on this topic!



Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox
 
Ah, seems like using an LVM thin pool might be at fault here.

In order to be able to use an LVM thin pool on top of iSCSI, you need to ensure to not share it across nodes, meaning that only a single node can use / access the LUN. You cannot put a thin pool on a volume group that is shared across nodes, since that can lead to data loss.

Looking at your logs, I can see Sense Key : Data Protect and Add. Sense: Space allocation failed write protect showing up. My guess is that this means that your thin-pool has overcommitted to your storage, which means that you probably ran out of disk space that the thin-pool can use. If that is really the case, then you'll have to add more disks and (most likely) reboot, so that your LUN is taken out of write-protect mode.

Is there perhaps any way to check how much storage you are currently using?



All that being said, I recommend setting up "regular" LVM instead. See our Multipath Guide for more information, if you haven't already. Regular LVM now also supports Snapshots as Volume-Chain, but note that it's still a tech preview (as of writing).
Sorry before that, we mean thin pool lun is stated only on QSAN Storage, but on the proxmox side we setup LVM on top ISCSI without thin. And the storage usage from vgs is:
Code:
VG                                        #PV #LV #SN Attr   VSize    VFree
Storage-QSAN                                1  60   0 wz--n-  <24.00t <21.22t

Coincidentally we just finished and published on article on this topic!



Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox
1789530654163.png
we setup the standard lvm, not lvm-thin on proxmox side. The thin pool is configured on QSAN Storage only.
1789530723482.png