Problem with thin lvm datastore

[christophe.aron]

Hi,

I'm testing Proxmox VE in order to migrate from Ovirt.
I have setup an environnement with 3 nodes and 1 DELL SVC 2020.
I encounter an issue when I setup the storage volume.
I saw with native ISCSI that snapshot is not avaliable and to improve this feature we need to setup thin pool on iscsi volume.
I configure the storage datastore like this :
Setup isci initiator to permit to discover the lun
Setup multipathing configuration on 3 nodes to permit to map the lun on system
Setup Thin pool LVM configuration following this thread :
https://forum.proxmox.com/threads/h...pool-and-vz-directory-on-the-same-disk.62901/
After that I can setup storage in Proxmox UI and the storage is fully visible on 3 nodes.
I can create VM, migrate disk, make clone, etc.
But after, some days of usage, an Thin pool is not active from lvs command ?
Then, there is no possibility to make any task on volume (lvchange -ay vg, lvconvert --repair, etc.)

below some errors output :
when moving disk from local storage to thin pool:

create full clone of drive virtio0 (vm-export:104/vm-104-disk-0.qcow2)
  Thin pool vg--prod-lvthin--prod-tpool (253:9) transaction_id is 10, while expected 13.
TASK ERROR: storage migration failed: lvcreate 'vg-prod/vm-104-disk-0' error:   Failed to suspend vg-prod/lvthin-prod with queued messages.

when make snapshot :

 Thin pool vg--prod-lvthin--prod-tpool (253:9) transaction_id is 11, while expected 13.
TASK ERROR: lvcreate 'vg-prod/vm-108-state-test' error:   Failed to suspend vg-prod/lvthin-prod with queued messages.

What I see is the transaction id is mismatch
There is a misconfiguration, needs help !!


There is my config :

root@xxxx:~# pveversion
pve-manager/6.4-13/9f411e79 (running kernel: 5.4.114-1-pve)

~# lvs
  LV                         VG                 Attr       LSize    Pool          Origin        Data%  Meta%  Move Log Cpy%Sync Convert
  home                       PSRXMASTERMGT01-vg -wi-ao----   30,00g                                                                   
  root                       PSRXMASTERMGT01-vg -wi-ao----   23,28g                                                                   
  swap_1                     PSRXMASTERMGT01-vg -wi-ao----  190,46g                                                                   
  tmp                        PSRXMASTERMGT01-vg -wi-ao----   <1,86g                                                                   
  var                        PSRXMASTERMGT01-vg -wi-ao---- <243,69g                                                                   
  lvthin-pilote              vg-pilote          twi-a-tz-- <798,00g                             0,00   1,65                           
  snap_vm-101-disk-0_agile_3 vg-pilote          Vri---tz-k  100,00g vmthin-pilote vm-101-disk-0                                       
  snap_vm-101-disk-1_agile_3 vg-pilote          Vri---tz-k   70,00g vmthin-pilote vm-101-disk-1                                       
  vm-100-disk-0              vg-pilote          Vwi---tz--   80,00g vmthin-pilote                                                     
  vm-101-disk-0              vg-pilote          Vwi---tz--  100,00g vmthin-pilote                                                     
  vm-101-disk-1              vg-pilote          Vwi---tz--   70,00g vmthin-pilote                                                     
  vm-101-state-Agile         vg-pilote          Vwi---tz--  <16,49g vmthin-pilote                                                     
  vm-101-state-version_agile vg-pilote          Vwi---tz--  <16,49g vmthin-pilote                                                     
  vm-102-disk-0              vg-pilote          Vwi---tz--   70,00g vmthin-pilote                                                     
  vm-102-disk-1              vg-pilote          Vwi---tz--  100,00g vmthin-pilote                                                     
  vm-103-disk-0              vg-pilote          Vwi---tz--   50,00g vmthin-pilote                                                     
  vmthin-pilote              vg-pilote          twi---tz-- <998,00g                                                                   
  vmthin-pilote_meta0        vg-pilote          -wi-------    1,00g                                                                   
  lvthin-prod                vg-prod            twi-aotz-- <998,00g                             6,84   2,51                           
  lvthin-prod_meta0          vg-prod            -wi-a-----    1,00g                                                                   
  vm-105-disk-0              vg-prod            Vwi-a-tz--   60,00g lvthin-prod                 100,00                                 
  vm-106-disk-1              vg-prod            Vwi---tz--  100,00g lvthin-prod                                                       
  vm-107-disk-0              vg-prod            Vwi-a-tz--   60,00g lvthin-prod                 0,01                                   
  vm-108-disk-0              vg-prod            Vwi---tz--  100,00g lvthin-prod                                                       
  lvthin-qa-01               vg-qa-01           twi---tz-- <997,00g                             0,00   1,67                           
  vm-103-disk-0              vg-qa-01           Vwi---tz--   50,00g lvthin-qa-01

~# pvecm status
Cluster information
-------------------
Name:             pve-sx-01
Config Version:   7
Transport:        knet
Secure auth:      on

Quorum information
------------------
Date:             Thu Sep 23 14:33:29 2021
Quorum provider:  corosync_votequorum
Nodes:            3
Node ID:          0x00000001
Ring ID:          1.ee
Quorate:          Yes

Votequorum information
----------------------
Expected votes:   3
Highest expected: 3
Total votes:      3
Quorum:           2 
Flags:            Quorate

Membership information
----------------------
    Nodeid      Votes Name
0x00000001          1 10.81.28.101 (local)
0x00000002          1 10.81.28.125
0x00000003          1 10.81.28.124

 

[bbgeek17]

You must not use lvm-thin in shared storage configuration - its not safe and will cause data corruption.
You can use LVM (thick), but there will be no snapshot support.

Either you have to create a LUN per virtual disk and pass it through directly to VM, or you must use Cluster-aware filesystem on top of your LUN.
In the first case you, again, wont have ability to do Proxmox initiated snapshots. You might be able to snapshot on the storage directly.
In the second case, you will be able to use snapshots in combination with using QCOW storage on top of the Cluster-aware filesystem.

At Blockbridge we essentially implement a variant of LUN per VM disk, but with a native Proxmox storage plugin we allow for snapshot/clone/etc operations to be initiated from Proxmox interfaces (UI/CLI/API).

---

Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[christophe.aron]

Thank for the reply but it's very regressive if we can't use snapshot natively.
In Ovirt, we can setup lun in a ovirt domains storage, and the snapshot feature is native for VM.
In fact, make one lun by vm disk is so heavy to manage quickly
It's probably an improvement to do in Proxmox.

 

[bbgeek17]

I agree that a lun per vm disk is very heavy for legacy storage appliances that are built for static workloads.

Its possible to use your storage with Proxmox, there is just not a pretty GUI wizard to do so, as it requires installation and support of 3rd party Cluster File system. An example of product with built-in functionality is ESX VMFS, but then you are talking about different cost factor.

---

Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox