device-mapper: message ioctl on failed: Operation not supported

Chiranjeevi

New Member
May 6, 2017
23
0
1
38
we have proxmox 4.4, whihc is having 2 LVM-thin Volumes, for some reason we have done hard-restart.

After that the 2nd Volume is not working,

if we try to create VM in that LVM-Thin it says

device-mapper: message ioctl on failed: Operation not supported
TASK ERROR: create failed - lvcreate 'vmdata/vm-115-disk-1' error: Failed to resume vmstore.


And we are not able to start the existing VM also, error is

kvm: -drive file=/dev/vmdata/vm-100-disk-1,if=none,id=drive-virtio0,format=raw,cache=none,aio=native,detect-zeroes=on: Could not open '/dev/vmdata/vm-100-disk-1': No such file or directory

if we see the content of the LVM-Thin storage from web ui it show the VM files(RAW files).

But i run the ls in /dev/vmdata the files are missing ( but other volume /dev/pve has the symlinks

how to recover this LVM? Please help me, and i am new to these things
 

dcsapak

Proxmox Staff Member
Staff member
Feb 1, 2016
8,640
1,126
174
34
Vienna
what does lvs, vgs and lsblk display?
 

Chiranjeevi

New Member
May 6, 2017
23
0
1
38
lvs :
Code:
LV            VG     Attr       LSize   Pool   Origin Data%  Meta%  Move Log Cpy%Sync Convert
  data          pve    twi-aotz--   8.98t               76.83  31.57           
  root          pve    -wi-ao----  96.00g                                       
  swap          pve    -wi-ao----   8.00g                                       
  vm-101-disk-1 pve    Vwi-aotz-- 260.00g data          100.00                 
  vm-103-disk-1 pve    Vwi-aotz-- 210.00g data          95.24                   
  vm-104-disk-1 pve    Vwi-aotz--   1.00t data          89.04                   
  vm-105-disk-1 pve    Vwi-aotz-- 610.00g data          39.46                   
  vm-107-disk-1 pve    Vwi-aotz--   1.02t data          49.20                   
  vm-108-disk-1 pve    Vwi-aotz-- 260.00g data          96.15                   
  vm-109-disk-1 pve    Vwi-aotz-- 260.00g data          96.15                   
  vm-110-disk-1 pve    Vwi-aotz--   1.29t data          99.24                   
  vm-111-disk-1 pve    Vwi-aotz-- 574.00g data          98.26                   
  vm-112-disk-1 pve    Vwi-aotz-- 460.00g data          58.37                   
  vm-113-disk-1 pve    Vwi-a-tz-- 550.00g data          70.22                   
  vm-114-disk-1 pve    Vwi-aotz-- 562.00g data          80.07                   
  lvol0         saarvg -wi-a----- 120.00m                                       
  lvol1         saarvg -wi-a----- 120.00m                                       
  saarlv        saarvg twi-aotzM-   3.63t               37.45  99.97           
  vm-100-disk-1 saarvg Vwi-a-tz-- 210.00g saarlv        3.50                   
  vm-102-disk-1 saarvg Vwi-aotz-- 260.00g saarlv        96.15                   
  vm-106-disk-1 saarvg Vwi-aotz--   1.29t saarlv        49.51

vgs
Code:
VG     #PV #LV #SN Attr   VSize VFree
  pve      1  15   0 wz--n- 9.09t 15.86g
  saarvg   1   6   0 wz--n- 3.64t  7.91g

lsblk
Code:
NAME                            MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
sda                               8:0    0  4.6T  0 disk
└─sda1                            8:1    0  4.6T  0 part
sdb                               8:16   0  9.1T  0 disk
├─sdb1                            8:17   0    1M  0 part
├─sdb2                            8:18   0  256M  0 part
└─sdb3                            8:19   0  9.1T  0 part
  ├─pve-root                    251:0    0   96G  0 lvm  /
  ├─pve-swap                    251:10   0    8G  0 lvm  [SWAP]
  ├─pve-data_tmeta              251:11   0   72M  0 lvm
  │ └─pve-data-tpool            251:13   0    9T  0 lvm
  │   ├─pve-data                251:14   0    9T  0 lvm
  │   ├─pve-vm--101--disk--1    251:15   0  260G  0 lvm
  │   ├─pve-vm--104--disk--1    251:16   0    1T  0 lvm
  │   ├─pve-vm--105--disk--1    251:17   0  610G  0 lvm
  │   ├─pve-vm--107--disk--1    251:18   0    1T  0 lvm
  │   ├─pve-vm--108--disk--1    251:19   0  260G  0 lvm
  │   ├─pve-vm--109--disk--1    251:20   0  260G  0 lvm
  │   ├─pve-vm--110--disk--1    251:21   0  1.3T  0 lvm
  │   ├─pve-vm--111--disk--1    251:22   0  574G  0 lvm
  │   ├─pve-vm--112--disk--1    251:23   0  460G  0 lvm
  │   ├─pve-vm--114--disk--1    251:24   0  562G  0 lvm
  │   ├─pve-vm--103--disk--1    251:25   0  210G  0 lvm
  │   └─pve-vm--113--disk--1    251:26   0  550G  0 lvm
  └─pve-data_tdata              251:12   0    9T  0 lvm
    └─pve-data-tpool            251:13   0    9T  0 lvm
      ├─pve-data                251:14   0    9T  0 lvm
      ├─pve-vm--101--disk--1    251:15   0  260G  0 lvm
      ├─pve-vm--104--disk--1    251:16   0    1T  0 lvm
      ├─pve-vm--105--disk--1    251:17   0  610G  0 lvm
      ├─pve-vm--107--disk--1    251:18   0    1T  0 lvm
      ├─pve-vm--108--disk--1    251:19   0  260G  0 lvm
      ├─pve-vm--109--disk--1    251:20   0  260G  0 lvm
      ├─pve-vm--110--disk--1    251:21   0  1.3T  0 lvm
      ├─pve-vm--111--disk--1    251:22   0  574G  0 lvm
      ├─pve-vm--112--disk--1    251:23   0  460G  0 lvm
      ├─pve-vm--114--disk--1    251:24   0  562G  0 lvm
      ├─pve-vm--103--disk--1    251:25   0  210G  0 lvm
      └─pve-vm--113--disk--1    251:26   0  550G  0 lvm
sdc                               8:32   0  3.7T  0 disk
└─sdc1                            8:33   0  3.7T  0 part
  ├─saarvg-lvol0                251:1    0  120M  0 lvm
  ├─saarvg-lvol1                251:2    0  120M  0 lvm
  ├─saarvg-saarlv_tmeta         251:3    0  120M  0 lvm
  │ └─saarvg-saarlv-tpool       251:5    0  3.6T  0 lvm
  │   ├─saarvg-saarlv           251:6    0  3.6T  0 lvm
  │   ├─saarvg-vm--100--disk--1 251:7    0  210G  0 lvm
  │   ├─saarvg-vm--102--disk--1 251:8    0  260G  0 lvm
  │   └─saarvg-vm--106--disk--1 251:9    0  1.3T  0 lvm
  └─saarvg-saarlv_tdata         251:4    0  3.6T  0 lvm
    └─saarvg-saarlv-tpool       251:5    0  3.6T  0 lvm
      ├─saarvg-saarlv           251:6    0  3.6T  0 lvm
      ├─saarvg-vm--100--disk--1 251:7    0  210G  0 lvm
      ├─saarvg-vm--102--disk--1 251:8    0  260G  0 lvm
      └─saarvg-vm--106--disk--1 251:9    0  1.3T  0 lvm
 

dcsapak

Proxmox Staff Member
Staff member
Feb 1, 2016
8,640
1,126
174
34
Vienna

Chiranjeevi

New Member
May 6, 2017
23
0
1
38
Thanks for reply

This is my storage.cfg
Code:
dir: local
    path /var/lib/vz
    content vztmpl,backup,iso

lvmthin: local-lvm
    vgname pve
    thinpool data
    content rootdir,images

lvmthin: maganet-hdd
    vgname saarvg
    thinpool saarlv
    content images,rootdir
I have inserted my HDD to Server and Configured the RAID1
then i did fdisk -l to know which is my new HDD
then i followed this to create LVM
https://pve.proxmox.com/wiki/LVM2#Administration

i thier anything i made wrong?
 

dcsapak

Proxmox Staff Member
Staff member
Feb 1, 2016
8,640
1,126
174
34
Vienna
i thier anything i made wrong?
if you never extended the thinpool, then not, but the metadatapool is too small nonetheless, you should extend it, instructions are in the wiki article

also can you post the output of
Code:
lvs -o name,metadata_percent,data_percent,chunk_size,size --all
cat $(readlink -f /sys/dev/block/*/sdb3/../queue/minimum_io_size)
cat $(readlink -f /sys/dev/block/*/sdc1/../queue/minimum_io_size)
 

Chiranjeevi

New Member
May 6, 2017
23
0
1
38
i have recreated the LV again with 1G metadata size
because i dont have any important VMs in it

lvs -o name,metadata_percent,data_percent,chunk_size,size --all

Code:
LV              Meta%  Data%  Chunk   LSize
  data            32.25  81.70    8.00m   8.98t
  [data_tdata]                       0    8.98t
  [data_tmeta]                       0   72.00m
  [lvol0_pmspare]                    0   72.00m
  root                               0   96.00g
  swap                               0    8.00g
  vm-100-disk-1          90.91       0  110.00g
  vm-101-disk-1          100.00      0  260.00g
  vm-102-disk-1          2.31        0  260.00g
  vm-102-disk-2          0.00        0  260.00g
  vm-103-disk-1          95.24       0  210.00g
  vm-104-disk-1          89.05       0    1.00t
  vm-105-disk-1          39.46       0  610.00g
  vm-107-disk-1          49.20       0    1.02t
  vm-108-disk-1          96.15       0  260.00g
  vm-109-disk-1          96.15       0  260.00g
  vm-110-disk-1          99.24       0    1.29t
  vm-111-disk-1          98.26       0  574.00g
  vm-112-disk-1          58.37       0  460.00g
  vm-113-disk-1          73.44       0  550.00g
  vm-114-disk-1          80.07       0  562.00g
  [lvol0_pmspare]                    0  116.00m
  saarlv          5.09   0.50   256.00k   3.60t
  [saarlv_tdata]                     0    3.60t
  [saarlv_tmeta]                     0  116.00m
  vm-106-disk-1          2.03        0  900.00g

do in increate for the metedata size for data_tmeta also?

i have another problem let me create another thread, if we do copy or clone VM the iodelay will increase because of this all other running VMs will be slow and web GUI will be un responsive
 

dcsapak

Proxmox Staff Member
Staff member
Feb 1, 2016
8,640
1,126
174
34
Vienna
do in increate for the metedata size for data_tmeta also?
no it seems it correctly increased the 'chunk_size' instead of the metadata pool size

i am guessing there is a problem where some raidcards report some values which limits the chunk size of lvm

could you also please post the other commands i wrote?:
cat $(readlink -f /sys/dev/block/*/sdb3/../queue/minimum_io_size)
cat $(readlink -f /sys/dev/block/*/sdc1/../queue/minimum_io_size)
 

Chiranjeevi

New Member
May 6, 2017
23
0
1
38
Code:
cat $(readlink -f /sys/dev/block/*/sda3/../queue/minimum_io_size)
262144
cat $(readlink -f /sys/dev/block/*/sdb1/../queue/minimum_io_size)
262144

These are after i recrated LVM
 

dcsapak

Proxmox Staff Member
Staff member
Feb 1, 2016
8,640
1,126
174
34
Vienna
how exactly did you create your thinpool?

if you can/want recreate the thinpool again, could you try with:
Code:
lvcreate -L<SIZE> -n <NAME> <VG>
lvconvert --type thin-pool <VG>/<NAME>

and again give the output of the 'lvs -o ..' command?
 

Chiranjeevi

New Member
May 6, 2017
23
0
1
38
i have recreated VD in RAID controller

Code:
sgdisk -N 1 /dev/sdb
pvcreate --metadatasize 1G -y -ff /dev/sdb1
vgcreate saarvg /dev/sdb1
lvcreate -L 3.6T -T -n saarlv saarvg

These are command exactly i ran

then From GUI i added storage to the DataCenter as LVM-Thin
 

dcsapak

Proxmox Staff Member
Staff member
Feb 1, 2016
8,640
1,126
174
34
Vienna
mhmm this is weird

you should extend the metadata pool (because you did not create the metadatapool with 1 GB, just reserved the space)
with
Code:
lvresize --poolmetadatasize +<size[M,G]> <VG>/<LVThin_pool>

can you maybe also post the output of:
cat $(readlink -f /sys/dev/block/*/sda3/../queue/optimal_io_size)
cat $(readlink -f /sys/dev/block/*/sdb1/../queue/optimal_io_size)
 

Chiranjeevi

New Member
May 6, 2017
23
0
1
38
@dcsapak When this problem started and you said we have to increase the metadatapool, i just deleted the LVM and recreated

Not resized the metadatapool

After recreateing the LVM again

Code:
cat $(readlink -f /sys/dev/block/*/sda3/../queue/minimum_io_size)
262144
cat $(readlink -f /sys/dev/block/*/sdb1/../queue/minimum_io_size)
262144
 

dcsapak

Proxmox Staff Member
Staff member
Feb 1, 2016
8,640
1,126
174
34
Vienna
please not the commands now are different:
Code:
cat $(readlink -f /sys/dev/block/*/sda3/../queue/optimal_io_size)
cat $(readlink -f /sys/dev/block/*/sdb1/../queue/optimal_io_size)
(optimal_io_size instead of minimum_io_size)
 

Chiranjeevi

New Member
May 6, 2017
23
0
1
38
i am sorry.. my bad

Code:
# cat $(readlink -f /sys/dev/block/*/sda3/../queue/optimal_io_size)
262144
 cat $(readlink -f /sys/dev/block/*/sdb1/../queue/optimal_io_size)
262144
 

Chiranjeevi

New Member
May 6, 2017
23
0
1
38
Again i am facing a strange issue,

the LVM-Thin which is have created as
pvcreate --metadatasize 1G -y -ff /dev/sdb1
Now if i do lvs -a

Code:
# lvs -a
  LV              VG     Attr       LSize   Pool   Origin Data%  Meta%  Move Log                                                                                                                                                              Cpy%Sync Convert
  data            pve    twi-aotz--   8.98t               88.36  33.87                                                                                                                                                                       
  [data_tdata]    pve    Twi-ao----   8.98t                                                                                                                                                                                                 
  [data_tmeta]    pve    ewi-ao----  72.00m                                                                                                                                                                                                 
  [lvol0_pmspare] pve    ewi-------  72.00m                                                                                                                                                                                                 
  root            pve    -wi-ao----  96.00g                                                                                                                                                                                                 
  swap            pve    -wi-ao----   8.00g                                                                                                                                                                                                 
  vm-100-disk-1   pve    Vwi-aotz-- 110.00g data          90.91                                                                                                                                                                             
  vm-101-disk-1   pve    Vwi-aotz-- 260.00g data          100.00                                                                                                                                                                             
  vm-102-disk-1   pve    Vwi-aotz-- 260.00g data          96.15                                                                                                                                                                             
  vm-103-disk-1   pve    Vwi-aotz-- 210.00g data          95.24                                                                                                                                                                             
  vm-104-disk-1   pve    Vwi-aotz--   1.00t data          89.05                                                                                                                                                                             
  vm-105-disk-1   pve    Vwi-aotz-- 610.00g data          39.46                                                                                                                                                                             
  vm-107-disk-1   pve    Vwi-aotz--   1.02t data          49.21                                                                                                                                                                             
  vm-108-disk-1   pve    Vwi-aotz-- 260.00g data          96.15                                                                                                                                                                             
  vm-109-disk-1   pve    Vwi-aotz-- 260.00g data          96.15                                                                                                                                                                             
  vm-110-disk-1   pve    Vwi-aotz--   1.29t data          99.24                                                                                                                                                                             
  vm-111-disk-1   pve    Vwi-aotz-- 574.00g data          98.26                                                                                                                                                                             
  vm-112-disk-1   pve    Vwi-aotz-- 460.00g data          58.46                                                                                                                                                                             
  vm-113-disk-1   pve    Vwi-aotz-- 550.00g data          82.68                                                                                                                                                                             
  vm-114-disk-1   pve    Vwi-aotz-- 562.00g data          80.07                                                                                                                                                                             
  base-115-disk-1 saarvg Vri---tz-k 200.00g saarlv                                                                                                                                                                                           
  [lvol0_pmspare] saarvg ewi------- 116.00m                                                                                                                                                                                                 
  saarlv          saarvg twi-aotzM-   3.60t               23.94  99.97                                                                                                                                                                       
  [saarlv_tdata]  saarvg Twi-ao----   3.60t                                                                                                                                                                                                 
  [saarlv_tmeta]  saarvg ewi-ao---- 368.00m                                                                                                                                                                                                 
  vm-106-disk-1   saarvg Vwi-aotz-- 900.00g saarlv        2.65                                                                                                                                                                               
  vm-116-disk-1   saarvg Vwi-aotz-- 200.00g saarlv        3.08                                                                                                                                                                               
  vm-117-disk-1   saarvg Vwi-aotz-- 500.00g saarlv        1.25                                                                                                                                                                               
  vm-118-disk-1   saarvg Vwi-aotz-- 200.00g saarlv        3.06                                                                                                                                                                               
  vm-119-disk-1   saarvg Vwi-aotz--   1.29t saarlv        62.81
99.97% usage and i tried to increase by running
Code:
 lvextend -L+250M saarvg/saarlv_tmeta
metadata is still 99.97% as we seen above
 

dcsapak

Proxmox Staff Member
Staff member
Feb 1, 2016
8,640
1,126
174
34
Vienna
please use the command from the wiki:
Code:
lvresize --poolmetadatasize +<size[M,G]> <VG>/<LVThin_pool>
 

Chiranjeevi

New Member
May 6, 2017
23
0
1
38
Thanks for reply
Code:
# lvresize --poolmetadatasize +10M saarvg/saarlv
  Rounding pool metadata size to boundary between physical extents: 12.00 MiB
  Extending logical volume saarlv_tmeta to 380.00 MiB.
  Logical volume saarlv successfully resized

Still i can see 99.97%

Code:
# lvs -a
  LV              VG     Attr       LSize   Pool   Origin Data%  Meta%  Move Log Cpy%Sync Convert
  data            pve    twi-aotz--   8.98t               88.36  33.87
  [data_tdata]    pve    Twi-ao----   8.98t
  [data_tmeta]    pve    ewi-ao----  72.00m
  [lvol0_pmspare] pve    ewi-------  72.00m
  root            pve    -wi-ao----  96.00g
  swap            pve    -wi-ao----   8.00g
  vm-100-disk-1   pve    Vwi-aotz-- 110.00g data          90.91
  vm-101-disk-1   pve    Vwi-aotz-- 260.00g data          100.00
  vm-102-disk-1   pve    Vwi-aotz-- 260.00g data          96.15
  vm-103-disk-1   pve    Vwi-aotz-- 210.00g data          95.24
  vm-104-disk-1   pve    Vwi-aotz--   1.00t data          89.05
  vm-105-disk-1   pve    Vwi-aotz-- 610.00g data          39.46
  vm-107-disk-1   pve    Vwi-aotz--   1.02t data          49.21
  vm-108-disk-1   pve    Vwi-aotz-- 260.00g data          96.15
  vm-109-disk-1   pve    Vwi-aotz-- 260.00g data          96.15
  vm-110-disk-1   pve    Vwi-aotz--   1.29t data          99.24
  vm-111-disk-1   pve    Vwi-aotz-- 574.00g data          98.26
  vm-112-disk-1   pve    Vwi-aotz-- 460.00g data          58.46
  vm-113-disk-1   pve    Vwi-aotz-- 550.00g data          82.68
  vm-114-disk-1   pve    Vwi-aotz-- 562.00g data          80.07
  base-115-disk-1 saarvg Vri---tz-k 200.00g saarlv
  [lvol0_pmspare] saarvg ewi------- 116.00m
  saarlv          saarvg twi-aotzM-   3.60t               23.94  99.97
  [saarlv_tdata]  saarvg Twi-ao----   3.60t
  [saarlv_tmeta]  saarvg ewi-ao---- 380.00m
  vm-106-disk-1   saarvg Vwi-aotz-- 900.00g saarlv        2.65
  vm-116-disk-1   saarvg Vwi-aotz-- 200.00g saarlv        3.08
  vm-117-disk-1   saarvg Vwi-aotz-- 500.00g saarlv        1.25
  vm-118-disk-1   saarvg Vwi-aotz-- 200.00g saarlv        3.06
  vm-119-disk-1   saarvg Vwi-aotz--   1.29t saarlv        62.81

do i need to restart the server?
 

Chiranjeevi

New Member
May 6, 2017
23
0
1
38
generally no, i guess the increase of 12MB was too little to make a difference, try increasing it more (eg. to 500MB)
before this i have already increased 250M so just added 12MB. Now its 380MB

Now it fine, after restart
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get your own in 60 seconds.

Buy now!