LVM Thin Pool metadata pool full

D

decibel83

Renowned Member
Oct 15, 2008
210
1
83
Hi.
I have some nodes with a local LVM Thin pool.
On one node the metadata pool is full and I cannot start any new containers:

Code: 
root@node3:~# lvs -a -o+metadata_percent
  LV                VG     Attr       LSize   Pool      Origin Data%  Meta%  Move Log Cpy%Sync Convert Meta%
  data              system -wi-ao----  10.00g                                                               
  data-thin         system twi-aotzM- 800.00g                  91.35  99.80                            99.80
  [data-thin_tdata] system Twi-ao---- 800.00g                                                               
  [data-thin_tmeta] system ewi-ao---- 100.00m                                                               
  [lvol0_pmspare]   system ewi------- 100.00m                                                               
  root              system -wi-ao----  18.62g                                                               
  swap              system -wi-ao----  14.90g                                                               
  tmp               system -wi-ao----   4.66g                                                               
  vm-301-disk-1     system Vwi-aotz--  32.00g data-thin        20.17                                        
  vm-302-disk-1     system Vwi-aotz-- 100.00g data-thin        46.67                                        
  vm-304-disk-1     system Vwi-a-tz--  20.00g data-thin        6.45                                         
  vm-305-disk-1     system Vwi-aotz--  60.00g data-thin        93.62                                        
  vm-305-disk-2     system Vwi-aotz-- 100.00g data-thin        98.40                                        
  vm-306-disk-1     system Vwi-a-tz--  40.00g data-thin        21.81                                        
  vm-307-disk-1     system Vwi-aotz-- 100.00g data-thin        95.30                                        
  vm-308-disk-1     system Vwi---tz-- 500.00g data-thin

I tried to resize the metadata pool size according to https://pve.proxmox.com/wiki/LVM2#Resize_metadata_pool, but I received this error:

Code: 
root@node3:~# lvresize --poolmetadatasize +100M system/data-thin
  Extending logical volume data-thin_tmeta to 200.00 MiB.
  device-mapper: message ioctl on  failed: Operation not supported
  Problem reactivating logical volume system/vm-304-disk-1.
  Releasing activation in critical section.
  libdevmapper exiting with 2 device(s) still suspended.

After this, the lvs command hangs and is not returning me any result.

Could you help me please?

Thanks!
 
Last edited:
decibel83 said:
Hi.
I have some nodes with a local LVM Thin pool.
On one node the metadata pool is full and I cannot start any new containers:

Code: 
root@node3:~# lvs -a -o+metadata_percent
  LV                VG     Attr       LSize   Pool      Origin Data%  Meta%  Move Log Cpy%Sync Convert Meta%
  data              system -wi-ao----  10.00g                                                              
  data-thin         system twi-aotzM- 800.00g                  91.35  99.80                            99.80
  [data-thin_tdata] system Twi-ao---- 800.00g                                                              
  [data-thin_tmeta] system ewi-ao---- 100.00m                                                              
  [lvol0_pmspare]   system ewi------- 100.00m                                                              
  root              system -wi-ao----  18.62g                                                              
  swap              system -wi-ao----  14.90g                                                              
  tmp               system -wi-ao----   4.66g                                                              
  vm-301-disk-1     system Vwi-aotz--  32.00g data-thin        20.17                                       
  vm-302-disk-1     system Vwi-aotz-- 100.00g data-thin        46.67                                       
  vm-304-disk-1     system Vwi-a-tz--  20.00g data-thin        6.45                                        
  vm-305-disk-1     system Vwi-aotz--  60.00g data-thin        93.62                                       
  vm-305-disk-2     system Vwi-aotz-- 100.00g data-thin        98.40                                       
  vm-306-disk-1     system Vwi-a-tz--  40.00g data-thin        21.81                                       
  vm-307-disk-1     system Vwi-aotz-- 100.00g data-thin        95.30                                       
  vm-308-disk-1     system Vwi---tz-- 500.00g data-thin

I tried to resize the metadata pool size according to https://pve.proxmox.com/wiki/LVM2#Resize_metadata_pool, but I received this error:

Code: 
root@node3:~# lvresize --poolmetadatasize +100M system/data-thin
  Extending logical volume data-thin_tmeta to 200.00 MiB.
  device-mapper: message ioctl on  failed: Operation not supported
  Problem reactivating logical volume system/vm-304-disk-1.
  Releasing activation in critical section.
  libdevmapper exiting with 2 device(s) still suspended.
Click to expand...

You have an overprovisioned pool, which very dangerous and can damage the pool without any notice - maybe it happened. Create a new one, move (still healthy) lvm(s) there and delete the old one.

For thin volume transfer see here some hints:
https://unix.stackexchange.com/questions/214065/can-thin-lvs-be-moved-to-a-different-lvm-pool
 
You must log in or register to reply here.
Top Bottom