HELP! metadata too large for circular buffer

[discontented]

Hello all,

We have hit a snag on our production system

A user attempted to create a VM this AM but got presented with the error:

TASK ERROR: create failed - lvcreate 'vmsan_vg/pve-vm-380' error: VG vmsan_vg metadata too large for circular buffer

Any ideas on how this can be cleared?
I can provide any outputs of lvdisplay etc. required.

Thanks in advance,
Rob

 

[discontented]

OK so having done a bit of reading this morning, I think maybe, I can change the attributes of the Physical Volume using pvchange to ignore metadata. Any ideas if this would 'fix' the problem?

 

[discontented]

Won't allow me to make the change:

pvchange -a --metadataignore y
Cannot disable all metadata areas in volume group vmsan_vg.
Cannot disable all metadata areas in volume group pve.
0 physical volumes changed / 2 physical volumes not changed

 

[dietmar]

Seems you have more than 250 LVs?

see http://www.redhat.com/archives/linux-lvm/2008-August/msg00014.html

 

[discontented]

Hi Dietmar,

Thanks for your response. I was looking at doing something similar but I need a bit of clarification please?


>Create a new PV with a much larger metadata area, add it to the VG,
>then stop using the too-small metadata areas on the old PVs.

>For extra resiliance it might be wise to add two PVs so you have twocopies of the metadata,
>or use the pvcreate option to have 2 metadatacopies on the single PV you're adding,
>one at the front and ane at the end.

>To remove the metadata areas you need to:

>get an up-to-date metadata backup (vgcfgbackup) pvcreate --restorefile pointing at a copy of that backup file requesting 0 metadata areas
>and specifying the same uuid as it had before vgcfgrestore from the backup file

Does this imply creating a new PV that will be used to replace the old one or does it imply building a PV that is used for metadata only (even possible?)? I think it's the latter but may be wrong. This is a live system with around 5Tb of data so I really don't want to get this wrong. My other option is to shrink my current PV, create a new, migrate some LVs over to the new PV and repeat until the original PV can be removed (ie it has no vms on) then re-partition. Ideas?I'm not a storage engineer as you may be able to tell.

Thanks in advance, Rob