LVM thin disaster today: ran out of metadata space

[Pavel Hruška]

Today there was ready a really bad surprise for me, as on PVE host with LVM-thin I started to receive notifications about disk I/O write problems and then all of VMs (Windows) slowly hung up...

Just after the server reboot and when first VM started to boot up I've received such error:

space map metadata: unable to allocate new metadata block

This was then followed by errors like this:

Buffer I/O error ... lost async page write

After quick googling I've noticed that there is problem with LVM-thin metadata and that there is no more space for them, so that's why those I/O write failures. All that turned out to be a complete VM recovery from backups. Damn. This time I've destroyed LVM-thin and went with classic LVM volumes!

I've came accross this post:
https://forum.proxmox.com/threads/lvm-ran-out-of-metadata-space-need-help.41325/

Where it seems that the issue originates in Proxmox default configuration where only small portion of space is defined for metadata.

Anybody has more info about this?

UPDATE: Just to give more info, the server was installed on 5.x this june (2019) and it is pretty low load server. Upgraded to 6.x last week.

 

[mailinglists]

What more info do you want?
https://forum.proxmox.com/threads/lvm-ran-out-of-metadata-space-need-help.41325/post-199225

 

[Pavel Hruška]

In that referenced post there is statement:

I'm afraid you have experienced loss of data due to bug in PM, for which they do not seem to care.

But also I've seen that bug has been "fixed":
https://bugzilla.proxmox.com/show_bug.cgi?id=1241#c9

But... I've installed fresh PVE host, created LVM thin and after 3 months of running that host collapsed due the same problem.

Just asking if anyone has the same problem, suggestions. Just think that issue that causes data corruption is important to share...

 

[ca_maer]

I remember having a similar problem before when I was using LVM Thin. The way I fixed it was a last resort. I added a 64GB USB key to the pool just so I can backup the VM (since the hypervisor was not usable before that and I had no disk slot available). Once done the only thing you can do is re-install and restore the VMs from backup. Do not run the machine using the USB key in the pool ! This is a temporary "mcgiver solution" just to buy you some time to backup your VMs.

 

[Pavel Hruška]

I remember having a similar problem before when I was using LVM Thin. The way I fixed it was a last resort. I added a 64GB USB key to the pool just so I can backup the VM (since the hypervisor was not usable before that and I had no disk slot available). Once done the only thing you can do is re-install and restore the VMs from backup. Do not run the machine using the USB key in the pool ! This is a temporary "mcgiver solution" just to buy you some time to backup your VMs.

So no longer using LVM-Thin?

 

[ca_maer]

So no longer using LVM-Thin?

No I'm using ZFS now. Don't get me wrong LVM is great and using thin is perfectly fine but make sure you plan accordingly. It's just that I prefer ZFS

 

[Pavel Hruška]

I am using LVM (not thin) on few servers and I'm fine, too. And using ZFS when possible.

You know, the best learning is from mistakes, so good leson today...

Thank you for replies, btw.

 

[mailinglists]

I sugggest to add monitoring of available metadata on your LVM installs.