LVM thin disaster today: ran out of metadata space

Pavel Hruška

Member
May 1, 2018
75
8
8
45
Today there was ready a really bad surprise for me, as on PVE host with LVM-thin I started to receive notifications about disk I/O write problems and then all of VMs (Windows) slowly hung up...

Just after the server reboot and when first VM started to boot up I've received such error:
Code:
space map metadata: unable to allocate new metadata block


This was then followed by errors like this:
Code:
Buffer I/O error ... lost async page write


After quick googling I've noticed that there is problem with LVM-thin metadata and that there is no more space for them, so that's why those I/O write failures. All that turned out to be a complete VM recovery from backups. Damn. This time I've destroyed LVM-thin and went with classic LVM volumes!

I've came accross this post:
https://forum.proxmox.com/threads/lvm-ran-out-of-metadata-space-need-help.41325/

Where it seems that the issue originates in Proxmox default configuration where only small portion of space is defined for metadata.

Anybody has more info about this?

UPDATE: Just to give more info, the server was installed on 5.x this june (2019) and it is pretty low load server. Upgraded to 6.x last week.
 
Last edited:
In that referenced post there is statement:
I'm afraid you have experienced loss of data due to bug in PM, for which they do not seem to care.

But also I've seen that bug has been "fixed":
https://bugzilla.proxmox.com/show_bug.cgi?id=1241#c9

But... I've installed fresh PVE host, created LVM thin and after 3 months of running that host collapsed due the same problem.

Just asking if anyone has the same problem, suggestions. Just think that issue that causes data corruption is important to share...
 
I remember having a similar problem before when I was using LVM Thin. The way I fixed it was a last resort. I added a 64GB USB key to the pool just so I can backup the VM (since the hypervisor was not usable before that and I had no disk slot available). Once done the only thing you can do is re-install and restore the VMs from backup. Do not run the machine using the USB key in the pool ! This is a temporary "mcgiver solution" just to buy you some time to backup your VMs.
 
  • Like
Reactions: PlOrAdmin
I remember having a similar problem before when I was using LVM Thin. The way I fixed it was a last resort. I added a 64GB USB key to the pool just so I can backup the VM (since the hypervisor was not usable before that and I had no disk slot available). Once done the only thing you can do is re-install and restore the VMs from backup. Do not run the machine using the USB key in the pool ! This is a temporary "mcgiver solution" just to buy you some time to backup your VMs.

So no longer using LVM-Thin?
 
I am using LVM (not thin) on few servers and I'm fine, too. And using ZFS when possible.

You know, the best learning is from mistakes, so good leson today...

Thank you for replies, btw.
 
  • Like
Reactions: ca_maer

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!