activating LV 'pve/data' failed: Check of pool pve/data failed (status:1). Manual repair required!

[muner]

Hello
everyone.
im facing during outage electricity, I have tried many things but it doesn't work.

activating LV 'pve/data' failed: Check of pool pve/data failed (status:1). Manual repair required!

I hope someone can help me with my problem.

Thank,

 

[alex#]

Please post: vgdisplay

Your pve-root is 95% filled with 0 FSAVAILABLE?

lvconvert --repair calls thin_repair. man thin_repair states that it's important to have enough space left for metadata - maybe that's the issue?

 

[muner]

Hi Mr Alex.



Vgdisplay here for ur reference

lvconvert --repair calls thin_repair. man thin_repair states that it's important to have enough space left for metadata - maybe that's the issue?

what should i do now.

thank support

 

[muner]

so need to increase the size hdd

 

[muner]

 

[muner]

lvremove /dev/pve/data -y

lvcreate -L 10G -n data pve -T

 

[muner]

lvconvert --repair pve/data

 

[alex#]

lvremove /dev/pve/data -y

Why trying this? If successful you would lose all your VM and snapshot data.

Start with freeing space on your pve-root.

 

[muner]

Why trying this? If successful you would lose all your VM and snapshot data.

Start with freeing space on your pve-root.

Not successful

 

[muner]

Why trying this? If successful you would lose all your VM and snapshot data.

Start with freeing space on your pve-root.

pls guide me sir

 

[alex#]

I'm sorry but maybe you can mount/repair your volumes when booting from a live USB stick (e.g. Debian or easier Ubuntu). When successful you can backup your needed data and reinstall Proxmox VE. It's a bit of work and learning but that's probably your best choice to get a well working default setup back again.

 

[muner]

I'm sorry but maybe you can mount/repair your volumes when booting from a live USB stick (e.g. Debian or easier Ubuntu). When successful you can backup your needed data and reinstall Proxmox VE. It's a bit of work and learning but that's probably your best choice to get a well working default setup back again.

ok i try. even a bit stuck

 

[muner]

what do you think build another proxmox than copy/mount Data into new proxmox.
but pve/data old proxmox inactive how to transfer.

 

[bagstoper]

Hey Muner,

Wanted to see if you made it any further since you last posted? I have a similar issue however my storage is only 6% full. I am about to make a separate post but was curios for any update you may have.

Were you able to use another OS to pull the data and move it to another Proxmox server? I have data I dont want to lose, but also hadnt made a backup yet...

 

[theblinkingusb]

Hey all - wanted to share my experience today if helpful, YMMV - I am not a pro by any means. Ultimately I was able to run Alex's lvconvert command (lvconvert --repair <thinpool>) and get back up and running.

Background -
My system powers down and restarts daily with no issues. The system has two physical volumes, one ssd and one nvme, Today I powered on the system and went to boot the vms and also received the Check of pool pve/ssd01 failed (status:1) message. The thin pool on the ssd reported "not available" in lvdisplay as well. vgdisplay on the ssd drive showed free of 16gb, alloc of 448GB, for a total of 464 on the vg.

On the drive is a thin pool (ssd01) with 338g total (96GB for root, 8G on swap (unsure where the missing 6g is)), with 5 thin provisioned vm disks in that pool (their sum is 390gb). Proxmox metrics in the UI showing about 45% utilization on the pool. Today however the pool showed as "not available" in lvdisplay and "inactive" in lvscan, and since the pool was down all the dependent lvs were reporting that as well. dmsetup info -c was not showing the lvs either. Tried a lvchange -a y pve/ssd01 first and received the same message. Then performed the lvconvert --repair pve/ssd01.

On running I received the below messages

WARNING: Sum of all thin volume sizes (390.01 GiB) exceeds the size of thin pools and the amount of free space in volume group (12.55 GiB).
WARNING: You have not turned on protection against thin pools running out of space.
WARNING: Set activation/thin_pool_autoextend_threshold below 100 to trigger automatic extension of thin pools before they get full.
WARNING: LV pve/ssd01_meta0 holds a backup of the unrepaired metadata. Use lvremove when no longer required.

After a couple successful reboots I removed the pve/ssd01_meta0 lvremove pve/ssd01_meta0
I also removed an extra clone to reduce over provisioning risk, and left the other messages slide per this thread

Not sure how the system got into this state. I ran a pvck /dev/sda3 for the drive in question and no metadata error messages. I read through syslog for a while looking for anything but did not find any message indicating a root cause for the pvestatd[1264]: activating LV 'pve/ssd01' failed. SMART values on the drive show no wearout and no failing attributes (PASSED).

I'll report back if I see anything further here.

Running proxmox version 8.0.3

Also just wanted to drop a general thanks to the Proxmox team - appreciate the awesome product as always.

 

[bert_to]

Hey all - wanted to share my experience today if helpful, YMMV - I am not a pro by any means. Ultimately I was able to run Alex's lvconvert command (lvconvert --repair <thinpool>) and get back up and running.

Background -
My system powers down and restarts daily with no issues. The system has two physical volumes, one ssd and one nvme, Today I powered on the system and went to boot the vms and also received the Check of pool pve/ssd01 failed (status:1) message. The thin pool on the ssd reported "not available" in lvdisplay as well. vgdisplay on the ssd drive showed free of 16gb, alloc of 448GB, for a total of 464 on the vg.

On the drive is a thin pool (ssd01) with 338g total (96GB for root, 8G on swap (unsure where the missing 6g is)), with 5 thin provisioned vm disks in that pool (their sum is 390gb). Proxmox metrics in the UI showing about 45% utilization on the pool. Today however the pool showed as "not available" in lvdisplay and "inactive" in lvscan, and since the pool was down all the dependent lvs were reporting that as well. dmsetup info -c was not showing the lvs either. Tried a lvchange -a y pve/ssd01 first and received the same message. Then performed the lvconvert --repair pve/ssd01.

On running I received the below messages


After a couple successful reboots I removed the pve/ssd01_meta0 lvremove pve/ssd01_meta0
I also removed an extra clone to reduce over provisioning risk, and left the other messages slide per this thread

Not sure how the system got into this state. I ran a pvck /dev/sda3 for the drive in question and no metadata error messages. I read through syslog for a while looking for anything but did not find any message indicating a root cause for the pvestatd[1264]: activating LV 'pve/ssd01' failed. SMART values on the drive show no wearout and no failing attributes (PASSED).

I'll report back if I see anything further here.

Running proxmox version 8.0.3

Also just wanted to drop a general thanks to the Proxmox team - appreciate the awesome product as always.

I ran into the same problem and was also solved with the lvconvert repair option.