TASK ERROR: activating LV 'pve/data' failed: Activation of logical volume pve/data is prohibited while logical volume pve/data_tdata is active.
- Thread starter EWek11
- Start date
Hi,
the relevant volume group is not named
pve, but images, so you need to adapt the lvchange commands.Thank you! I did figure that out eventually, still didn't seem to solve the problem and it wouldn't be permanent anyway. I did go through the suggested thread for a permanent config change, and that didn't work either. After multiple reboots, I managed to get the servers back up, but I feel like it was a timing / luck thing. Still looking for the "right" solution before I have to reboot them again
Did you run the update-initramfs command too? Otherwise you can also try to increase the udev timeout as suggested in the other thread.
By "udev timeout" do you mean where they mention: "Add --skip-mappings in the udev rules for LVM"
Which is mentioned in this thread ... yes? That's where I see the command
update-initramfs -u ... is that what you meant by Did you run the update-initramfs command too?EDIT:
To be clear, that's what I meant in my other thread by
adding / converting / rebooting / removing / rebootingAdded:
thin_check_options = [ "-q", "--skip-mappings" ]Ran:
update-initramfs -u as recomendedConverted with
lvconvert --repair images/imagesRebooted (still failed to mount "images")
Removed the
thin_check options aboveRe-ran
update-initramfs -u as recomendedRebooted
After a few tries, I managed to get the system back up and running stable... but it doesn't feel like I solved it, it feels like I got lucky and I would like to know what causes it
EDIT EDIT:
Thank you for your help
Last edited:
No, the other workaround that is mentioned there, i.e.: https://forum.proxmox.com/threads/l...rnel-update-on-pve-7.97406/page-3#post-558890
Did you boot into the latest installed kernel? Otherwise you'll need to specify update-initramfs -u -k all to rebuild it for all kernels (or specify the version you want to boot).
Why did you do this? Did you get a prompt from LVM that a repair is required? If that is the case, your issue is likely not the same as in the other thread, and I would check the disk health.
Removed thethin_checkoptions above
Re-ranupdate-initramfs -uas recomended
Rebooted
After a few tries, I managed to get the system back up and running stable... but it doesn't feel like I solved it, it feels like I got lucky and I would like to know what causes it
EDIT EDIT:
Thank you for your help
Oh, I didn't see that further down, you mean this? https://forum.proxmox.com/threads/local-lvm-not-available-after-kernel-update-on-pve-7.97406/page-3#post-558890:~:text=EDIT: INCREASING THE UDEV TIMEOUT DOES WORK
I will try this, thank you
Valid question, I assumed update-initramfs was updating the current kernel. I'll have to double-check that... thanks
Mentioned in the other thread, I was getting this when trying to start the VMs:
Code:
cannot perform fix without a full examination
Usage: thin_check [options] {device|file}
Options:
{-q|--quiet}
{-h|--help}
{-V|--version}
{-m|--metadata-snap}
{--auto-repair}
{--override-mapping-root}
{--clear-needs-check-flag}
{--ignore-non-fatal-errors}
{--skip-mappings}
{--super-block-only}
TASK ERROR: activating LV 'images/images' failed: Check of pool images/images failed (status:1). Manual repair required!I didn't know if that's related to this problem (given that images/images was not active at all), or something else entirely. I also don't know what is meant by "Manual repair required" and couldn't find anything that directly explained that message (maybe I missed something). After googling frantically and searching the forums, I came up with a post that recommended trying lvconvert to repair it. Didn't help... didn't seem to hurt either. But that's how I got there.
What is the recommended action for resolving the "Manual repair required!" message?
Last edited:
The above and the skip-mappings option will only help if the reason for activation failure is that there was a timeout.
But in your case, it's likely not a timeout, but corruption or disk errors. First thing I'd check is the disk health with e.g.
smartctl. And also the system logs/journal for any messages regarding the underlying device.If there are indeed disk errors, you should try to salvage the data to another disk.
If not, what does
lvconvert --repair images/images -v show? You might need to run lvchange -an images to deactivate the pool first.There also exists an alternative implementation that has been reported to be able to fix other issues: https://github.com/jthornber/thin-provisioning-tools
Is there a way to know, from the journal or otherwise, if there was a timeout? When I looked in the journal boot messages (journalctl -b) I saw nothing indicative of a problem
I have looked through the journal for any errors, and found none. When I ran "lvconvert --repair images/images" I didn't use "-v" but I can tell you it didn't report any errors.
I can't take this cluster down for testing, it's a development system that's highly in-use. I will have to run through some checks when there is system downtime, and I can risk not being able to reboot it.
Currently we're working on a full system backup. That matches with your recommendation, to salvage the data. When that completes, I will have more flexibility in terms of risking taking the system down without being sure I can restart it.
Thanks for the help, I will repost here when I am able to do maintenance on the cluster
Last edited:
Also, follow-up question: I have three systems in this cluster, all set up as standalone systems. They are identically configured, but they do not share the "images" volumes. They all have exactly the same problem, with the same messages, and I went through the same processes to get the to restart.
Do you feel that all three nodes in the cluster are having identical disk issues, independently?
Do you feel that all three nodes in the cluster are having identical disk issues, independently?
IIRC (but disclaimer, I looked into the issue was years ago), it's not logged by default.
Should the
manual repair required message come again, please share the verbose output. Feel free to post the full journal for a problematic boot.You can still try the method with increasing the udev timeout. I'd not put the
--skip-mappings then.Okay, that sounds very unlikely. In rare cases, if the hardware was bought at the same time, identical disks, it could still happen, so I'd not rule it out completely, but yes, very unlikely.
Hi all. With these commands (lvchange -an pve/data_tdata, lvchange -an pve/data_tmeta and lvchange -ay pve/data) I solved the problem, but every time I have to do it, I have a problem. I have 24TB of LVM-Thin storage, every time I have to use the lvchange command, the nightly backup snapshot of the VM takes 18 hours (usually 20 or 30 minutes) the first time after using lvchange, then it takes the normal time for the next day, until I have to repeat lvchange (if I have to reboot the server, of course). the snapshot backup is done on Proxmox Backup Server. Does anyone have any idea why every time I use the commands indicated, I have an exaggerated slowdown on the first snapshot backup I do? Thanks in advance to anyone who can help me :]