[SOLVED] LVM-Thin blocks boot, and now is missing

[IsThisThingOn]

I think I somehow misunderstood and misconfigured LVM. I come from Windows and went straight into ZFS Fortunately only unimportant VMs are on that disk but any help would still be highly appreciated. Here is what I did:

My system uses two NVME mirror ZFS drives. Later on, some friends and I wanted to have a LinuxGameServer for Minecraft and stuff. I don't play these games but thought, sure I will just add another HDD and explained to them, that nothing is saved! This is my slow trash HDD.

I created the LVM Volume group (picture 1), a thinpool on that volumegroup (pic 2).


So today I updated proxmox and rebooted the system. EFI Stub takes quiet some time, but I got used to that. Later on it runs out the a timer on a startup job (that tries to mount the disk?) and goes into emergency mode.
I tried booting the old kernel but no luck. Drive seems to be working and cabeling seems also fine. So I removed the drive from fstab (pic 3)

and proxmox booted.

But now my slowthin with the VM disks is offline . If I click on it, I get this error pic .



This is my datacenter view:




Any idea where I went wrong?

 

[IsThisThingOn]

Using lvchange -an
from this blogpost and rebooting helped.
https://blog.talhamangarah.com/home..._volumes_after_failure_to_attach_on_boot.html

Everything seems to be working. But I am unsure if I should inster the entry in fstab again?

 

[IsThisThingOn]

So after a reboot, I am running into the same problem again

Any ideas how to fix this?

 

[fiona]

Hi,
you might be running into the issue from this thread and the workaround mentioned there could help. Likely the LVM-thin disk is just too big and/or slow to do the full check with the mappings.

 

[IsThisThingOn]

Hi fiona

That would also explain the very slow boot.
Is there a reason that this mapping check takes that long? Am I better of by switching to ZFS for the other disk?

 

[fiona]

Hi fiona

That would also explain the very slow boot.
Is there a reason that this mapping check takes that long? Am I better of by switching to ZFS for the other disk?

Well, I guess, because the disk is large there are a lot of mappings to check. And if the disk is also not fast that might take too long. You can try the workaround with disabling the check for the mappings, or switch to ZFS. Your choice

 

[IsThisThingOn]

Thanks, probably will switch later on to ZFS, when I have more time.
I got this error after setting the check options:

update-initramfs -u
update-initramfs: Generating /boot/initrd.img-5.15.53-1-pve
Running hook script 'zz-proxmox-boot'..
Re-executing '/etc/kernel/postinst.d/zz-proxmox-boot' in new private mount namespace..
Copying and configuring kernels on /dev/disk/by-uuid/A9DC-4C2C
        Copying kernel and creating boot-entry for 5.13.19-6-pve
        Copying kernel and creating boot-entry for 5.15.39-3-pve
        Copying kernel and creating boot-entry for 5.15.39-4-pve
cp: error writing '/var/tmp/espmounts/A9DC-4C2C/EFI/proxmox/5.15.39-4-pve/initrd.img-5.15.39-4-pve': No space left on device
run-parts: /etc/initramfs/post-update.d//proxmox-boot-sync exited with return code 1
root@proxmox:~#

 

[fiona]

Sounds like the ESP partition is full. Please see this thread or this thread for how to remove some old kernels to free up some space.

 

[IsThisThingOn]

proxmox-boot-tool status

ls -al /dev/disk/by-uuid/A9DC-4C2C
lrwxrwxrwx 1 root root 15 Sep 15 13:51 /dev/disk/by-uuid/A9DC-4C2C -> ../../nvme0n1p2


ls -al /dev/disk/by-uuid/A9DC-982D
lrwxrwxrwx 1 root root 15 Sep 15 13:51 /dev/disk/by-uuid/A9DC-982D -> ../../nvme1n1p2

mkdir /tmp/myesp0
mkdir /tmp/myesp1
mount /dev/nvme0n1p2 /tmp/myesp0
mount /dev/nvme1n1p2 /tmp/myesp1
df -h |grep myesp

shows 100% for myesp1



rm -r /tmp/myesp1/loader/entries/proxmox-5.13.19-6-pve.conf
rm -r /tmp/myesp1/initrd.img-5.3.18-3-pve
rm -r /tmp/myesp1/initrd.img-5.3.18-3-pve



After that, the update went trough!

 

[fixjunk]

sorry to resurrect this thread but I find myself having to repeat this process of cleaning out old kernels often.

how can I prevent it? this last time it was 6.5 sticking around.

 

[fxandrei]

I think you should just run apt autoremove after everyupdate (at least when a new kernel shows up).
I had this problem on one node, and not the other, because it seems that i did not run apt autoremove after the updates.