For some reason, my Proxmox server stopped responding earlier today, so I logged in with ssh, did a "reboot," and when it came back up, none of my VM's came up because my main LVM was missing. Upon further investigation, I see that it is trying to mount one UUID, but I don't even see that UUID on my system, and the disk that is supposed to be that drive is completely different. Is there a way to re-attach or re-assign a drive to the volume? I don't really understand how this happened, I've been running this thing for over two years with no problems until this.
Disk UUID changed somehow, how to re-assign?
- Thread starter bepstein111
- Start date
Bash:
root@pve:~# lsblk -f
NAME FSTYPE FSVER LABEL UUID FSAVAIL FSUSE% MOUNTPOINTS
sda
├─sda1 zfs_member 5000 z_alpha 15731755732445305245
└─sda9
sdb
└─sdb1 LVM2_member LVM2 001 WyyKHr-CCsc-lO47-159y-3Q2B-O8Id-vyQjWh
sdc
├─sdc1
├─sdc2 vfat FAT32 1E50-6481 510.7M 0% /boot/efi
└─sdc3 LVM2_member LVM2 001 1ol4W5-UbJ6-Pfc7-HEuc-PGHj-Kxch-6WxLJ6
├─pve-swap swap 1 a105bae5-c06e-4839-8025-b7125ca4183a [SWAP]
└─pve-root ext4 1.0 0a543d55-12f5-4642-b95d-ad1180c6614d 12.5G 73% /
sdd
└─sdd1 ext4 1.0 aae4af9f-0b57-4218-9bd0-b46218de93c7 328.4G 77% /mnt/pve/LaCie
zd0
zd16
├─zd16p1
└─zd16p2 zfs_member 5000 boot-pool 792684971688526867
root@pve:~# pvscan
PV /dev/sdc3 VG pve lvm2 [232.38 GiB / 166.38 GiB free]
WARNING: Couldn't find device with uuid AhtisH-vct5-LfYl-n5C7-mhAn-f2Wc-ctIhlz.
WARNING: VG alpha is missing PV AhtisH-vct5-LfYl-n5C7-mhAn-f2Wc-ctIhlz (last written to /dev/sdc1).
PV [unknown] VG alpha lvm2 [<1.82 TiB / 0 free]
PV /dev/sdb1 VG alpha lvm2 [<1.82 TiB / 0 free]
Total: 3 [<3.87 TiB] / in use: 3 [<3.87 TiB] / in no VG: 0 [0 ]
root@pve:~# lvscan
ACTIVE '/dev/pve/swap' [8.00 GiB] inherit
ACTIVE '/dev/pve/root' [58.00 GiB] inherit
WARNING: Couldn't find device with uuid AhtisH-vct5-LfYl-n5C7-mhAn-f2Wc-ctIhlz.
WARNING: VG alpha is missing PV AhtisH-vct5-LfYl-n5C7-mhAn-f2Wc-ctIhlz (last written to /dev/sdc1).
inactive '/dev/alpha/proxthin' [<3.64 TiB] inherit
inactive '/dev/alpha/vm-103-disk-0' [256.00 GiB] inherit
inactive '/dev/alpha/vm-108-disk-0' [8.00 GiB] inherit
inactive '/dev/alpha/vm-111-disk-0' [8.00 GiB] inherit
inactive '/dev/alpha/vm-100-disk-1' [256.00 GiB] inherit
inactive '/dev/alpha/vm-100-disk-2' [256.00 GiB] inherit
inactive '/dev/alpha/vm-100-disk-3' [256.00 GiB] inherit
inactive '/dev/alpha/vm-100-disk-4' [256.00 GiB] inherit
inactive '/dev/alpha/vm-100-disk-5' [256.00 GiB] inherit
inactive '/dev/alpha/vm-100-disk-6' [256.00 GiB] inherit
inactive '/dev/alpha/vm-100-disk-7' [128.00 GiB] inherit
inactive '/dev/alpha/vm-102-disk-0' [128.00 GiB] inherit
inactive '/dev/alpha/vm-101-disk-0' [4.00 MiB] inherit
inactive '/dev/alpha/vm-101-disk-1' [64.00 GiB] inherit
inactive '/dev/alpha/vm-113-disk-0' [40.00 GiB] inherit
inactive '/dev/alpha/vm-109-disk-0' [64.00 GiB] inherit
inactive '/dev/alpha/vm-106-disk-0' [40.00 GiB] inherit
inactive '/dev/alpha/vm-107-disk-0' [24.00 GiB] inherit
inactive '/dev/alpha/vm-104-disk-0' [40.00 GiB] inherit
inactive '/dev/alpha/vm-106-disk-1' [40.00 GiB] inherit
inactive '/dev/alpha/vm-106-disk-2' [40.00 GiB] inherit
inactive '/dev/alpha/vm-110-disk-0' [4.00 MiB] inherit
inactive '/dev/alpha/vm-110-disk-1' [100.00 GiB] inherit
inactive '/dev/alpha/vm-110-disk-2' [100.00 GiB] inherit
inactive '/dev/alpha/base-103-disk-1' [8.00 GiB] inherit
inactive '/dev/alpha/vm-103-cloudinit' [4.00 MiB] inherit
inactive '/dev/alpha/vm-102-cloudinit' [4.00 MiB] inherit
inactive '/dev/alpha/vm-101-cloudinit' [4.00 MiB] inherit
inactive '/dev/alpha/vm-107-cloudinit' [4.00 MiB] inherit
inactive '/dev/alpha/vm-112-cloudinit' [4.00 MiB] inherit
root@pve:~# vgscan
Found volume group "pve" using metadata type lvm2
WARNING: Couldn't find device with uuid AhtisH-vct5-LfYl-n5C7-mhAn-f2Wc-ctIhlz.
WARNING: VG alpha is missing PV AhtisH-vct5-LfYl-n5C7-mhAn-f2Wc-ctIhlz (last written to /dev/sdc1).
Found volume group "alpha" using metadata type lvm2
root@pve:~# blkid /dev/sdb
/dev/sdb: PTUUID="04941d8d-149a-45e1-a9d8-91321e78123d" PTTYPE="gpt"
root@pve:~# blkid /dev/sdb1
/dev/sdb1: UUID="WyyKHr-CCsc-lO47-159y-3Q2B-O8Id-vyQjWh" TYPE="LVM2_member" PARTUUID="678f480e-ee47-3149-96f8-69cbefb00e71"
root@pve:~# blkid /dev/sdc1
/dev/sdc1: PARTUUID="89425a5d-60bf-4ed4-b047-dd772861086b"
sdc1 indeed looks a bit lost. Its anybody's guess as to what happened: a power loss, a dying disk, a solar flare, misconfiguration.
Look into your recovery options, ie restore from backup if you have one. Or something like: https://access.redhat.com/documenta...on-an-lvm-physical-volume_troubleshooting-lvm
Good luck
Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox
I saw that, but I don't get how that's possible, /dev/sdc1 is the boot partition for the root drive, its just over 1000k, but pvscan shows that what its looking for is 1.82TiB (right?) which can only be /dev/sdb or /dev/sdd.
I dont know what the size of sdc1 is - thats not shown in any of your outputs.
I believe 1.82 is possibly the total size of LVM, but I am not sure about it.
If you believe it should be sdd which is currently formatted with ext4 and mounted, then you overwrote it at some point.
Go back in your logs and see if you can find the inflection point. Only you have the view of the entire system and ability to query and identify the drives. Only you know what the sizes should be and what matches/doesnt.
Any thoughts on what happened would be speculation, unless its backed by logs. Some things will never be explained.
Good luck.
Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox
I believe 1.82 is possibly the total size of LVM, but I am not sure about it.
If you believe it should be sdd which is currently formatted with ext4 and mounted, then you overwrote it at some point.
Go back in your logs and see if you can find the inflection point. Only you have the view of the entire system and ability to query and identify the drives. Only you know what the sizes should be and what matches/doesnt.
Any thoughts on what happened would be speculation, unless its backed by logs. Some things will never be explained.
Good luck.
Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox