V5.1 Reboot Error - Volume group "pve" not found

[VM Magic]

Installed a new V5.1 server. Did all the apt-get updates. Server was running fine for a day. We then had to replace the RAID cache battery. Now upon reboot, we get the following:

Volume group "pve" not found
Cannot process volume group pve
Gave up waiting for root file system device.
.....
ALERT! /dev/mapper/pve-root does not exist. Dropping to shell.

This is a Dell R710 w/H700 RAID controller. Adding rootdelay=10 does not work. Looking in /dev/mapper only control is listed. lvm lvmscan returns nothing. Running vgchange -ay also has no effect.

Any thoughts or suggestions as to what could be wrong?

 

[dietmar]

We then had to replace the RAID cache battery.

RAID configuration lost?

 

[VM Magic]

RAID configuration is fine. Shows the drives and partition.

 

[fabian]

what happens if you boot some kind of live CD - are the PVs/VGs/LVs visible then?

 

[VM Magic]

Yes, they are visible when booting Ubuntu. lvm lvscan shows all the partitions.

 

[VM Magic]

/dev/pve/swap & root are active, /dev/pve/data is inactive.

 

[fabian]

Yes, they are visible when booting Ubuntu. lvm lvscan shows all the partitions.

I'd try chrooting into the PVE installation and regenerating the initramfs with "update-initramfs" then.

 

[VM Magic]

That didn't work. Seems I need to repair pve/data. However, I can't find a live DVD that includes
the thin provisioning tools. Any ideas?

 

[VM Magic]

Using CentOS live DVD shows no problem with the partition but Ubuntu and Proxmox do. Any ideas?

 

[VM Magic]

I was able to resolve the issue. Here are the following changes I made:

1. Commented out global_filter and filter from /etc/lvm/lvm.conf
2. Made the following changes to /etc/default/grub:
GRUB_CMDLINE_LINUX_DEFAULT="quiet rootdelay=10"
GRUB_CMDLINE_LINUX="scsi_mod.scan=sync"
GRUB_PRELOAD_MODULES="lvm"

 

[DaleS]

Hi All,

I just upgraded the kernel on a node from 4.4.134-1-pve to 4.15.18-2-pve
Please note the rest of the packages were already up to date, just the kernel versions are always victim to long times between true maintenance periods.

Upon reboot I got the dreaded 'Volume group 'pve' not found'.
Selecting the older kernel manually allows the system to boot without issue.

I followed VM Magic's advice above. I already had 'quiet rootdelay=15' in CMD_LINUX_DEFAULT, so I left that. I did however make every other change mentioned.

The system still will not boot with the latest kernel. A manual boot to 4.4.134-1-pve is fine. Anyone got any tips for me, or anything I should probe further while in busybox?

Thanks

*quick edit, yes I ran update-grub!

 

[pawsey]

did you find a solution to this issue? I just updated to
proxmox-ve: 5.4-1 (running kernel: 4.15.18-15-pve)
and experiencing the same issue.

 

[sebcbien]

looks like I have the same problem.
I posted here: https://forum.proxmox.com/threads/kernel-5-4-44-2-pve-problem.72377/

 

[Wild Wind]

I was able to resolve the issue. Here are the following changes I made:

1. Commented out global_filter and filter from /etc/lvm/lvm.conf
2. Made the following changes to /etc/default/grub:
GRUB_CMDLINE_LINUX_DEFAULT="quiet rootdelay=10"
GRUB_CMDLINE_LINUX="scsi_mod.scan=sync"
GRUB_PRELOAD_MODULES="lvm"

ProLiant DL360p Gen8, HW RAID1.
This solution solved my issue. I thought that it was a kernel issue, but if I wait a few more seconds and run new kernel manually it works fine.

Before that I tried to purge old kernels. Not helps.

 

[plastilin]

Hello. I have a similar problem. However, in my case, the RAID controller failed. When connecting a new controller, the disk configuration refused to be imported. I had to manually recreate the configuration in the controller utility without initializing the disks. What is the way out of this situation. The configuration is recreated identical to the one that was. The controller is exactly the same. I understand that my file system sizes have shifted, at least sgdisk swears at this However, I can't see lvm

 

[plastilin]

UPD

I made a backup of gpt partitions on another node since the logic of creating disks and the disks used are identical using the command:
# sgdisk --backup={/path/to/file} {/dev/device/here}

and restored it on the problem node using the command:

# sgdisk --load-backup={/path/to/file} {/dev/device/here}

At first, I was delighted because LVM disks of virtual machines appeared, but it turned out that the size of the disks is still slightly different. As a result, it turned out that the PV size is larger than the actual size of the disk. How can I correct the PV size according to the actual disk size?

Error following:

Warning: Device /dev/sdd1 has size of 974606303 sectors which is smaller than corresponding PV size of 975697887 sectors. Was resized?
Warning: One or more devices used as PVs in VG ssd have changed sizes.

 

[fabian]

that sounds like your "reinitialization" of the raid config somehow went wrong? if you didn't change the underlying physical disks, how can the (logical) disk be smaller now? it's not nothing either, we are talking about roughly half a GB of your 464GB disk!

 

[plastilin]

Unfortunately, there is no way to return to the old controller. Perhaps it was a firmware issue. It was older than on other servers. But now the question is how to backup LVM, if vgchange -ay does not work. I am already considering the option of directly correcting the lvm configuration through vgcfgbackup Maybe there are some ideas?

 

[fabian]

if your new RAID controller behaves differently than the old one, the only "safe" option is to restore using backups from before the raid controller change. it's probably still a good idea to take a full disk backup of the current state in order to attempt to recover individual files from after the last working backup. you effectively lost 500MB of disc contents, and without having before and after raw backups of the whole disk you don't know what was stored in those 500MB. and that is under the assumption that nothing else changed with the behavior except the truncation of the "logical" disk provided by the array, it could very well be that 10% of your data is corrupt now. of course you can try to convince LVM that the PV is only supposed to be that small, but that just papers over the issue with unknown consequences.

basically, a hardware raid array controller and the disks attached are a single entity - if the controller is gone, you should consider the disks gone as well and start over, unless your vendor guarantees something else.

 

[plastilin]

Unfortunately, I can not find software to solve the problem of information recovery. I need to restore entire LVM partitions and export them to raw. Now I'm trying to copy the data from the problem disk to a larger disk to test the theory that pv will stop complaining about the size.