[SOLVED] PBS does not boot after kernel upgrade

[emcp]

Hi,

We have a HPE server with proxmox backup server on it. It is working great as production for a long time now.
We did upgrade to the latest version, and after the reboot, it does not find the LVM group anymore to boot from.
Volume group "pbs" not found Cannot process volume group pbs ALERT! /dev/mapper/pbs-root does not exist. Dropping to a shell!

If I go choose in grub an old kernel like 5.13.19-6-pve, it boot again and works as espected.

It fails with the latest kernels:
5.15.104-1-pve
5.15.102-1-pve

Only 5.13 and before worked.

Is there any issue with 5.15.x ?

The system is installed on a basic HPE server with 2 ssd configured in RAID1 within the HPE Raid utility. It's only for the OS, the datastores are mounted from elsewhere.

Thank for help

 

[Chris]

Hi,
please try to rebuild the initramfs after booting from the working kernel by running update-initramfs -u -k all. Do you see any errors/warnings during the initramfs creation? Also consider adding a rootdelay=10 to your kernel command line to pause for 10 seconds before trying to mount the root filesystem. Also, check if cat /proc/partitions shows your partitions from the rescue shell if the problem persists.

 

[emcp]

Hi,

I tried to update initramfs, with no luck:

╭─root@pbs-host1 ~
╰─➤  update-initramfs -u -k all
update-initramfs: Generating /boot/initrd.img-5.15.104-1-pve
I: The initramfs will attempt to resume from /dev/dm-0
I: (/dev/mapper/pbs-swap)
I: Set the RESUME variable to override this.
Running hook script 'zz-proxmox-boot'..
Re-executing '/etc/kernel/postinst.d/zz-proxmox-boot' in new private mount namespace..
No /etc/kernel/proxmox-boot-uuids found, skipping ESP sync.
update-initramfs: Generating /boot/initrd.img-5.15.102-1-pve
I: The initramfs will attempt to resume from /dev/dm-0
I: (/dev/mapper/pbs-swap)
I: Set the RESUME variable to override this.
Running hook script 'zz-proxmox-boot'..
Re-executing '/etc/kernel/postinst.d/zz-proxmox-boot' in new private mount namespace..
No /etc/kernel/proxmox-boot-uuids found, skipping ESP sync.
update-initramfs: Generating /boot/initrd.img-5.13.19-6-pve
I: The initramfs will attempt to resume from /dev/dm-0
I: (/dev/mapper/pbs-swap)
I: Set the RESUME variable to override this.
Running hook script 'zz-proxmox-boot'..
Re-executing '/etc/kernel/postinst.d/zz-proxmox-boot' in new private mount namespace..
No /etc/kernel/proxmox-boot-uuids found, skipping ESP sync.
update-initramfs: Generating /boot/initrd.img-5.13.19-1-pve
I: The initramfs will attempt to resume from /dev/dm-0
I: (/dev/mapper/pbs-swap)
I: Set the RESUME variable to override this.
Running hook script 'zz-proxmox-boot'..
Re-executing '/etc/kernel/postinst.d/zz-proxmox-boot' in new private mount namespace..
No /etc/kernel/proxmox-boot-uuids found, skipping ESP sync.

I already tried adding rootdelay=10, it does not make any difference.

cat /proc/partitions does show sda partitions as expected.

 

[emcp]

Now that I updated all initramfs (including the old kernel on) It does not boot anymore with 5.13 neither...

Is something missing in the initramfs config? Like some drivers for ext4/lvm2/... ?

 

[Chris]

Now that I updated all initramfs (including the old kernel on) It does not boot anymore with 5.13 neither...

Is something missing in the initramfs config? Like some drivers for ext4/lvm2/... ?

What does cat /proc/modules; ls /dev give you? What does blkid and lvm pvs tell you?

Yes, you probably are missing modules for your raid controller. What exact model is this?

 

[emcp]

The server is an HP DL360p G8 with a standard Smart Array P420i controller

Here are /proc/modules




And listing of /dev






Actually the Raid controller is recognized, because /dev/sdaX are present. It's LVM that does not find the device...

 

[fabian]

could you post the output of "lvm config" in the initrd? if it's too long to sensibly copy/screenshot, at least the "devices" and "activation" part would be interesting!

 

[emcp]

Oh damn... I found the culprit... A filter was added in lvm.conf because of multipath/iscsi. sdb was added instead of sda...

Sorry and thank you for the help to find the issue.

 

[fabian]

that was my hunch as well great that it works now!