[SOLVED] After upgrade today, "error not a correct xfs inode"

[cdukes]

I updated one of our servers today to pve-manager/4.4-12/e71b7a74 (running kernel: 4.4.40-1-pve)
I was in an ssh console and the upgrade dropped my connection after:

Setting up corosync-pve (2.4.2-1) ...

And at that point, it looks like the server just rebooted on its own and never came back up.
I have console access now and find that it won't boot because it says:

error: not a correct XFS node.

It says that about 20 times, then has this at the bottom:

error: 'password_pbkdf2' is already loaded.

I found this, which seems relevant, but I don't know how to recover the boot.
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=802138;msg=31

My guess here is that your update did something with xfs and boot?

I have also uploaded the full SSH term log.

 

[fabian]

do you see the grub menu at all? what happens if you "press any key"? if you get dropped to a (rescue) shell, which one is it - Grub or initramfs?

are you able to boot using the "rescue mode" of the PVE 4.4 installer cdrom?

 

[cdukes]

pve rescue did not work. But I was able to boot using an Ubuntu Server 16.04 LTS install disk and then the rescue mode from that.
Once I booted, I repaired grub (from the menu), then mounted the drive, then re-ran update-grub.

The server now comes up and I can get to the UI. But I cannot log in. No error, just returns to the login screen.
I can't ssh in either, just drops connection.
The system logs show:

Mar 10 10:10:01 bfs CRON[6249]: PAM no modules loaded for `cron' service
Mar 10 10:10:01 bfs CRON[6249]: Permission denied

 

[cdukes]

I fixed the PAM error by booting up with the Ubuntu 16 server iso again and mounting the pve/root lvm volume.
Once mounted, I ran `pam-auth-update --force`, then rebooted and everything came up.

 

[newuser123]

do you see the grub menu at all? what happens if you "press any key"? if you get dropped to a (rescue) shell, which one is it - Grub or initramfs?

are you able to boot using the "rescue mode" of the PVE 4.4 installer cdrom?

hello Fabian and team, this happened to me as well - when I "press any key" everything freezes. I can't boot to Grub or a recovery shell. Please help if you have any tips. Using XFS on an HP Elitedesk Mini 705 G4, Ryzen 5 2400GE.

 

[fabian]

please open a new thread and describe your symptoms and your system:
- pveversion and how it was installed
- hardware
- when exactly which error happens

thanks!

 

[newuser123]

Posted, thank you
https://forum.proxmox.com/threads/n...oot-initrd-img-6-5-11-4-pve-not-found.137284/