[SOLVED] After upgrade today, "error not a correct xfs inode"

cdukes

Renowned Member
Sep 11, 2015
89
10
73
Raleigh, NC
www.logzilla.net
I updated one of our servers today to pve-manager/4.4-12/e71b7a74 (running kernel: 4.4.40-1-pve)
I was in an ssh console and the upgrade dropped my connection after:
Code:
Setting up corosync-pve (2.4.2-1) ...
And at that point, it looks like the server just rebooted on its own and never came back up.
I have console access now and find that it won't boot because it says:
Code:
error: not a correct XFS node.
It says that about 20 times, then has this at the bottom:
Code:
error: 'password_pbkdf2' is already loaded.

I found this, which seems relevant, but I don't know how to recover the boot.
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=802138;msg=31

My guess here is that your update did something with xfs and boot?

I have also uploaded the full SSH term log.
 

Attachments

do you see the grub menu at all? what happens if you "press any key"? if you get dropped to a (rescue) shell, which one is it - Grub or initramfs?

are you able to boot using the "rescue mode" of the PVE 4.4 installer cdrom?
 
pve rescue did not work. But I was able to boot using an Ubuntu Server 16.04 LTS install disk and then the rescue mode from that.
Once I booted, I repaired grub (from the menu), then mounted the drive, then re-ran update-grub.

The server now comes up and I can get to the UI. But I cannot log in. No error, just returns to the login screen.
I can't ssh in either, just drops connection.
The system logs show:
Code:
Mar 10 10:10:01 bfs CRON[6249]: PAM no modules loaded for `cron' service
Mar 10 10:10:01 bfs CRON[6249]: Permission denied
 
I fixed the PAM error by booting up with the Ubuntu 16 server iso again and mounting the pve/root lvm volume.
Once mounted, I ran `pam-auth-update --force`, then rebooted and everything came up.
 
do you see the grub menu at all? what happens if you "press any key"? if you get dropped to a (rescue) shell, which one is it - Grub or initramfs?

are you able to boot using the "rescue mode" of the PVE 4.4 installer cdrom?
hello Fabian and team, this happened to me as well - when I "press any key" everything freezes. I can't boot to Grub or a recovery shell. Please help if you have any tips. Using XFS on an HP Elitedesk Mini 705 G4, Ryzen 5 2400GE.
 
please open a new thread and describe your symptoms and your system:
- pveversion and how it was installed
- hardware
- when exactly which error happens

thanks!