I received this from one of our operations staff:
Two of the grid servers rebooted due to a power outage and failed to come back up properly.
The reason in both cases was a failed root filesystem fsck.
On looking at the systems, root is an LVM volume, and in /etc/rcS.d/, chkroot.sh comes before LVM is configured. The LVM devices and mapper don't exist to find the filesystem at this point.
The fix is to manually start services required for LVM, then run fsck (checking that root is mounted read-only), and reboot.
I've written a script to do this currently located on grid02 and attached (inline here since the forum says it is an invalid file)
=========================================
#!/bin/sh
# Karsten M. Self
# Thu Jan 22 23:18:04 GMT 2009
#
# If we fail fsck on boot we need to enable LVM before we can fsck.
#
/etc/init.d/glibc.sh start
/etc/init.d/mountkernfs.sh start
/etc/init.d/udev start
/etc/init.d/mountdevsubfs.sh start
/etc/init.d/libdevmapper1.02 start
/etc/init.d/lvm start
# Hail Mary fsck
mount -o remount,ro / && e2fsck -y /dev/mapper/pve-root
echo "Check for the smell of burning rubber. You should probably reboot."
Two of the grid servers rebooted due to a power outage and failed to come back up properly.
The reason in both cases was a failed root filesystem fsck.
On looking at the systems, root is an LVM volume, and in /etc/rcS.d/, chkroot.sh comes before LVM is configured. The LVM devices and mapper don't exist to find the filesystem at this point.
The fix is to manually start services required for LVM, then run fsck (checking that root is mounted read-only), and reboot.
I've written a script to do this currently located on grid02 and attached (inline here since the forum says it is an invalid file)
=========================================
#!/bin/sh
# Karsten M. Self
# Thu Jan 22 23:18:04 GMT 2009
#
# If we fail fsck on boot we need to enable LVM before we can fsck.
#
/etc/init.d/glibc.sh start
/etc/init.d/mountkernfs.sh start
/etc/init.d/udev start
/etc/init.d/mountdevsubfs.sh start
/etc/init.d/libdevmapper1.02 start
/etc/init.d/lvm start
# Hail Mary fsck
mount -o remount,ro / && e2fsck -y /dev/mapper/pve-root
echo "Check for the smell of burning rubber. You should probably reboot."