[SOLVED] Recovery image/Live CD?

[stefws]

Just lost one of my pve4.1 HP DL360 G9 hypervisor nodes as it got shot by HP' ASR/NMI during apt-get dist-upgrade when installing a new kernel/running grub (which takes too long most properly due to tasting all our iSCSI devices). Booting it now just makes the kernel crash.
This is so annoying, I've now turned off ASR in all my hypervisors BIOSs to avoid this happing again. We primarily utilize HA to separate certain VMs from every run on the same hypervisors, so f.ex. to SW clustered VMs never get hit by the same hypervisor failure/crash.

Anyway what would be a good live CD image to boot, so one could mount/fix the LVM fs of the node?

 

[tom]

Proxmox VE ISO (debug mode)?

 

[fabian]

Depends on what exactly got interrupted resp. what the exact error is, but I would wager that any Debian Jessie+ based Live system should work as well..

 

[stefws]

Proxmox VE ISO (debug mode)?

Only when selected and boot debug mode from install iso, my USB keyboard doesn't work at all

 

[stefws]

Booted install iso and switch to tty1 w/Ctrl-Alt-F1 and to a prompt w/Ctrl-C and find my logical volume but no device nodes for them, so impossible to mount them see attach image n3_lvs.jpg, n3_bootfailure.jpg shows the failure to mount boot after getting a NMI during grub install of a new kernel.

Why doesn't Debian/PVE keep at least one former kernel around when patching a new kernel like eg. Fedora/CentOS/RHEL in case booting the new kernel fails?

Will try to find a live image either uBuntu or +Jessie...

 

[tom]

your screenshot shows the LVM volumes, you just have to activate them (check LVM docs).

 

[stefws]

your screenshot shows the LVM volumes, you just have to activate them (check LVM docs).

right O, will try: vgchange -a y pve...

 

[stefws]

Improving, but not quite there yet. I boot from VE iso, break to a prompt, config simple network to get access to the world and then do this:

$ vgchange -a y pve
$ mkdir /tmp/a; mount /dev/pve/root /tmp/a
$ mount -o bind /sys /tmp/a/sys; mount -o bind /dev /tmp/a/dev; mount -o bind /proc /tmp/a/proc
$ chroot /tmp /bin/sh -c 'apt-get update; apt-get -y dist-upgrade'

but it fails with complains about:

E: Can not write log (Is /dev/pts mounted?) - posix_openpt (2: No such file or directory)

which is true, /dev doesn't hold any pts nodes. After this I still can't boot PVE, I end up in an initramfs prompt and exiting from this a kernel trace, see attached images.

Any hints on howto recover image without reinstalling?

 

[fabian]

1. you mount /dev/pve/root to /tmp/a and bind mount stuff there, but then you chroot to /tmp ?
2. try chrooting to /tmp/a, and then execute the commands individually:
$ chroot /tmp/a
# apt-get -y update
# apt-get -y dist-upgrade
3. try adding -y to apt-get update as well

 

[stefws]

1. you mount /dev/pve/root to /tmp/a and bind mount stuff there, but then you chroot to /tmp ?

Sorry a typo of course I chroot to /tmp/a

 

[stefws]

makes no diff. w/-y to update, see attached

 

[fabian]

You could try running "update-initramfs -u" and "update-grub2" and/or reinstalling the kernel package with apt-get in the chroot.

 

[stefws]

You could try running "update-initramfs -u" and "update-grub2" and/or reinstalling the kernel package with apt-get in the chroot.

Thanks, worked like charm, up and running again