PVE 7 to 8 upgrade gone wrong

[jimjam]

I have spent a couple hours on this already and haven't made it too far.

I ran through the upgrade procedure and ultimately ended up with an error with apt. I figured I would reboot (I knew it was a bad idea). My server would not boot.

There wasn't an entry in grub for the new 6.8.8-2-pve kernel nor was there an initramfs entry. I can boot into recovery mode on an older kernel version 5.15-xxx so I booted into that kernel and ran:

update-grub -c -k all
update-initramfs

This gave me an entry in grub and I tried to boot to it. This gets me to a screen that says this:

Found volume group "pve" using metadata type lvm2
6 logical volume(s) in volume group "pve" new active
/dev/mapper/pve-root: clean 187608/6291456 files, 4667332/25165824 blocks

The server will just sit here and do nothing for hours. Not sure where to go. Help is greatly appreciated.

Edit: Added actual update-grub command used

 

[fschauer]

To get more information on what makes the system hang, try removing the quiet option from the Linux kernel arguments.
You can do so by pressing e while having the boot entry selected in GRUB. Then go to the line that starts with linux and remove the quiet option. Then press F10 to boot.

 

[jimjam]

After removing quiet from linux kernel arguments, these are the last two lines output:

mounted sys-kernel-config.mount - kernel configuration file system
vfio_pci: add [8086:150e[ffffffff:ffffffff]] class 0x000000/00000000

 

[jimjam]

I have rebooted quite a few times now and it seems to be relatively random where the boot process stops.

I can boot into recovery mode if that means anything. Biggest difference I can see is IOMMU is disabled in recovery mode.

 

[jimjam]

The only recovery mode I can boot into is from an old kernel:

5.15.158-1-pve

 

[gfngfn256]

I have spent a couple hours on this already and haven't made it too far.

From experience & assuming you have fully restorable backups + all notes on your specific configs, this procedure will save you time:

Reinstall Proxmox - reconfigure storages etc. as they were & restore all LXCs & VMs from backups.

The above assumes that you haven't messed heavily with the PVE host for your specific setup (which you shouldn't).

 

[jimjam]

From experience & assuming you have fully restorable backups + all notes on your specific configs, this procedure will save you time:

Reinstall Proxmox - reconfigure storages etc. as they were & restore all LXCs & VMs from backups.

The above assumes that you haven't messed heavily with the PVE host for your specific setup (which you shouldn't).

I have started this process. I have a spare boot drive, so I took out my old one and put in the new one.

I installed Proxmox 8.x on it and it boots fine. I’ve noticed that it is running a different kernel than my upgraded install. Whatever moving on.

One question:

I dont have any notes on my old configuration, but I do have the old drive. I also have VM backups on my PBS server. I can see all of my VM configurations on my old drive so I can copy those settings if need be.

What I don’t have is network configurations. Do you know where in the file system those configurations would be?

 

[czechsys]

/etc/network/interfaces

 

[jimjam]

Ok, so setting this up from scratch is proving to be an even bigger nightmare that I imagined. I had a decent amount of virtual machines (including my pbs server) that run critical infrastructure (routers).

I think I would rather try and fix the old installation. Any idea on where to get a copy of the kernel image? I think it maybe corrupted.

 

[jimjam]

Alright, meeting somewhere in the middle here. My Proxmox install was on a separate drive compared to where I store all of my VM disks. So, I have copied all of my configuration files from my old install drive. Currently copying them to the new drive. Hopefully this will allow me to easily get my VMs running.

This might be a good time to note how bad of an idea it is to have your PBS server running on your main Proxmox host. I’d like to think if that were on a different machine, it would have been easier to restore, but I’m not sure.

 

[gfngfn256]

it would have been easier to restore, but I’m not sure.

Most definitely. PBS & for that matter any backup server, by its nature of being a backup, should be as independent & remote from the machine / service it is supposed to be covering for redundancy.

Your NW mappings/configurations - should all be detailed in saved notes that you make constantly.

 

[Kingneutron]

> This might be a good time to note how bad of an idea it is to have your PBS server running on your main Proxmox host

Bruv, I try to warn people about this alllllll the time. Homelabbers don't seem to care until they have an incident, and then it's FA&FO. Chicken/egg situation.

https://github.com/kneutron/ansitest/tree/master/proxmox

For future CYA - Look into the bkpcrit script, set target to non-root separate disk or NAS, put it nightly in cron