Web login and SSH broken after upgrade from 8.4 to 9.x

T

therealivandrago

Member
Jun 6, 2023
31
5
13
Hi and thanks for the help!

Not sure where I went wrong. I was following __ the proxmox upgrade steps. Ran for about 20 minutes.

Last I recall was running `grub-update` then `reboot` then no more access.

Was smart enough to get a paste bin before I cleared the terminal. https://pastebin.com/mzgXdVbC the password is fsfXjjHAiq

Just want to know where I went wrong so I can fix it. all the VMs are worthless tbh except for the trueNAS install, that one I kinda need.
 
It seems like the upgrade flipped your host to Debian 13 without a valid PVE repo/key, so APT removed proxmox-ve and friends; then a GRUB 2.12 update + update-grub finished the brick.

You might want to try this - Boot the PVE installer in “Rescue Shell”, mount your root, restore proper PVE 8/9 repos and the Proxmox keyring, then reinstall the bootloader (proxmox-boot-tool or GRUB) and pve-manager.

If your host boots from ZFS, ensure the correct PVE kernel/initramfs is selected before rebooting (avoid mixing generic Debian kernel + ZFS dkms here).

When back up, run pve8to9 and follow the official 8→9 steps end-to-end before another full-upgrade.

Your TrueNAS VM disks shouldn’t be touched by this; once the host boots and PVE stack returns, you should be able to re-start that VM.

All the best.
 
  • Like
Reactions: therealivandrago
readyspace said:
It seems like the upgrade flipped your host to Debian 13 without a valid PVE repo/key, so APT removed proxmox-ve and friends; then a GRUB 2.12 update + update-grub finished the brick.

You might want to try this - Boot the PVE installer in “Rescue Shell”, mount your root, restore proper PVE 8/9 repos and the Proxmox keyring, then reinstall the bootloader (proxmox-boot-tool or GRUB) and pve-manager.

If your host boots from ZFS, ensure the correct PVE kernel/initramfs is selected before rebooting (avoid mixing generic Debian kernel + ZFS dkms here).

When back up, run pve8to9 and follow the official 8→9 steps end-to-end before another full-upgrade.

Your TrueNAS VM disks shouldn’t be touched by this; once the host boots and PVE stack returns, you should be able to re-start that VM.

All the best.
Click to expand...
I know how to do maybe half of what you mentioned :rolleyes: wish me luck!

p.s. Thank you for the guidance!
 
readyspace said:
It seems like the upgrade flipped your host to Debian 13 without a valid PVE repo/key, so APT removed proxmox-ve and friends; then a GRUB 2.12 update + update-grub finished the brick.

You might want to try this - Boot the PVE installer in “Rescue Shell”, mount your root, restore proper PVE 8/9 repos and the Proxmox keyring, then reinstall the bootloader (proxmox-boot-tool or GRUB) and pve-manager.

If your host boots from ZFS, ensure the correct PVE kernel/initramfs is selected before rebooting (avoid mixing generic Debian kernel + ZFS dkms here).

When back up, run pve8to9 and follow the official 8→9 steps end-to-end before another full-upgrade.

Your TrueNAS VM disks shouldn’t be touched by this; once the host boots and PVE stack returns, you should be able to re-start that VM.

All the best.
Click to expand...
so far I created the repos based on the proxmox documentation.

I did check the directory where the keyring is and it still exists also.

I also managed to get the /boot/efi reattached to nvmeon1p2 using fstab, prior was seeting /boot not found. I booted into the PVE ISO and then grabbed the drive UUID (figured nvmeon1p2 was the EFI due to the 1G file size) I don't use ZFS to boot so i just ran the following commands to reinstall grub then update grub

I can log into the proxmox console locally only now, the web gui is still down. Also the network is completely not working right now, testing pinging google.com and a internal address both are failing :/

Not sure what next steps to take, funny enough I was doing the upgrade so it could be the same version as a new proxmox machine so I can join them and then setup my 1 - 2 - 3 backup

I also attached some of the journalctl output, seeing pveproxy.service is not found. not sure if that helps

1760419163212.png
 
Looks like your host now boots into plain Debian — pveproxy and spiceproxy are missing, which means proxmox-ve and pve-manager were removed during the upgrade.

Re-enable the proper PVE repo in /etc/apt/sources.list.d/pve-enterprise.list (or no-subscription repo) and reinstall with apt install proxmox-ve pve-manager.

Then fix /etc/network/interfaces to restore your bridge (vmbr0) config and bring networking back.

Once the web UI starts again, check systemctl status pveproxy and confirm your cluster/network settings.

Your TrueNAS VM disks should still be intact — just recover the host stack first before importing it back.
 
readyspace said:
Looks like your host now boots into plain Debian — pveproxy and spiceproxy are missing, which means proxmox-ve and pve-manager were removed during the upgrade.

Re-enable the proper PVE repo in /etc/apt/sources.list.d/pve-enterprise.list (or no-subscription repo) and reinstall with apt install proxmox-ve pve-manager.

Then fix /etc/network/interfaces to restore your bridge (vmbr0) config and bring networking back.

Once the web UI starts again, check systemctl status pveproxy and confirm your cluster/network settings.

Your TrueNAS VM disks should still be intact — just recover the host stack first before importing it back.
Click to expand...
I thought maybe proxmox got removed lol but I was following the official guide to upgrade and strangely didn't see any errors when running the recommended checks.

That said I think I can def manage re-installing proxmox but have some questions:

1. "Recover the host stack" what does this mean?

2. How do i fix /etc/network/interfaces? I was trying yesterday. Feel free to drop a link since a write up may be a lot of work.
 
An update:

I confirmed the non subscription repo's and both debian repos are setup per https://pve.proxmox.com/pve-docs/pve-admin-guide.html#sysadmin_no_subscription_repo

Tried reinstalling proxmox and the pve manager
Code: 
root@pve:~# apt install proxmox-ve pve-manager
Package proxmox-ve is not available, but is referred to by another package.
This may mean that the package is missing, has been obsoleted, or
is only available from another source
However the following packages replace it:
  proxmox-kernel-helper


Package pve-manager is not available, but is referred to by another package.
This may mean that the package is missing, has been obsoleted, or
is only available from another source

Error: Package 'proxmox-ve' has no installation candidate
Error: Package 'pve-manager' has no installation candidate
root@pve:~#


running apt update states failed to parse proxmox-archive-keyring.gpg due to no such file or directory, but the file is there : /
Code: 
root@pve:~# apt update
Hit:1 http://deb.debian.org/debian trixie InRelease
Hit:2 http://deb.debian.org/debian trixie-updates InRelease
Hit:3 http://ftp.us.debian.org/debian trixie InRelease
Hit:4 http://ftp.us.debian.org/debian trixie-updates InRelease
Hit:5 http://security.debian.org trixie-security InRelease
Get:6 http://download.proxmox.com/debian/pve trixie InRelease [2,771 B]
Err:6 http://download.proxmox.com/debian/pve trixie InRelease
  Sub-process /usr/bin/sqv returned an error code (1), error message is: Error: Failed to parse keyring "/usr/share/keyrings/proxmox-archive-keyring.gpg"  Caused by:     0: Reading "/usr/share/keyrings/proxmox-archive-keyring.gpg": No such file or directory (os error 2)     1: No such file or directory (os error 2)
Hit:7 http://security.debian.org/debian-security trixie-security InRelease
Warning: OpenPGP signature verification failed: http://download.proxmox.com/debian/pve trixie InRelease: Sub-process /usr/bin/sqv returned an error code (1), error message is: Error: Failed to parse keyring "/usr/share/keyrings/proxmox-archive-keyring.gpg"  Caused by:     0: Reading "/usr/share/keyrings/proxmox-archive-keyring.gpg": No such file or directory (os error 2)     1: No such file or directory (os error 2)
Error: The repository 'http://download.proxmox.com/debian/pve trixie InRelease' is not signed.
Notice: Updating from such a repository can't be done securely, and is therefore disabled by default.
Notice: See apt-secure(8) manpage for repository creation and user configuration details.
root@pve:~#


Based on the above I also tried instead
Code: 
apt update --allow-unauthenticated
but produces the exact same error as above.

Not too sure where to go next since it's not letting me reinstall proxmox
 
another update, I found from going down another rabbit whole this post that showed how to reinstall the keyring.

Next ran apt update and apt install proxmox-ve pve-manager. The web portal is up, neat! still googling furiously to figure out how to fix the broken VMBR0 does not exist

So far things are looking good!
 
  • Like
Reactions: readyspace
Current error hitting apply on the network settings is "command 'ifreload -a' failed: exit code 1". I copied the formatting from my fresh proxmox 9 install to fix the networking from my broken proxmox upgrade, why? Just to avoid trial and error.

The output is below and included screenshot of the interface settings

Code: 
enp1s0 : warning: enp1s0: <bridge.bridge object at 0x7cf6347b8c20>: error getting dependent interfaces (misconfiguration of bridge attribute(s) on existing non-bridge interface (enp1s0))
TASK ERROR: command 'ifreload -a' failed: exit code 1

This is what vmbr0 looks like

1760473252262.png

This is what enp1s0

1760473294744.png
 
Last edited:
Based on the error, it means enp1s0 is still configured as a normal interface instead of being in manual mode, so it can’t be used as a bridge port.
Edit /etc/network/interfaces, remove any IP lines under enp1s0, and make it look like this:

Code: 
auto enp1s0
iface enp1s0 inet manual

auto vmbr0
iface vmbr0 inet static
    address 192.168.1.177/24
    gateway 192.168.1.1
    bridge-ports enp1s0
    bridge-stp off
    bridge-fd 0

Save and run ifreload -a again

If all im understanding is correct, it should work.
 
I suggest in future to pay careful attention to apt, it "will" tell you what packages its going to remove before you approve its actions.
 
And forgot to reply I got it working a few minutes after this post.

TY so much @readyspace couldn't have done it without you, for real! onto the next problem now :D


therealivandrago said:
Current error hitting apply on the network settings is "command 'ifreload -a' failed: exit code 1". I copied the formatting from my fresh proxmox 9 install to fix the networking from my broken proxmox upgrade, why? Just to avoid trial and error.

The output is below and included screenshot of the interface settings

Code: 
enp1s0 : warning: enp1s0: <bridge.bridge object at 0x7cf6347b8c20>: error getting dependent interfaces (misconfiguration of bridge attribute(s) on existing non-bridge interface (enp1s0))
TASK ERROR: command 'ifreload -a' failed: exit code 1

This is what vmbr0 looks like

View attachment 91668

This is what enp1s0

View attachment 91669
Click to expand...
 
  • Like
Reactions: readyspace
You must log in or register to reply here.
Top Bottom