Failed upgrade to pve 8 - need help

[chudak]

Hello all

Well, pve8 upgrade did not work for me.
All seems fine, followed all steps, no warnings, all VMs down etc. but now I have no VMs no GUI access, no pve

I still can ssh

See no version:
root@pve:~# pveversion
-bash: pveversion: command not found


root@pve:~/.ssh# grep '' /etc/apt/sources.list* /etc/apt/sources.list.d/*
/etc/apt/sources.list:deb http://ftp.us.debian.org/debian bookworm main contrib
/etc/apt/sources.list:
/etc/apt/sources.list:deb http://ftp.us.debian.org/debian bookworm-updates main contrib
/etc/apt/sources.list:
/etc/apt/sources.list:# security updates
/etc/apt/sources.list:deb http://security.debian.org bookworm-security main contrib
/etc/apt/sources.list:
/etc/apt/sources.list:#Community repo
/etc/apt/sources.list:# deb http://download.proxmox.com/debian/pve bookworm pve-no-subscription
/etc/apt/sources.list:
/etc/apt/sources.list:#Enable 2 line below for intel-microcode updates
/etc/apt/sources.list:#run as "apt-get install -t bookworm-backports intel-microcode"
/etc/apt/sources.list:# deb http://deb.debian.org/debian bookworm main contrib non-free
/etc/apt/sources.list:
grep: /etc/apt/sources.list.d: Is a directory
/etc/apt/sources.list.d/pve-enterprise.list:deb https://enterprise.proxmox.com/debian/pve bookworm pve-enterprise
/etc/apt/sources.list.d/pve-enterprise.list.dpkg-dist:deb https://enterprise.proxmox.com/debian/pve bullseye pve-enterprise



Debian was upgraded
root@pve:~# lsb_release -a
No LSB modules are available.
Distributor ID: Debian
Description: Debian GNU/Linux 12 (bookworm)
Release: 12
Codename: bookworm


what do I do now?

TIA

 

[chudak]

Found and changed in /etc/apt/sources.list.d/pve-enterprise.list.dpkg-dist:deb https://enterprise.proxmox.com/debian/pve bullseye pve-enterprise to
`/etc/apt/sources.list.d/pve-enterprise.list.dpkg-dist:deb https://enterprise.proxmox.com/debian/pve bullseye pve-enterprise`

Can I auto-remove the below?
And then?

root@pve:~/.ssh# apt dist-upgrade
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
Calculating upgrade... Done
The following packages were automatically installed and are no longer required:
g++-10 idn libboost-coroutine1.74.0 libbpf0 libcbor0 libdns-export1110 libicu67 libidn12 libipsec-mb0 libisc-export1105
libjs-qrcodejs libjs-sencha-touch libjson-glib-1.0-0 libjson-glib-1.0-common libleveldb1d libmpdec3 libnetaddr-ip-perl
libopts25 libperl5.32 libposix-strptime-perl libprocps8 libprotobuf23 libproxmox-acme-plugins libpython3.11 libpython3.9
libpython3.9-minimal libpython3.9-stdlib libstdc++-10-dev libtiff5 libtpms0 liburing1 libwebp6 libyaml-libyaml-perl
perl-modules-5.32 proxmox-archive-keyring proxmox-backup-client proxmox-backup-file-restore proxmox-backup-restore-image
proxmox-kernel-helper proxmox-mail-forward proxmox-offline-mirror-docs proxmox-offline-mirror-helper
proxmox-websocket-tunnel python3-ldb python3-talloc python3.9 python3.9-minimal qrencode swtpm swtpm-libs swtpm-tools
telnet
Use 'apt autoremove' to remove them.
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.

 

[chudak]

I also see:

root@pve:~# apt update
Hit:1 http://security.debian.org bookworm-security InRelease
Hit:2 http://ftp.us.debian.org/debian bookworm InRelease
Hit:3 http://ftp.us.debian.org/debian bookworm-updates InRelease
Hit:4 https://enterprise.proxmox.com/debian/pve bookworm InRelease
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
All packages are up to date.
N: Repository 'Debian bookworm' changed its 'firmware component' value from 'non-free' to 'non-free-firmware'
N: More information about this can be found online in the Release notes at: https://www.debian.org/releases/bookworm/amd64/release-notes/ch-information.html#non-free-split

 

[chudak]

I don't know how I missed one place `bullseye` to `bookworm`

But I really don't want to do a fresh install and wonder if anyone can point me in the right direction on how to restart the failed upgrade


To make the matter even more interesting, I manage the pve host remotely and won't be able to get console access soon. So I need to do it from the ssh session.

Thx in advance!

 

[trudge]

First, edit your /etc/apt/sources.list, where ever you see a line which ends in non-free, append non-free-firmware.

e.g, where you see this

deb http://deb.debian.org/debian-security bookworm-security main contrib non-free

change it to

deb http://deb.debian.org/debian-security bookworm-security main contrib non-free non-free-firmware

Make sure you do it for all lines ending in non-free

Second, run -> apt update

Third, re-install Proxmox -> apt install proxmox-ve postfix open-iscsi

the apt autoremove it is asking you to run should become irrelevant at this point. (they are all the dependencies for proxmox, which you "uninstalled" by running the upgrade before updating the config of the enterprise repo.)

 

[chudak]

First, edit your /etc/apt/sources.list, where ever you see a line which ends in non-free, append non-free-firmware.

e.g, where you see this

deb http://deb.debian.org/debian-security bookworm-security main contrib non-free

change it to

deb http://deb.debian.org/debian-security bookworm-security main contrib non-free non-free-firmware

Make sure you do it for all lines ending in non-free

Second, run -> apt update

Third, re-install Proxmox -> apt install proxmox-ve postfix open-iscsi

the apt autoremove it is asking you to run should become irrelevant at this point. (they are all the dependencies for proxmox, which you "uninstalled" by running the upgrade before updating the config of the enterprise repo.

Do you expect it will "restore" my old pve configuration?

Does this look right to you '/etc/apt/sources.list'?

deb http://ftp.us.debian.org/debian bookworm main contrib non-free deb http://ftp.us.debian.org/debian bookworm-updates main contrib non-free # security updates deb http://deb.debian.org/debian-security bookworm-security main contrib non-free non-free-firmware

(if this is not right pls paste the correct '/etc/apt/sources.list')

PS: I am trying to be extra careful before doing the next steps
(Not to myself - it'd be nice if it were more careful before )

 

[trudge]

Your config doesn't need restoring, it's still there - you just need the binaries back..

deb http://ftp.us.debian.org/debian bookworm main contrib non-free non-free-firmware
deb http://ftp.us.debian.org/debian bookworm-updates main contrib non-free non-free-firmware

# security updates
deb http://security.debian.org bookworm-security main contrib non-free non-free-firmware

Then -> apt update
Then -> apt install proxmox-ve postfix open-iscsi

 

[chudak]

Your config doesn't need restoring, it's still there - you just need the binaries back..



Then -> apt update
Then -> apt install proxmox-ve postfix open-iscsi

My proxmox is back online and is happy !

Thank you a million for your help!


This example made me think about better recovery options.
Even tho I have /etc/* backed up, I am thinking it's time for me to build a second pve server.

Do you think it will make such situations better?

 

[chudak]

Your config doesn't need restoring, it's still there - you just need the binaries back..



Then -> apt update
Then -> apt install proxmox-ve postfix open-iscsi

Do I need to keep `non-free non-free-firmware` in my repo sources?

 

[trudge]

Do I need to keep `non-free non-free-firmware` in my repo sources?

yes.

 

[trudge]

My proxmox is back online and is happy !

Thank you a million for your help!


This example made me think about better recovery options.
Even tho I have /etc/* backed up, I am thinking it's time for me to build a second pve server.

Do you think it will make such situations better?

well, you would have had less downtime if they are clustered or if you had a process to backup and restore between the two.

I think here you just needed the golden rules of upgrades.

1) Don't do it unless you *need* new features
2) If you do it, don't rush to be the first
3) Read twice, type once.

There are 2 things which happened here

1) Debian changed their repo structure, which is what caused this warning -> non-free non-free-firmware - nothing you could have done about this once you committed to upgrade. Upgrading early just caused you to be one of the first to encounter this.
2) you removed the proxmox software from your server due to your typo when changing the repos.