[SOLVED] Borked my 8>9 upgrade, looking for help

nsne

New Member
Oct 26, 2025
7
1
3
I attempted to upgrade from 8.x to 9.x and something's gone horribly wrong. I can access the PVE from SSH, but it's spitting out errors and there's no GUI.

Before anyone asks, no, I violated the cardinal rule and did not make a backup before updating because I assumed my install was straightforward. Yes, it was stupid.

At any rate, here's the output from the SSH terminal:

Code:
root@worldland:~# pve8to9
Attempt to reload PVE/HA/Config.pm aborted.
Compilation failed in require at /usr/share/perl5/PVE/HA/Env/PVE2.pm line 20.
BEGIN failed--compilation aborted at /usr/share/perl5/PVE/HA/Env/PVE2.pm line 20.
Compilation failed in require at /usr/share/perl5/PVE/API2/LXC/Status.pm line 24.
BEGIN failed--compilation aborted at /usr/share/perl5/PVE/API2/LXC/Status.pm line 29.
Compilation failed in require at /usr/share/perl5/PVE/API2/LXC.pm line 28.
BEGIN failed--compilation aborted at /usr/share/perl5/PVE/API2/LXC.pm line 28.
Compilation failed in require at /usr/share/perl5/PVE/CLI/pve8to9.pm line 10.
BEGIN failed--compilation aborted at /usr/share/perl5/PVE/CLI/pve8to9.pm line 10.
Compilation failed in require at /usr/bin/pve8to9 line 6.
BEGIN failed--compilation aborted at /usr/bin/pve8to9 line 6.
root@worldland:~# pveversion
pve-manager/8.4.14/b502d23c55afcba1 (running kernel: 6.14.11-4-pve)
worldland:~# apt update
Hit:1 http://download.proxmox.com/debian/pve trixie InRelease
Err:2 https://enterprise.proxmox.com/debian/ceph-squid trixie InRelease
401  Unauthorized [IP: 66.70.154.82 443]
Reading package lists... Done
E: Failed to fetch https://enterprise.proxmox.com/debian/ceph-squid/dists/trixie/InRelease  401  Unauthorized [IP: 66.70.154.82 443]
E: The repository 'https://enterprise.proxmox.com/debian/ceph-squid trixie InRelease' is not signed.
N: Updating from such a repository can't be done securely, and is therefore disabled by default.
N: See apt-secure(8) manpage for repository creation and user configuration details.
root@worldland:~# apt policy
Package files:
100 /var/lib/dpkg/status
release a=now
500 http://download.proxmox.com/debian/pve trixie/pve-no-subscription amd64 Packages
release o=Proxmox,a=stable,n=trixie,l=Proxmox Debian Repository,c=pve-no-subscription,b=amd64
origin download.proxmox.com
Pinned packages:

... I assumed it was a repository issue (apt update was not kicking up any errors before the reboot) and I repeated the following upgrade steps:

Code:
root@worldland:~# sed -i 's/bookworm/trixie/g' /etc/apt/sources.list
root@worldland:~# sed -i 's/bookworm/trixie/g' /etc/apt/sources.list.d/pve-enterprise.list
root@worldland:~# cat > /etc/apt/sources.list.d/proxmox.sources <<EOF
Types: deb
URIs: http://download.proxmox.com/debian/pve
Suites: trixie
Components: pve-no-subscription
Signed-By: /usr/share/keyrings/proxmox-archive-keyring.gpg
EOF
root@worldland:~# apt update
Hit:1 http://download.proxmox.com/debian/pve trixie InRelease
Err:2 https://enterprise.proxmox.com/debian/ceph-squid trixie InRelease
401  Unauthorized [IP: 66.70.154.82 443]
Reading package lists... Done                       
E: Failed to fetch https://enterprise.proxmox.com/debian/ceph-squid/dists/trixie/InRelease  401  Unauthorized [IP: 66.70.154.82 443]
E: The repository 'https://enterprise.proxmox.com/debian/ceph-squid trixie InRelease' is not signed.
N: Updating from such a repository can't be done securely, and is therefore disabled by default.
N: See apt-secure(8) manpage for repository creation and user configuration details.
root@worldland:~# apt policy
Package files:
100 /var/lib/dpkg/status
release a=now
500 http://download.proxmox.com/debian/pve trixie/pve-no-subscription amd64 Packages
release o=Proxmox,a=stable,n=trixie,l=Proxmox Debian Repository,c=pve-no-subscription,b=amd64
origin download.proxmox.com
Pinned packages:

I really have no idea where to go from here. Are there any potential solutions that can save the install?

Also, during the upgrade process, I was not presented with any of the "confirm file changes" options.
 
Last edited:
Although you appear to have the Proxmox no subscription repo enabled, you do not have (any) Debian Base Repositories. See here & here.
 
Here is "typical" output for a PVE9 non-subscription server:
Code:
apt policy

Package files:
 100 /var/lib/dpkg/status
     release a=now
 500 http://download.proxmox.com/debian/pve trixie/pve-no-subscription amd64 Packages
     release o=Proxmox,a=stable,n=trixie,l=Proxmox Debian Repository,c=pve-no-subscription,b=amd64
     origin download.proxmox.com
 500 http://security.debian.org/debian-security trixie-security/main amd64 Packages
     release v=13,o=Debian,a=stable-security,n=trixie-security,l=Debian-Security,c=main,b=amd64
     origin security.debian.org
 500 http://deb.debian.org/debian trixie-updates/main amd64 Packages
     release v=13-updates,o=Debian,a=stable-updates,n=trixie-updates,l=Debian,c=main,b=amd64
     origin deb.debian.org
 500 http://deb.debian.org/debian trixie/contrib amd64 Packages
     release v=13.1,o=Debian,a=stable,n=trixie,l=Debian,c=contrib,b=amd64
     origin deb.debian.org
 500 http://deb.debian.org/debian trixie/main amd64 Packages
     release v=13.1,o=Debian,a=stable,n=trixie,l=Debian,c=main,b=amd64
     origin deb.debian.org
 500 http://download.proxmox.com/debian/ceph-squid trixie/no-subscription amd64 Packages
     release o=Proxmox,a=stable,n=trixie,l=Proxmox Ceph 19 Squid Debian Repository,c=no-subscription,b=amd64
     origin download.proxmox.com
Pinned packages:
 
The thing is, prior to the upgrade, I was getting the exact "apt policy" output that @gfngfn256 mentioned. I had configured all the repositories and everything looked okay. They were also appearing in the GUI as updated to the trixie repos. It's like the upgrade wiped all of them.
 
I had configured all the repositories

Please show output for the following (paste it into the online code-editor, ,marked '</>' on the formatting bar, as I have done):
Code:
grep '' /etc/apt/sources.list* /etc/apt/sources.list.d/*
 
Please show output for the following (paste it into the online code-editor, ,marked '</>' on the formatting bar, as I have done):
Code:
grep '' /etc/apt/sources.list* /etc/apt/sources.list.d/*

Code:
root@worldland:~# grep '' /etc/apt/sources.list* /etc/apt/sources.list.d/*
/etc/apt/sources.list:#deb http://deb.debian.org/debian trixie main contrib non-free-firmware
/etc/apt/sources.list:#deb http://deb.debian.org/debian trixie-updates main contrib non-free-firmware
/etc/apt/sources.list:#deb http://security.debian.org/debian-security trixie-security main contrib non-free-firmware
grep: /etc/apt/sources.list.d: Is a directory
/etc/apt/sources.list.d/ceph.list:# deb https://enterprise.proxmox.com/debian/ceph-quincy bookworm enterprise
/etc/apt/sources.list.d/ceph.list:# deb http://download.proxmox.com/debian/ceph-quincy bookworm no-subscription
/etc/apt/sources.list.d/ceph.list:# deb https://enterprise.proxmox.com/debian/ceph-reef bookworm enterprise
/etc/apt/sources.list.d/ceph.list:# deb http://download.proxmox.com/debian/ceph-reef bookworm no-subscription
/etc/apt/sources.list.d/ceph.sources:Types: deb
/etc/apt/sources.list.d/ceph.sources:URIs: https://enterprise.proxmox.com/debian/ceph-squid
/etc/apt/sources.list.d/ceph.sources:Suites: trixie
/etc/apt/sources.list.d/ceph.sources:Components: enterprise
/etc/apt/sources.list.d/ceph.sources:Signed-By: /usr/share/keyrings/proxmox-archive-keyring.gpg
/etc/apt/sources.list.d/proxmox.sources:Types: deb
/etc/apt/sources.list.d/proxmox.sources:URIs: http://download.proxmox.com/debian/pve
/etc/apt/sources.list.d/proxmox.sources:Suites: trixie
/etc/apt/sources.list.d/proxmox.sources:Components: pve-no-subscription
/etc/apt/sources.list.d/proxmox.sources:Signed-By: /usr/share/keyrings/proxmox-archive-keyring.gpg
/etc/apt/sources.list.d/pve-enterprise.list:# deb https://enterprise.proxmox.com/debian/pve trixie pve-enterprise
/etc/apt/sources.list.d/pve-install-repo.list:#deb http://download.proxmox.com/debian/pve trixie pve-no-subscription
/etc/apt/sources.list.d/pvetest-for-beta.list:# deb http://download.proxmox.com/debian/pve trixie pvetest

... so everything there is looking like it should, right? Or did I not configure something correctly?

[EDIT: Oh, wait. It looks like the repos in the Debian sources file are commented out. I did that because I thought the new apt sources were in the deb822 format. Should I un-comment those lines? Or use the "modernize-sources" command?]
 
Last edited:
Okay, I've now made some changes that I think are correct, but I'm still seeing errors and "pve8to9" won't run. This is the terminal output:

Code:
root@worldland:~# apt update
Hit:1 http://deb.debian.org/debian trixie InRelease
Hit:2 http://security.debian.org/debian-security trixie-security InRelease                                                                             
Hit:3 http://deb.debian.org/debian trixie-updates InRelease                                                                                           
Hit:4 http://download.proxmox.com/debian/pve trixie InRelease                                                                   
Err:5 https://enterprise.proxmox.com/debian/ceph-squid trixie InRelease     
  401  Unauthorized [IP: 66.70.154.82 443]
Reading package lists... Done
E: Failed to fetch https://enterprise.proxmox.com/debian/ceph-squid/dists/trixie/InRelease  401  Unauthorized [IP: 66.70.154.82 443]
E: The repository 'https://enterprise.proxmox.com/debian/ceph-squid trixie InRelease' is not signed.
N: Updating from such a repository can't be done securely, and is therefore disabled by default.
N: See apt-secure(8) manpage for repository creation and user configuration details.
root@worldland:~# apt policy
Package files:
 100 /var/lib/dpkg/status
     release a=now
 500 http://download.proxmox.com/debian/pve trixie/pve-no-subscription amd64 Packages
     release o=Proxmox,a=stable,n=trixie,l=Proxmox Debian Repository,c=pve-no-subscription,b=amd64
     origin download.proxmox.com
 500 http://security.debian.org/debian-security trixie-security/non-free-firmware amd64 Packages
     release v=13,o=Debian,a=stable-security,n=trixie-security,l=Debian-Security,c=non-free-firmware,b=amd64
     origin security.debian.org
 500 http://security.debian.org/debian-security trixie-security/main amd64 Packages
     release v=13,o=Debian,a=stable-security,n=trixie-security,l=Debian-Security,c=main,b=amd64
     origin security.debian.org
 500 http://deb.debian.org/debian trixie-updates/main amd64 Packages
     release v=13-updates,o=Debian,a=stable-updates,n=trixie-updates,l=Debian,c=main,b=amd64
     origin deb.debian.org
 500 http://deb.debian.org/debian trixie/non-free-firmware amd64 Packages
     release v=13.1,o=Debian,a=stable,n=trixie,l=Debian,c=non-free-firmware,b=amd64
     origin deb.debian.org
 500 http://deb.debian.org/debian trixie/contrib amd64 Packages
     release v=13.1,o=Debian,a=stable,n=trixie,l=Debian,c=contrib,b=amd64
     origin deb.debian.org
 500 http://deb.debian.org/debian trixie/main amd64 Packages
     release v=13.1,o=Debian,a=stable,n=trixie,l=Debian,c=main,b=amd64
     origin deb.debian.org
Pinned packages:
root@worldland:~# pve8to9
Attempt to reload PVE/HA/Config.pm aborted.
Compilation failed in require at /usr/share/perl5/PVE/HA/Env/PVE2.pm line 20.
BEGIN failed--compilation aborted at /usr/share/perl5/PVE/HA/Env/PVE2.pm line 20.
Compilation failed in require at /usr/share/perl5/PVE/API2/LXC/Status.pm line 24.
BEGIN failed--compilation aborted at /usr/share/perl5/PVE/API2/LXC/Status.pm line 29.
Compilation failed in require at /usr/share/perl5/PVE/API2/LXC.pm line 28.
BEGIN failed--compilation aborted at /usr/share/perl5/PVE/API2/LXC.pm line 28.
Compilation failed in require at /usr/share/perl5/PVE/CLI/pve8to9.pm line 10.
BEGIN failed--compilation aborted at /usr/share/perl5/PVE/CLI/pve8to9.pm line 10.
Compilation failed in require at /usr/bin/pve8to9 line 6.
BEGIN failed--compilation aborted at /usr/bin/pve8to9 line 6.
root@worldland:~# pveversion
pve-manager/8.4.14/b502d23c55afcba1 (running kernel: 6.14.11-4-pve)

What do I do now to safely upgrade to 9? Is it as simple as "dist-upgrade"? Should I address the ceph repo errors (I don't use ceph) first? And based on the upgrade guide, it looks "apt modernize-sources" shouldn't be run until I've successfully upgraded to 9, or is that not accurate?
 
Last edited:
You should remove all references to "enterprise.proxmox.com". For a first step only use the "no-subscription".
Then a "dist-upgrade" should work and leave your debian installation in a stable (trixie aka 13) state.
If you have a valid subscription you can add it afterwards.
 
Should I address the ceph repo errors (I don't use ceph) first?
Yes. Even if you don't use ceph, configure it to the no subscription repo.
shouldn't be run until I've successfully upgraded to 9, or is that not accurate?
AFAIK that should not make a difference.

pve-manager/8.4.14/b502d23c55afcba1 (running kernel: 6.14.11-4-pve)
That kernel (on PVE 8) was/is opt-in. Did you install that yourself? Is this the product of the "borked" upgrade?


I don't know the situation/history of your installation & update attempt(s). Was your install (prior to update attempt) fully updated? Did you have at that time full Debian/PVE/Ceph (all no subscription / other) repos correctly setup?

I assume you do not have a valid subscription, but if you do - this is probably the time to use a ticket, if you really need your current setup.

Maybe a cleanup/fix will be successful, but if not I would start again with a fresh install (PVE 9) & restore all VMs & LXCs from backups.
If you don't have backups of the VMs & LXCs (your OP is vague) - your workload is to be assumed non-critical, & should be re-constructable.

Good luck.
 
That kernel (on PVE 8) was/is opt-in. Did you install that yourself? Is this the product of the "borked" upgrade?
Yes, I assume it is the product of the borked upgrade. The only non-standard tinkering I've ever done is to install the Intel microcode, which involved appending "non-free-firmware" to the apt sources. Other than that, my PVE install was as vanilla as you can get.

I don't know the situation/history of your installation & update attempt(s). Was your install (prior to update attempt) fully updated? Did you have at that time full Debian/PVE/Ceph (all no subscription / other) repos correctly setup?
Yes, the install was 100% fully updated. And though I was pretty sure that I had all the repos set up correctly at the time, it's very possible that I did not based on the outcome.

After switching to the ceph-squid no-subscription repo during these troubleshooting steps, I'm not getting any errors when running "apt update". [EDIT for typo]

I assume you do not have a valid subscription, but if you do - this is probably the time to use a ticket, if you really need your current setup.
If I lose my current setup entirely, it will not be pretty because I have months of un-invoiced freelance job-tracking in a Kimai LXC. As I said, it was utterly stupid for me to upgrade without at least exporting that (although I did make an LXC backup within the PVE GUI). I really don't know what I was thinking, because I usually tend to be cautious to a fault.

There were about a half-dozen other VMs/LXCs that can be set up again fresh. Home Assistant, two Ubuntu VMs without much on them, an Immich LXC that had two test photos, that sort of thing. The Kimai LXC is my big concern.

Maybe a cleanup/fix will be successful, but if not I would start again with a fresh install (PVE 9) & restore all VMs & LXCs from backups.
If you don't have backups of the VMs & LXCs (your OP is vague) - your workload is to be assumed non-critical, & should be re-constructable.
If I can get it up and running from the current state, I plan to back everything up and then re-import it to a fresh install. Just in case there's some lingering cruft from the botched upgrade.

Before running the "dist-upgrade," one thing I considered is removing the SSD from the mini PC where PVE is installed, connecting it to my desktop as an external drive, and then scouring the data (ZFS format) to backup essential files to prevent any further data loss. But I'd really be just wandering in the dark there and don't know how identifiable/accessible anything would be.

To clone the PVE SSD to an image on my local (macOS) machine via SSH, would the following command work?

Code:
ssh root@{pve-LAN_IP-address} "sudo dd if=/dev/nvme0n1" | pv | dd of=/Users/{variable}/Desktop/PVEbackup.img

Good luck.
Thanks for all your help. Seriously, it's hugely appreciated.

Any other last-minute advice before I take the plunge into "apt dist-upgrade"?
 
Last edited:
apt upgrade
You should never run plain apt upgrade on a PVE server. It should rather be (as you will see is also done with the GUI button 'Upgrade'):
Code:
apt dist-upgrade

#or as is done in the Proxmox GUI with:
apt-get dist-upgrade
You can search these forums as to why.

although I did make an LXC backup within the PVE GUI
Saved backups could/should probably be retrievable even in your current situation. Attach another medium & copy them there.

one thing I considered is removing the SSD from the mini PC where PVE is installed
That may anyway be a good idea & installing PVE to a fresh disk - so that in any event you still have the old one for eventual retrieval.

To clone the PVE SSD to an image on my local (macOS) machine via SSH, would the following command work?
See my post here, what I use/do for full host image backup. Be terribly careful when using dd, it is a powerful steam-roller, that has the potential to wipe disks when making a singular erroneous character in the command line!


I am going to reiterate the absolute necessity of having backups of any semi-critical work. Also, storing backups on the same medium as the original (or OS that runs them), can often be considered as wasted space! You should at a minimum, have them stored on a separate medium.

Again, good luck.
 
One other thing, you should attempt to retrieve, is the VM/LXC configuration files.
They are usually available by searching:
Code:
#lxc
ls /etc/pve/nodes/<node>/lxc/

#vm
ls /etc/pve/nodes/<node>/qemu-server/
 
Okay, after several backups of the critical directories, I took the plunge and upgraded from PVE 8.x to 9.0.11. It brought the system back online, and I can now access it through the GUI. Most importantly, I can access my Kimai time-tracking LXC.

[EDIT: I should also note that "pve8to9 --full" indicated a PASS for every check.]

However, there are a few glitches, even after running the "apt modernize-sources" command. When I refresh the available updates for the node, I get the following output:

Code:
starting apt-get update
Hit:1 http://deb.debian.org/debian trixie InRelease
Hit:2 http://security.debian.org/debian-security trixie-security InRelease
Hit:3 http://deb.debian.org/debian trixie-updates InRelease
Hit:4 http://download.proxmox.com/debian/ceph-squid trixie InRelease
Hit:5 http://download.proxmox.com/debian/pve trixie InRelease
Err:6 https://enterprise.proxmox.com/debian/pve trixie InRelease
  401  Unauthorized [IP: 66.70.154.82 443]
Reading package lists...
E: Failed to fetch https://enterprise.proxmox.com/debian/pve/dists/trixie/InRelease  401  Unauthorized [IP: 66.70.154.82 443]
E: The repository 'https://enterprise.proxmox.com/debian/pve trixie InRelease' is not signed.
TASK ERROR: command 'apt-get update' failed: exit code 100

Below is a GUI screenshot of my current repositories.

Screenshot 2025-11-02 at 12.38.24 PM.png

To eliminate these error messages, should I disable the enterprise repo?

Also, when upgrading, I instinctively hit N for the "/etc/lvm/lvm.conf" prompt and did not install the package maintainer's version. Is there a way to install the package maintainer's version retroactively?
 
Last edited:
To eliminate these error messages, should I disable the enterprise repo?
If you don't have a valid subscription (as it would appear) - then yes.

Is there a way to install the package maintainer's version retroactively?
This should work - and should ask you again about which version to use:
Code:
apt-get --reinstall -o Dpkg::Options::="--force-confask" install lvm2
 
If you don't have a valid subscription (as it would appear) - then yes.

This should work - and should ask you again about which version to use:
This was wonderful. Everything seems to be working normally now. Thanks once again—sincerely—for all your help and patience.
 
  • Like
Reactions: gfngfn256
Happy you got fully sorted. Maybe mark this thread as Solved. At the top of the thread, choose the Edit thread button, then from the (no prefix) dropdown choose Solved.
 
  • Like
Reactions: nsne