after upgrade, unable to boot

why-be-banned

Member
Apr 4, 2023
56
4
8
After I executed the command apt upgrade && reboot, I can't start pve, I get this error:

批注 2025-03-12 001243.png


If I select kernel 6.8.12-4-pve, it can boot.

批注 2025-03-12 001301.png


As a temporary measure, I switched to the old kernel "pve-efiboot-tool kernel pin 6.8.12-4-pve".

What should I do ? thx.


After upgrade:

批注 2025-03-12 001559.png
 
apt upgrade && reboot
You should be using apt-get dist-upgrade, as shown here. Using apt upgrade may ( break | have broken ) your system.

Alternatively, you could use the Proxmox provided command (wrapper): pveupgrade (since ver. 3.1), which also uses the above command - with more info/advice.
 
Last edited:
You should be using apt-get dist-upgrade, as shown here. Using apt update may ( break | have broken ) your system.
Those aren't the same thing though...
apt update updates the repositories
apt upgrade updates the packages to their newest version in the repository
apt-get dist-upgrade updates the kernel, may have breaking changes, & should never be done casually like apt update && apt upgrade. You should only dist-upgrade when you are able to spend time making sure everything works after & have time to fix problems that arise. As well as having a backup before as well
 
  • Like
Reactions: why-be-banned
apt-get dist-upgrade updates the kernel, may have breaking changes, & should never be done casually like apt update && apt upgrade. You should only dist-upgrade when you are able to spend time making sure everything works after & have time to fix problems that arise. As well as having a backup before as well

that's not true for PVE, because it follows a rolling release model, which means that installing new packages or removing no longer working ones can be part of any upgrade, so you should always use dist-upgrade or full-upgrade. in the best case, you "just" get stuck using old versions if you use upgrade. in the worst case, we've missed some version constraint in a package and you are stuck on an incompatible combination of packages.
 
that's not true for PVE, because it follows a rolling release model, which means that installing new packages or removing no longer working ones can be part of any upgrade, so you should always use dist-upgrade or full-upgrade. in the best case, you "just" get stuck using old versions if you use upgrade. in the worst case, we've missed some version constraint in a package and you are stuck on an incompatible combination of packages.
That all sounds nice, but when 8 came out one of my nodes had to be rebuilt after a dist-upgrade so I cannot use that on things that I cannot have going down or when I don't have a couple hours to fix a problem
 
Last edited:
I have been using pve for over 5 years and have only ever used “upgrade” in each of several different clusters. a test on several different clusters also shows me no difference between “dist-upgrade” and “upgrade”.

i never had any problems with it, as i said on several clusters. luck or maybe not exactly as @fabian writes?
 
  • Like
Reactions: LostOnTheLine
i never had any problems with it, as i said on several clusters. luck or maybe not exactly as @fabian writes?
You are following the standard Linux procedures.
Due to the nature that installed packages can be removed by apt full-upgrade, it should not be called non-interactively.
apt full-upgrade is the apt equivalent of the apt-get apt-get dist-upgrade.

Anyone telling you that it should be used in a script or when you aren't there is not following Linux best practices.
Proxmox has always kept packages up to date & we have to manually full-upgrade when there's a new version... So that means internally it is using upgrade & not dist-upgrade on a regular basis.

In fact
You should be using apt-get dist-upgrade, as shown here. Using apt upgrade may ( break | have broken ) your system.
Is ironic because in general in Linux the practice of using upgrade regularly is exactly because apt-get dist-upgrade could Bork your system or certain packages.
 
Last edited:
So much misinformation: apt upgrade, dist-upgrade and full-upgrade are NOT the same. So too apt and apt-get commands are 2 different tools.
Upgrade will upgrade your packages, but not downgrade or remove conflicting packages, under no circumstances are currently installed packages removed, or packages not already installed retrieved and installed.
Dist-upgrade will upgrade your packages “intelligently” including dependencies and changes, so dist-upgrade may remove some packages
Full-upgrade (which is part of the apt front-end, not the apt-get backend) calls dist-upgrade but the apt tool is intended to be more interactive, depending on the other options you may have passed, it may or may not ‘do’ exactly the same thing under either one

If you are dealing with DKMS, Ceph, nVIDIA and other customizations - as said by the Proxmox Staff above, ALWAYS use dist-upgrade, that way you are sure that old things are cleaned up and ‘new’ dependencies installed. If you don’t, you may put yourself into a situation where your system functions a certain way (or doesn’t function properly) and then after a proper dist-upgrade, the behavior changes.
 
this makes no sense. practice shows that it does not cause any problems with countless clusters. to be exact, never had any problems with it. use ceph and co. various clusters run with unattended upgrades without problems, on enterprise repo. other clusters with manual “apt update && apt -y upgrade” no matter if enterprise or no-subscription simply work. most different hardware.

I would like to please understand technically with code, where this runs differently in pve than in debian.

If you don’t, you may put yourself into a situation where your system functions a certain way (or doesn’t function properly) and then after a proper dist-upgrade, the behavior changes.
your statement means that if i do a dist-upgrade, there might be changes. just run several times on different clusters, there are no changes.

apt full-upgrade is definitely an equivalent to apt-get dist-upgrade. it is also used by proxmox in its own instructions, see: https://pve.proxmox.com/wiki/Ceph_Quincy_to_Reef

for major upgrades of ceph from e.g. 17 to 18, i can understand the full-upgrade or dist-upgrade, but not in the normal “daily update business”. as already described, clusters have not only been getting security fixes for years without problems via unattended upgrades.
 
Last edited:
  • Like
Reactions: LostOnTheLine
It doesn’t run any differently in PVE than in Debian or Ubuntu, they all use the same package manager. The fact (or chance) YOU haven’t been bitten by it, doesn’t mean others haven’t been impacted. As I said, upgrade does not remove or install packages, that is what the man page says, even if they should be, that is a problem.

The nVIDIA drivers are notorious for having interdependencies that break version to version. There are others eg Dell DCISM where they compile for a non-rolling release of Debian/Ubuntu, which worked for a while, but since then relies on some low level C library (forgot which one) that is out of date on Ubuntu, but upgraded on PVE. If you do an upgrade, you get stuck but if you then follow the recommendations from dpkg, it basically breaks every compiled program that hooks that library, do a dist-upgrade and the old library is upgraded and DCISM is properly removed.
 
Last edited:
your statement means that if i do a dist-upgrade, there might be changes. just run several times on different clusters, there are no changes.
Yes dist-upgrade may cause changes, but it doesn't always or even usually cause changes.

apt upgrade, dist-upgrade and full-upgrade are NOT the same. So too apt and apt-get commands are 2 different tools.
apt upgrade is different but apt-get dist-upgrade & apt full-upgrade do the same thing.
As far as the differences between apt & apt-get there are some differences, sure, but they are the same tool. Some differences would be apt-get leaves behind things in cache while apt doesn't. In most uses apt is considered the newer, better, version of apt-get, though that's not 100% accurate, it's got a few more protections to prevent errors, I think it does a more complete hash check or maybe just does one where apt-get doesn't, & it has some less used things left off.
Before apt there was APT but it was a different thing. APT was the Administrative Packaging Tool & apt-get was the user tool to interact with it. There were also a few other tools like apt-cache & one for settings like apt-settings or apt-set & a few others that I don't remember because I never used them. apt was created as a cleaner interface for the APT that included all of the main functions of the APT that would be called by all the tools. So instead of needing to run apt-cache update or apt-cache search & apt-get install you'd use apt for all of them. But they all interact with the same APT. apt also turns on a few optional features that apt-get requires you to manually choose, like the progress bar for installing packages
Both apt-get dist-upgrade & apt full-upgrade send the same DoDistUpgrade command to APT. So they do the same thing. Since apt-get doesn't clean up it's cache after it finishes there might be some leftover cache files if you apt-get dist-upgrade that there wouldn't be with apt full-upgrade but that's pretty much it.

for major upgrades of ceph from e.g. 17 to 18, i can understand the full-upgrade or dist-upgrade, but not in the normal “daily update business”. as already described, clusters have not only been getting security fixes for years without problems via unattended upgrades.
The biggest difference is that say you run bobsprogramtodothething & there's an update. Both will do the update the same. But now say you have another package coolprogram7 that has an update. If you run apt-get upgrade coolprogram7 it might say that the program couldn't be upgraded, likely with a reason like "requires programextension9 which is not installed". if you ran apt upgrade coolprogram7 it will upgrade the main program, installing the dependencies the new version needs that the old didn't. Also if the old version used otherprogramextension4 to run that is now replaced with programextension9 then if you install programextension9 then run it again with apt-get upgrade coolprogram7 it will update but the unused otherprogramextension4 will still be installed even if it isn't used by anything as both apt upgrade & apt-get upgrade will not remove anything. apt-get does have an option to install new dependencies if needed apt-get upgrade [B]--fix-missing [/B]coolprogram7 & I believe that will do the same as apt except not with the progress bar.
apt full-upgrade on the other hand will see coolprogram7 installed & it will upgrade it, install the new dependencies, remove the old ones, & all's good... Except sometimes there are multiple packages that need things that can't exist together. In that case it will decide which is more important, upgrade that one & completely remove the other one, not even leave it at the old version.

The nVIDIA drivers are notorious for having interdependencies that break version to version. There are others eg Dell DCISM where they compile for a non-rolling release of Debian/Ubuntu, which worked for a while, but since then relies on some low level C library (forgot which one) that is out of date on Ubuntu, but upgraded on PVE. If you do an upgrade, you get stuck but if you then follow the recommendations from dpkg, it basically breaks every compiled program that hooks that library, do a dist-upgrade and the old library is upgraded and DCISM is properly removed.
That may be true for certain things, I'm surprised that a big comonly-used one is one of them because that usually only happens because something isn't built right. It happens on occasions for smaller, less-used packages, but that's a silly reason to could easily cause problems more often than solve them because one things is built wrong so it will break it. I'd honestly reconsider using a driver that has regular issues like that, instead I'd find an alternative one
 
Last edited:
this makes no sense. practice shows that it does not cause any problems with countless clusters. to be exact, never had any problems with it. use ceph and co. various clusters run with unattended upgrades without problems, on enterprise repo. other clusters with manual “apt update && apt -y upgrade” no matter if enterprise or no-subscription simply work. most different hardware.

I would like to please understand technically with code, where this runs differently in pve than in debian.

the difference is that Debian doesn't follow a rolling release model - once a release is declared stable, only targeted fixes/updates happen, and those don't change the structure of packages or their interdependencies (with some exceptions - the kernel might add a new package with a new ABI for example). in PVE, this is different - we might change some part of the code that requires changing two packages, and encode this fact in the package relationships. later on, one of those packages might get a new dependency or conflict, and if you are just using plain "upgrade", then you will get stuck on an older version. I am not saying this happens every week, but it can happen, and thus it is required to use one of the variants that allow more involved changes.

your statement means that if i do a dist-upgrade, there might be changes. just run several times on different clusters, there are no changes.

apt full-upgrade is definitely an equivalent to apt-get dist-upgrade. it is also used by proxmox in its own instructions, see: https://pve.proxmox.com/wiki/Ceph_Quincy_to_Reef

it's not 100% identical, but for the matter at hand, there is no real difference. "apt" has slightly nicer output, but the output is not stable. the next big upgrade of apt in Trixie will bring a lot of visual changes for example ;)

for major upgrades of ceph from e.g. 17 to 18, i can understand the full-upgrade or dist-upgrade, but not in the normal “daily update business”. as already described, clusters have not only been getting security fixes for years without problems via unattended upgrades.
well, you can believe the person with the most packaging experience in this thread, or not - I can tell you the recommendation is there for a reason :) our own APT wrapper "pveupgrade" (as used by the UI) also does "apt-get dist-upgrade" by the way.
 
That all sounds nice, but when 8 came out one of my nodes had to be rebuilt after a dist-upgrade so I cannot use that on things that I cannot have going down or when I don't have a couple hours to fix a problem

without more details I can't tell you what happened, but I can tell you from years of trouble shooting these things - in 99% of the cases it's a repository misconfiguration on the user end and not reading what apt(-get) tells you that causes such fallout. in particular when upgrading between major versions actually reading which packages will get installed/upgraded/removed is a must.
 
  • Like
Reactions: cheesecake
without more details I can't tell you what happened, but I can tell you from years of trouble shooting these things - in 99% of the cases it's a repository misconfiguration on the user end and not reading what apt(-get) tells you that causes such fallout. in particular when upgrading between major versions actually reading which packages will get installed/upgraded/removed is a must.
lol, it's fine, it was over a year ago, I fortunately had a recent backup, so it wasn't hard to revert, but it was down for a few hours. I then went & did a new build from scratch when I had more time, but it's an example of why, according to Debian, the base of what you're dealing with here, you should ONLY do a full-upgrade when you are at the terminal & have a recent backup. Like they say to not include it in scripts specifically. I also had a similar issue when I did a full-upgrade last year on one of my nodes that after a week or so after the upgrade stopped working. I had done the same on a different one of my nodes a few days after the first & was not planning on doing it on my last & main one until I had things migrated to the others, which was smart because the others went down without warning. The 2nd one was still working after the 1st went down for a few days as well, then went down too. I eventually discovered the cause & it was due to the fact that I had a lightweight GUI installed on all 3 of them, either XFCE or LXDE, & after the dist-upgrade it was the most confusing thing because they couldn't connect to the internet but could still RDP & SSH. I eventually discovered it was a network manager, I think the ConMan or something like that that was part of the junk that's installed by the Desktop. The same was still installed on the remaining un-upgraded node, & it was not causing any problems. But that was an upgrade that updated the version of Debian if I remember correctly.

the difference is that Debian doesn't follow a rolling release model - once a release is declared stable, only targeted fixes/updates happen, and those don't change the structure of packages or their interdependencies (with some exceptions - the kernel might add a new package with a new ABI for example). in PVE, this is different - we might change some part of the code that requires changing two packages, and encode this fact in the package relationships. later on, one of those packages might get a new dependency or conflict, and if you are just using plain "upgrade", then you will get stuck on an older version. I am not saying this happens every week, but it can happen, and thus it is required to use one of the variants that allow more involved changes.
In general practice you don't just ignore updates, but you do them at a planned time to prevent problems. From my experience Proxmox has never broken because I didn't update right away, but the opposite is true. I haven't, on the other hand, had Proxmox automatically upgrade me to a newer version without manual intervention, so it can't be true that PVE exclusively uses dist-upgrade. Updating packages, apt upgrade, is done at a much more frequent rate & shouldn't be the cause of things breaking. If that's Proxmox's way of doing things then Proxmox is not following Linux Best Practices. Period. Full Stop. Now I think Proxmox runs well, & whatever you do to ensure that major releases aren't automatically pushed works well, but if your setup is "Do this thing that could break things all the time so that it doesn't break things"... that's a clear conflict.
 
In general practice you don't just ignore updates, but you do them at a planned time to prevent problems. From my experience Proxmox has never broken because I didn't update right away, but the opposite is true. I haven't, on the other hand, had Proxmox automatically upgrade me to a newer version without manual intervention, so it can't be true that PVE exclusively uses dist-upgrade. Updating packages, apt upgrade, is done at a much more frequent rate & shouldn't be the cause of things breaking. If that's Proxmox's way of doing things then Proxmox is not following Linux Best Practices. Period. Full Stop. Now I think Proxmox runs well, & whatever you do to ensure that major releases aren't automatically pushed works well, but if your setup is "Do this thing that could break things all the time so that it doesn't break things"... that's a clear conflict.
you are still not understanding what I am telling you.

"dist-upgrade" is called like that because historically, it was the command used for upgrading from one major release to the next in Debian. it is not - on its own - causing an upgrade from one major version to the next. for that you need to also change your repository sources, which is a manual extra step.

for a regular Debian install, it is fine to use "apt upgrade", since there should never be changes that require "dist-upgrade" semantics within a single stable release.

for Proxmox, that is not the case - there are more changes within a stable release compared to Debian, so some upgrades will require "dist-upgrade" semantics. yes, that means if you somehow misconfigure your system, "dist-upgrade" might want to remove packages you want to keep (that is why we have a safeguard that prevents proxmox-ve from being removed). but the inverse is also true - if you only use "upgrade", you won't get all upgrades. and if the packages are missing some version constraint (mistakes do happen!), you might upgrade one package but not the other and the resulting combination is completely broken. this is the reason why PVE uses "dist-upgrade" and the documentation tells you to use it as well - so that APT will always attempt to upgrade all packages to the latest version, even if that requires installing extra packages or removing ones that no longer match. as a result, this is also the upgrade path and package set that is most tested. you can of course ignore that recommendation - it is your system - but you also get to keep the pieces afterwards when it breaks ;)
 
I'd just like to add to fabian's excellent analysis & comments, one point; Proxmox is supposed to be a hypervisor - not an AIO computer workstation, and as such the less tinkering, adding packages, dangling desktops on it etc. the more robust it will be. My rule of thumb is; EVRYTHING goes into an LXC or a VM, but NOT on the host. Yes there maybe something you absolutely cannot avoid that does not go into an LXC or VM, research it, document it & minimally install it - if it is truly unavoidable; it won't happen often, if at all. Yes, there is a wiki on how to install a desktop on Proxmox, but you can clearly read there the warning-on-the-label;
Proxmox VE is primarily used as virtualization platform with NO additional software installed. In some case it makes sense to have a full desktop running on Proxmox VE, for example for developers using Proxmox VE as their primary workstation/desktop.
Installing additional packages could lead to a hardly upgradeable system and is not supported from the Proxmox support team and therefore only for expert use.
In my experience an altered HV host system will eventually break.
 
"dist-upgrade" is called like that because historically, it was the command used for upgrading from one major release to the next in Debian. it is not - on its own - causing an upgrade from one major version to the next. for that you need to also change your repository sources, which is a manual extra step.
I don't remember the exact steps, but I do know that when doing dist-upgrade on one of my nodes just before I found this forum it went from 8.2.2 to 8.3.5. No repository changes or anything like that. nothing broke, but it definitely DID do an update that I didn't want it doing if I wasn't in front of the terminal.
for a regular Debian install, it is fine to use "apt upgrade", since there should never be changes that require "dist-upgrade" semantics within a single stable release.

for Proxmox, that is not the case - there are more changes within a stable release compared to Debian, so some upgrades will require "dist-upgrade" semantics.
You keep talking like Proxmox is just similar to Debian, but it isn't it IS Debian.
Bash:
 5:27 root@boxmox:~# cat /etc/os-release
PRETTY_NAME="Debian GNU/Linux 12 (bookworm)"
NAME="Debian GNU/Linux"
VERSION_ID="12"
VERSION="12 (bookworm)"
VERSION_CODENAME=bookworm
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"
The OS is Debian & it has things built on the Debian kernel. Sure it's not the same as running a standalone Debian machine or a Debian Desktop, But it's like saying this Windows machine that you use to run Firefox isn't a Windows machine... It still is... So the Windows rules still apply to it. Now the Edge rules don't apply because it runs Firefox, the Chrome rules don't apply, but the WINDOWS rules will always still apply. the Debian rules say specifically not to full-upgrade/dist-upgrade when you aren't physically there. Advise people to regularly dist-upgrade all you want, but as all the packages that are on the system are Debian packages & most of them aren't Proxmox packages making a generalized statement about how to do things on Windows contrary to Windows rules because they are better that way for Firefox means that Firefox is doing things the wrong way, not that people who follow the Windows rules are.
I'd just like to add to fabian's excellent analysis & comments, one point; Proxmox is supposed to be a hypervisor - not an AIO computer workstation, and as such the less tinkering, adding packages, dangling desktops on it etc. the more robust it will be. My rule of thumb is; EVRYTHING goes into an LXC or a VM, but NOT on the host.
Yes, I understand what you're saying & even agree, but I had a need to manage things outside of the WebUI on the host, it was by no means a regular workstation. The only things I had outside of VMs/LXCs were Docker & the desktop & xRDP. I've since moved away from needing to do that, but there were things that I lost or were made more inconvenient when I did. I literally now have a whole OneNote page of commands I regularly need to use that before I didn't because I could do them without having to type long commands. Most people aren't terminal monkeys who like to claim everything is better in terminal. & a bunch of things would have been a lot harder to figure out without a GUI, like changing the Favicons for my servers. Once I knew where the files were I could do it via script, but using a GUI to find them to begin with it was quite easy. Unfortunately Proxmox doesn't offer such a necessary option for people running multiple nodes, so I had to do it myself & have to run the script again any time there's a version update. Something I never have to do when using apt upgrade...

Some things, like Docker, in the older versions when run inside a VM or LXC had MAJOR limitations. That's not the case anymore, but it's an issue. Things like NetData or other system monitoring collectors also do not perform fully when installed inside a cage. You should always be careful, aware, & cautious when installing things like that, but there's plenty of reason to do so
 
Last edited:
I don't remember the exact steps, but I do know that when doing dist-upgrade on one of my nodes just before I found this forum it went from 8.2.2 to 8.3.5. No repository changes or anything like that. nothing broke, but it definitely DID do an update that I didn't want it doing if I wasn't in front of the terminal.
8.2.2 to 8.3.5 is not upgrading from one major release to another, both are based on Debian Bookworm. you would also have upgraded from 8.2.2 to 8.3.5 with "apt upgrade", just with a slightly higher chance of breaking things by accidentally doing a partial upgrade.

The OS is Debian & it has things built on the Debian kernel. Sure it's not the same as running a standalone Debian machine or a Debian Desktop, But it's like saying this Windows machine that you use to run Firefox isn't a Windows machine... It still is... So the Windows rules still apply to it. Now the Edge rules don't apply because it runs Firefox, the Chrome rules don't apply, but the WINDOWS rules will always still apply. the Debian rules say specifically not to full-upgrade/dist-upgrade when you aren't physically there. Advise people to regularly dist-upgrade all you want, but as all the packages that are on the system are Debian packages & most of them aren't Proxmox packages making a generalized statement about how to do things on Windows contrary to Windows rules because they are better that way for Firefox means that Firefox is doing things the wrong way, not that people who follow the Windows rules are.

Proxmox VE is what's called a "Debian Derivative", and it takes over a lot of core packages and does deviate from Debian behaviour in some areas (bootloaders and the installer are also handled differently, for example). you can refuse to believe me how PVE upgrades should be done and do things however you want - but like I said, please don't complain when you break things as a result.
 
8.2.2 to 8.3.5 is not upgrading from one major release to another, both are based on Debian Bookworm. you would also have upgraded from 8.2.2 to 8.3.5 with "apt upgrade", just with a slightly higher chance of breaking things by accidentally doing a partial upgrade.
That's definitely, provably not true. How do I know? Because 1 of my nodes, which I never fully setup & doesn't currently do anything is still on 8.2.2 after multiple apt upgrade processes while the other 2 have updated ONLY after a dist-upgrade on 1 & full-upgrade on the other
Proxmox VE is what's called a "Debian Derivative", and it takes over a lot of core packages and does deviate from Debian behaviour in some areas (bootloaders and the installer are also handled differently, for example). you can refuse to believe me how PVE upgrades should be done and do things however you want - but like I said, please don't complain when you break things as a result.
& you can say that all you like but I have 2 examples of where things were broken by not following the Debain standard. & I have one example of things "breaking" by following the Debain standard, & also quickly fixed by following the same standard, to do Dist-upgrade ONLY WHEN AT A TERMINAL. You keep going back to this mistaken premise of NOT doing it at all, rather than the Debian standard of doing it supervised. & I've seen no instances where doing it only when supervised caused a problem.

From my experience
Times things have gone wrong with Dist-Upgrade2
Times not using Dist-Upgrade has caused a problem1
Time & Difficulty recovering from Dist-Upgrade problemModerate
Time & Difficulty recovering from non Dist-Upgrade problemEasy