Node crashes on migration of vm - host is PVE 8.0.4

[fiona]

I give up. I destroyed the cluster and turned off my secondary node. The command pvecm delnode even sent the inbound node to grave and by that the inbound node still thinks it is member of the cluster....

That's unfortunate. What you still could've tried is using a bandwidth limit and also booting an older kernel to see if it's a regression there.

 

[esi_y]

Hi,

this is just not true for Proxmox VE. It does not have the same packaging guarantees as Debian. Repeating what I already said:

No, with Proxmox VE you can break your system when just using apt upgrade, just search the forum. Of course you should check which packages apt will remove, it will tell you.

I would take this opportunity (since this thread is already reached its end for the OP) to point out that:

1. I could not find this documented anywhere in:
https://pve.proxmox.com/wiki/Package_Repositories
It is the sole command used in major-to-major upgrade, but again this if anything implies the full-upgrade|dist-upgrade is necessary for this case only:
https://pve.proxmox.com/wiki/Upgrade_from_7_to_8

It's really A LITTLE STRANGE that even yourself could reference a Reddit post only regarding such an important distinction.

2. Considering PVE is based on Debian, even if PVE could be considered an appliance, and even if it does have DEDICATED COMMAND (which I was not aware of before), namely pveupgrade, but this is just a general wrapper around dist-upgrade, which of course will proceed to pull packages from all repos, not just PVE's - this is at the least mind-boggling, i.e. hypervisor that can have itself broken if run with the "safe" apt upgrade.

3. I do not even know where to start, if to consider the plain wrapper a logical bug, or if minor-to-minor upgrade require packages to be removed a bug in itself, or if to ask what is the logic behind major vs minor distinction in PVE's world.

4. Debian-based distributions seem to go the opposite way (in terms of being extra safe with apt upgrade, e.g. Ubuntu has even phased upgrades), i.e. they do not even dare to roll out packages to everyone at once to avoid a disaster, let alone remove packages. In fact, if running a standard "safer" upgrade command can ruin a hypervisor, why is that command not e.g. patched.

Given all of the above, I would suggest (at least some of the following):
a) update the PVE docs
b) improve the PVE wrapper (to only pull PVE and dependencies strictly)
c) patch a standard Debian command (I cannot believe this would be considered sane in any situation)

Now that I think of it, I wonder if e.g. having e.g. unattended-upgrades to pull just security related patches will not break PVE all of a sudden too (I guess not, but this finding did not instill confidence).

 

[fiona]

1. I could not find this documented anywhere in:
https://pve.proxmox.com/wiki/Package_Repositories

It's documented here: https://pve.proxmox.com/pve-docs/chapter-sysadmin.html#system_software_updates

 

[esi_y]

It's documented here: https://pve.proxmox.com/pve-docs/chapter-sysadmin.html#system_software_updates

I expected this kind of answer*, of course I had seen this same place quoted in the Reddit thread at the top.

Two issues with this:

5. What is the wrapper made for when even official PVE docs do not make use of it?

6. When I make a car that has left and right indicator controls reverse to the whole rest of the industry (this was not answered above conveniently, why), sure I can document it by a regular sentence telling the driver to use it the opposite way, or knowing how odd my design is, I could emphasize it at least in bold with explicit remark why (not in a Reddit thread, but in the official docs).

*It's very strange how time is spent on forum proving to everyone everything is good, when the same time (i.e. typing out one sentence) could have been invested into the docs improvement (at the least).

Now I am fully aware I have not paid for any support or anything like that to demand answers to the rest of the questions, but since already replying to me, is there really no good answers to the rest?

 

[esi_y]

It's documented here: https://pve.proxmox.com/pve-docs/chapter-sysadmin.html#system_software_updates

There's huge inconsistency how one can perceive PVE docs quality. On one hand, you had omitted to mention this odd behaviour explicitly, which would owe any (even potential serious customer) an explanation, on another you qoute yourself a comprehensive authoritative source on Bullseye [1], where in Ch. 6.2.3 "System Upgrade" one can find the following:

Regular upgrades are recommended, because they include the latest security updates. To upgrade, use apt upgrade, apt-get upgrade or aptitude safe-upgrade (of course after apt update).

...

For more important upgrades, such as the change from one major Debian version to the next, you need to use apt full-upgrade. With this instruction, apt will complete the upgrade even if it has to remove some obsolete packages or install new dependencies. This is also the command used by users who work daily with the Debian Unstable release and follow its evolution day by day. It is so simple that it hardly needs explanation: APT's reputation is based on this great functionality.

(emphasis mine)

Now nothing in your answer indicates the PVE Subscription only repos provide any different options than having to run dist-upgrade, so should one infer the whole PVE is nightly build quality (after initial major release, before next one)?


[1] Raphaël Hertzog, Roland Mas., Freexian SARL The Debian Administrator's Handbook: Debian Bullseye from Discovery to Mastery, Freexian, 2021. ISBN 979-10-91414-20-3; as quoted, retrieved as ISBN: 979-10-91414-22-7

 

[spirit]

There's huge inconsistency how one can perceive PVE docs quality. On one hand, you had omitted to mention this odd behaviour explicitly, which would owe any (even potential serious customer) an explanation, on another you qoute yourself a comprehensive authoritative source on Bullseye [1], where in Ch. 6.2.3 "System Upgrade" one can find the following:


...

(emphasis mine)

Now nothing in your answer indicates the PVE Subscription only repos provide any different options than having to run dist-upgrade, so should one infer the whole PVE is nightly build quality (after initial major release, before next one)?


[1] Raphaël Hertzog, Roland Mas., Freexian SARL The Debian Administrator's Handbook: Debian Bullseye from Discovery to Mastery, Freexian, 2021. ISBN 979-10-91414-20-3; as quoted, retrieved as ISBN: 979-10-91414-22-7

The difference between proxmox && debian, is that debian use a fixed packages list during the whole release.

Proxmox packages can have major upgrade during the major lifetime, with new depencies and newer packages.

apt-upgrade only upgrade already installed packages, but don't install any newer package (that's why it can break with proxmox, if an upgraded package depend a new package).

 

[esi_y]

The difference between proxmox && debian, is that debian use a fixed packages list during the whole release.

I understand that, but I posed questions 1-6 and you answered on number 3 partially.

Proxmox packages can have major upgrade during the major lifetime, with new depencies and newer packages.

If you meant "during [one release's lifetime]" then I take the issue of what is the purpose of calling one version a major when every "minor" upgrade is a major. This is still my question number 3 unanswered.

apt-upgrade only upgrade already installed packages, but don't install any newer package (that's why it can break with proxmox, if an upgraded package depend a new package).

There's the option to have something more clever done in the wrapper that does not do full-upgrade from all the repos. Otherwise why even have such a wrapper.

And of course my major issue is the lack of documentation of this (i.e. why were people wondering this in Reddit 2 years ago and now do again here)...

 

[jptechnical]

Hi,

this is just not true for Proxmox VE. It does not have the same packaging guarantees as Debian. Repeating what I already said:

E.g.: https://www.reddit.com/r/Proxmox/comments/ujqig9/use_apt_distupgrade_or_the_gui_not_apt_upgrade/

I deleted my prior comment. It was pedantic and didn't really help this thread. My bad, I want to be supportive of this community and that wasn't a supportive comment. Thanks for the commend @fiona

 

[esi_y]

I deleted my prior comment. It was pedantic and didn't really help this thread. My bad, I want to be supportive of this community and that wasn't a supportive comment. Thanks for the commend @fiona

May I ask what you thought was wrong with your prior comment (if I remember about dist|full-upgrade distinction and that it is meant to be used differently on other Debian-based systems)? I just do not appreciate how it is considered to be somehow unsupportive being argumentative (as in, having points to reason with) when in the end the product may improve, or the docs may improve. In this case, having e.g. proper wrappers around upgrading?

 

[esi_y]

There's the option to have something more clever done in the wrapper that does not do full-upgrade from all the repos. Otherwise why even have such a wrapper.

And of course my major issue is the lack of documentation of this (i.e. why were people wondering this in Reddit 2 years ago and now do again here)...

Adding the following (or similar e.g. echo Use the pveupgrade instead. See docs ...) to .bash_aliases would help prevent these posts:

apt() { if [[ $@ == "upgrade" ]]; then command apt full-upgrade; else command apt "$@"; fi; }

@fiona Any good reason why PVE does not come with such out of the box?

 

[fiona]

Hi,

Adding the following (or similar e.g. echo Use the pveupgrade instead. See docs ...) to .bash_aliases would help prevent these posts:

apt() { if [[ $@ == "upgrade" ]]; then command apt full-upgrade; else command apt "$@"; fi; }

@fiona Any good reason why PVE does not come with such out of the box?

See: https://lore.proxmox.com/pve-devel/64d5051c-8ed4-45af-998c-5399eb8d47ad@proxmox.com/

The clean solution would be to do it via an apt hook, but IIRC apt doesn't tell you whether upgrade or dist-upgrade was executed when invoking a hook (or at least I missed something when briefly looking into that in the past).

 

[esi_y]

Hi,

Thanks for the response!

See: https://lore.proxmox.com/pve-devel/64d5051c-8ed4-45af-998c-5399eb8d47ad@proxmox.com/

Alright, my bad for not following on that one, but that has a very straightforward solution at bottom of the .bashrc (you actually ship the stock
Debian / empty one for root):

case $- in
    *i*)
        if [ -f ~/.bash_aliases ]; then
                . ~/.bash_aliases
        fi
        ;;
esac

By extension, wrt to the linked conversation, rather then documenting a pitfall, the .bashrc might be auto-generated (with a # BIG DISCLAIMER on top) and then user is safe to put their thing in to ~/.bash_your_neofetch here.

With that out of the way ...

The clean solution would be to do it via an apt hook

I really would have better ideas about systemic solutions (something I do not want to go around in circles), but what I proposed above is going to eliminate recurrent troubleshooting of PVE installs (coming up on this forum non-stop) with not much complexity introduced.

, but IIRC apt doesn't tell you whether upgrade or dist-upgrade was executed when invoking a hook (or at least I missed something when briefly looking into that in the past).

I really do not think hooks are for this, do not want to fuel this again (last one was with Fabian [1]). I believe that the fact that there's no APT::Upgrade hook (but there is one for APT::Update) is supporting me. You would need to use DPkg::Pre-Install-Pkgs and fiddle with APT_HOOK_INFO_FD to probably be regex'ing CommandLine::AsString. Now before even considering that, this would kick in only after fetching all the packages (and what if I know what I am doing and actually want to get expected vanilla apt behaviour, as in the (non)-feature-report [1], with a hook, that's a headache), I can imagine the user experience (as bad as the current hook).

Compared to a shell function, which can literally just echo to root in interactive shell only to use pveupgrade (why else have it?) instead (while allowing to still do command apt) .. I do not see why not do that?

[1] https://bugzilla.proxmox.com/show_bug.cgi?id=5258

 

[fiona]

Thanks for the response!



Alright, my bad for not following on that one, but that has a very straightforward solution at bottom of the .bashrc (you actually ship the stock
Debian / empty one for root):

case $- in
    *i*)
        if [ -f ~/.bash_aliases ]; then
                . ~/.bash_aliases
        fi
        ;;
esac

By extension, wrt to the linked conversation, rather then documenting a pitfall, the .bashrc might be auto-generated (with a # BIG DISCLAIMER on top) and then user is safe to put their thing in to ~/.bash_your_neofetch here.

With that out of the way ...

Not everybody is using bash and we would like to avoid forcing certain shell configurations on people.

I really do not think hooks are for this, do not want to fuel this again (last one was with Fabian [1]). I believe that the fact that there's no APT::Upgrade hook (but there is one for APT::Update) is supporting me. You would need to use DPkg::Pre-Install-Pkgs and fiddle with APT_HOOK_INFO_FD to probably be regex'ing CommandLine::AsString. Now before even considering that, this would kick in only after fetching all the packages (and what if I know what I am doing and actually want to get expected vanilla apt behaviour, as in the (non)-feature-report [1], with a hook, that's a headache), I can imagine the user experience (as bad as the current hook).

Thanks! I did miss that we get the whole apt config (since in our current hook we just skip over it). I proposed a patch for this now: https://lists.proxmox.com/pipermail/pve-devel/2024-September/065249.html

 

[esi_y]

Not everybody is using bash and we would like to avoid forcing certain shell configurations on people.

Just to be clear (on the non-interactive shell protection), I did not expect you cover all the possibilities, but out of the box, for new user, even just a comment # put your stuff into .bash_interactive would save lots of mysterious ssh tunnel related troubleshooting here too.

Thanks! I did miss that we get the whole apt config (since in our current hook we just skip over it). I proposed a patch for this now: https://lists.proxmox.com/pipermail/pve-devel/2024-September/065249.html

As you can tell, I really do not like that hook. (I believe these hooks are for managing dpkg aftermath, not altering apt behaviour.)

But choices are yours, of course. I am all good with either way (I have that script purged on any install, but that's besides the point, I am in the should-have-known-better category, as would be people using shells other than bash).

Btw one thing to consider as you recognised in your comment is that there's certain types of people who do curl | bash anything and those would have apt -y update && apt -y upgrade on top, but at least those people would themselves (hopefully) consider to mention it when coming to the forum.

Your hook solves the common situation of piled up frustration of people who: 1) did not get success with ISO installer; 2) proceed with PVE on Debian; 3) install kernel; 4) report all hell broke loose on apt update && apt upgrade right after.

So thanks for that!

 

[esi_y]

No, with Proxmox VE you can break your system when just using apt upgrade, just search the forum. Of course you should check which packages apt will remove, it will tell you.

I have now looked at the list [1] follow-up. I understand people do not like me for bringing these things up, but I came here a year ago and the information above is ever since provided to everyone who had run "accidentally" upgrade only.

Then in the list [2] from @fabian:

you should only end up with a broken system if we miss properly tracking some inter-package relationship. it can happen happen (and probably does, from time to time), but in the vast majority of cases "apt[-get] upgrade" should at most leave you stuck with an outdated system (with APT telling you that there are still packages to be upgraded), not a broken one. we did get a lot better about accounting for these things over the past few years (but of course, we don't have anywhere close to the infrastructure that Debian has for automated tracking and testing).

Consider the messaging to the outside is confusing:

Is it a bug-to-be-reported when I end up with "broken" system upon upgrade or not? If it is, wouldn't you want to know and check dependencies every time this happens?

While at it, I found my reported bug on pve apt hook [3] that was WFM initially gets fixed 6 months later after all (thanks @fiona!).

I do not really know how to put it without hearing back "staff complaining about me", but all this is factual and very confusing experience. And most importantly it has nothing with bugs per se, there will always be plenty of things to fix up in software.

[1] https://lists.proxmox.com/pipermail/pve-devel/2024-September/065276.html
[2] https://lore.proxmox.com/pve-devel/232400452.25716.1725868091670@webmail.proxmox.com/
[3] https://bugzilla.proxmox.com/show_bug.cgi?id=5258