Proxmox (as a company) - what the HELL are you doing? Kernel update to 7 broke networking IN A VM

Kingneutron

Renowned Member
Feb 21, 2024
1,244
478
93
github.com
I use PDM and noticed a kernel upgrade available to 7.0 for my 3x PBS VMs. Upgraded (1) and rebooted it. Should be simple, right?

NO NETWORK after reboot. WTF?

This is my time off work for homelab maint, I'm not gonna try and troubleshoot that dumpster fire. Restored the VM from backup and all is fine now. If I have to pin the kernel to 6.17 I will.

But come on - if you guys want to be taken seriously as an Enterprise-level solution, DO NOT RELEASE BREAKING UPDATES!! I am not your beta tester, I rely on this stuff to be reliable to upgrade, even for homelab! Debian is supposed to be SIMPLE!

My $DAYJOB boss is having me evaluate Proxmox and PBS as a possible side-by-side solution for our Vmware ESXI stuff. There is NO WAY I can recommend going forward with proxmox at work if this kind of stuff keeps happening with what should be a run-of-the-mill upgrade process.

Please tell your devs to get their s--t together. Thank God I did this at home first instead of at work, your obviously untested rev would have made me look bad. I have been paying out of my own pocket for a support license for 2 years now, but this kind of breakage happening again would make me seriously rethink supporting proxmox going forward.

As an aside, this event made me rethink my backup strategy. I had been backing up my 3x PBS VMs about twice a week to other PBS VMs. If all 3 PBS instances had gone down at once after a simple upgrade... I have now scheduled .tar backups daily to local storage so I always have something to restore.
 
I use PDM and noticed a kernel upgrade available to 7.0 for my 3x PBS VMs. Upgraded (1) and rebooted it. Should be simple, right?

NO NETWORK after reboot. WTF?

This is my time off work for homelab maint, I'm not gonna try and troubleshoot that dumpster fire. Restored the VM from backup and all is fine now. If I have to pin the kernel to 6.17 I will.

But come on - if you guys want to be taken seriously as an Enterprise-level solution, DO NOT RELEASE BREAKING UPDATES!! I am not your beta tester, I rely on this stuff to be reliable to upgrade, even for homelab! Debian is supposed to be SIMPLE!

My $DAYJOB boss is having me evaluate Proxmox and PBS as a possible side-by-side solution for our Vmware ESXI stuff. There is NO WAY I can recommend going forward with proxmox at work if this kind of stuff keeps happening with what should be a run-of-the-mill upgrade process.

Please tell your devs to get their s--t together. Thank God I did this at home first instead of at work, your obviously untested rev would have made me look bad. I have been paying out of my own pocket for a support license for 2 years now, but this kind of breakage happening again would make me seriously rethink supporting proxmox going forward.

As an aside, this event made me rethink my backup strategy. I had been backing up my 3x PBS VMs about twice a week to other PBS VMs. If all 3 PBS instances had gone down at once after a simple upgrade... I have now scheduled .tar backups daily to local storage so I always have something to restore.
Seriously? Did you even try to understand what you were doing with this? 7.0 is not production release code. This was self inflicted unfortunately. I would suggest starting over after reading some basic installation materials and stick with 6.x code until 7 is ready for production.

So yes, if you are using kernel 7, you are a beta tester lol.
 
  • Like
Reactions: zodiac
No. I think it’s fair to voice my concerns here.

Anyone other than subscribers could be affected.

This is an opt-in feature, but it is located in the no-subscription repository rather than the test repository.

if it really was a kernel issue, he was the victim.

*Since I haven’t encountered the issue myself, I can’t say for sure without knowing the cause.


Known Issues:

None at the time of writing.

Edit 2026-04-20: the kernel is now also available on the no-subscription repository.
 
Last edited:
  • Like
Reactions: Kingneutron
Anyone other than subscribers could be affected.
It is not (yet) installed automatically during updates, at least not if you’re using the non-subscription repositories. You have to explicitly install it via apt install proxmox-kernel-7.0.

How do I know? I’m using the non-subscription repositories and just ran an apt full-upgrade on PBS and all my PVE hosts. And guess what I’m still on 6.17.13-3-pve. ;)
 
Last edited:
I'm sorry. It's fine if it's a test repository, but a “no-subscription” repository isn't intended for testing purposes.

Also, I don't think the “option” part implies it's for testing, does it?

If there really is an issue and testing hasn't been done yet, I think it's better not to put it in a repository like that.

If that was the intention, I think it's okay to voice your concerns.

*The important thing is whether there is actually a problem with the kernel.
 
Last edited:
Also, I don't think the “option” part implies it's for testing, does it?
3rd paragrapoh in the post you linked:

We have run this kernel on some of our test setups over the last few days without encountering any significant issues. However, for production setups, we strongly recommend either using the 6.17-based kernel or testing on similar hardware/setups before upgrading any production nodes to 7.0.

If there really is an issue and testing hasn't been done yet, I think it's better not to put it in a repository like that.
What do you mean by 'putting it in a repository like that'? They added it to the repository, but it doesn't upgrade by itself, so how is that an issue? Just because people can’t resist installing the latest version of everything straight away doesn't mean it shouldn't be released for those who want to test it, or for those who have newer hardware that could benefit from it, or might even need it.
 
Last edited:
a “no-subscription” repository isn't intended for testing purposes.
Let me quote the official wiki about the No-Subscription Repository
As the name suggests, you do not need a subscription key to access this repository. It can be used for testing and non-production use. It’s not recommended to use this on production servers, as these packages are not always as heavily tested and validated.

Packages are landing in the Test Repository first. Depending on feedback from the community and internal testing (iirc), it will be moved to the No-Subscription Repository and after some time with additional feedback from the community and internal testing, it will be moved to the Enterprise Repository.
 
I was mistaken. Since I don’t have a subscription, I suppose there’s nothing I can do about it.

But… to be treated almost exactly the same as a test repository? It’s sad that there’s nothing we can do about issues outside of Enterprise.
 
Last edited:
Let me quote the official wiki about the No-Subscription Repository


Packages are landing in the Test Repository first. Depending on feedback from the community and internal testing (iirc), it will be moved to the No-Subscription Repository and after some time with additional feedback from the community and internal testing, it will be moved to the Enterprise Repository.
Yes, but in this case it’s not really an issue at all, because even in the no-subscription repository you still have to explicitly install the new kernel. And if someone does that without fully reading the above post, that’s on them. And yes, if you use the testing repository, then even more so. ;)
 
I was mistaken. Since I don’t have a subscription, I suppose there’s nothing I can do about it.
Proxmox keeps at least one or two older kernel version installed, so it should still be possible to boot the previous kernel if needed.

But… to be treated almost exactly the same as a test repository? It’s sad that there’s nothing we can do about issues outside of Enterprise.

That’s simply not correct. It first appeared in the testing repository and only moved into the no-subscription repository about two days ago. And even now, it still isn’t installed by default—you only get it if you explicitly install it.

I’ve been using the no-subscription repositories for years, and they are generally very stable. In practice, no-sub is much closer to enterprise than to testing—if anything, it feels like (I don’t have exact data) it often takes longer for packages to move from testing to no-sub than from no-sub to enterprise.

Of course, there’s no guarantee that a bug won’t occasionally slip through. This is partly because the developers can’t realistically test new packages and kernels across as wide a range of hardware as the community uses them on after they’re released to the no-sub repos. However, you can drastically reduce the risk of encountering serious bugs by avoiding installing optional bleeding-edge packages, such as a new kernel. ;)
 
Last edited:
  • Like
Reactions: Stubennerd
Instead of complaining and just rejecting any responsibility for your own actions of installing an opt-in kernel marked as experimental, that was moved to no-subscriptions to make it easier to test without pulling in other updates for less experienced users, which we hardly can take serious, it would be really great to get actually some details about the setup and config here so that we can look into this. As we definitively take such reports serious, but for that to work we need - to repeat myself for the points' sake - details not ramblings so that we can look into actually reproducing and - depending on the outcome - fixing this.
 
I’ve been using the no-subscription repositories for years, and they are generally very stable.
I can confirm that. I've used the no-subscription repo on PVE from end of 2021 until end of 2024.
I remember having problems twice in that time. No big deal for me.
Since end of 2024 I'm using the Enterprise repo on PVE without any issues.

On PBS I'm using the no-subscription repo without any issue so far.
 
  • Like
Reactions: proxuser77
Wow, you guys are fast. I read the post, setup a new PBS to check if this is really the case and ended up knowing what you guys already wrote.

The problem OPs complaining is IMHO that PDM suggested to install the 7-series kernel, isn't it?

Edit: Just upgrading to kernel 7-series worked without any problem in my just-installed-from-4.0-iso version, so this is not a general problem and I suggest, as @t.lamprecht already wrote in providing more information.
 
Last edited:
The problem OPs complaining is IMHO that PDM suggested to install the 7-series kernel, isn't it?
It didn't for me. Either the OP is using the testing repos, in which case it might be already updated by apt by default (not sure), or OP deliberately installed the new kernel.

EDIT: The OP was actually talking about Proxmox Backup Server, but according to the opening post, if I understand correctly, the OP got the suggestion for the new 7-series kernel for those PBS instances via Proxmox Datacenter Manager. So I’m not sure how that would look in practice, as I’m not using PDM.

What I can definitely say is that, at the moment, with the no-subscription repositories, neither on my Proxmox Backup Server nor on Proxmox VE the 7-series kernel is suggested when running apt update && apt full-upgrade, nor does such a suggestion appear in the WebUI of those instances for me. However, I did receive an update for the 6.17 kernel on my PBS today: 2026-04-22 06:33:46 upgrade proxmox-kernel-6.17:amd64 6.17.13-2 6.17.13-3

I can confirm that. I've used the no-subscription repo on PVE from end of 2021 until end of 2024.
I remember having problems twice in that time. No big deal for me.
Yep, there can occasionally be issues, but mostly minor ones.

That said, a new kernel is a completely different story compared to a Proxmox package that might have a bug causing some feature to break somewhere, which you might not even be using. Those kinds of issues—unlike a kernel, which in the worst case can prevent the system from booting at all—are often not critical and can sometimes be worked around or even ignored for a while.

So I'd say the no-subscription repositories are generally fine for homelabs. However, I would only upgrade to a new optional kernel if I explicitly want to test it, or if I have hardware that requires it to run properly.

And just to mention it as well: if you still decide to upgrade to an optional bleeding-edge kernel package, you should at least be familiar enough with your system to know how to boot into an older kernel in case something goes wrong. ;)
 
Last edited:
What I don't get: Why would you add PBS to PDM in an enterprise setup in the first place? Any bad actor who manages to take over the PDM can also take over the PBS making the whole point of having backups useless.

Others already said enough on using experimental kernels or the non-enterprise repos for enterprise so I won't say anything further on it ;)
 
  • Like
Reactions: packetmonkey
hank God I did this at home first instead of at work, your obviously untested rev would have made me look bad.
running software at home and for production are two completely seperate skillsets, mindsets, and realms of responsibility. As others have pointed out, you opted to install an optional kernel, and got bit. it happens.

if you did that on a production environment without lab and approval and I was your boss (and aware of what you did) I would discipline if not fire you. My production clusters are still running PVE 8, and would have likely stayed there if not for looming EOS date.

More broadly, I wouldnt touch 7.0 for at least 2-3 point releases; its a HUGE upgrade and will likely have a lot of sharp edges at first- none of which the PVE devs have any real control over.
 
> What I don't get: Why would you add PBS to PDM in an enterprise setup in the first place

This upgrade was done at home, with homelab equipment. PBS 4 running as a VM under PVE 9.

This PBS instance was upgraded in-place from 3 to 4, so "network interface pinning" was IIRC not available at the original time of install.

PDM is designed to monitor both PVE and PBS instances so you have a "single pane of glass" quick overview. "Any bad actor" is why we have backups. Sorry if I flew off the handle a bit, mea culpa - sometimes it's the sysadmin with the PEBCAK. I got mad because I thought the net interface name changing was a fixed issue.

The PBS 3-to-4 VM running under PVE was on non-prod and Testing repos. I did not specifically ask to have the 7.0 kernel installed, it came up as available in PDM Remotes / Updates. I clicked the "Upgrade" button in PDM and it fired up the web UI for the PBS instance as usual to apply it. After the VM rebooted the web UI did not come back, and I noticed there was no network information at the PVE web GUI level. The VM console still displayed the IP address but it was unreachable, so as the quickest fix I just restored the PBS VM from backup.

Yes, I know all about booting to the previous kernel version in grub, thanx. This was an unexpected problem encountered after a full day of work. I even wrote code and published it almost 2 years ago to assist with fixing this issue. And the bug tracker has my response at the time.

Kindly be understanding and try to be forgiving of my original off-the-cuff response, I honestly expected this post to be deleted but it has some good discussion.

All my homelab instances have now been removed from Testing repos. Most are on no-subscription (who has the money if you're not running at least a small business) but I have my original node on Enterprise subscription to help support the company.

I'll have to look into interface pinning after-the-fact since this wasn't an original PBS 4 install. Hopefully this clears things up but feel free to ask if you need more info.