PBS goes crazy when upgrading to version 6.8.12-4.

[wisepds]

I have tried installing from the official iso and no problems. I update with Debian repositories and no problems.
The moment I update with the Proxmox repositories and the kernel version is updated to 6.8.12-4 the CPU consumption goes crazy and freezes the virtual machine.
I have documented it all in this X thread.

https://x.com/wisepds/status/1856398182948700605

Luckily I have kept everything on 6.8.12-3 where there is no problem.
I can replicate this bug with no problem and it has happened to people in my circle. From one day to the next the Proxmox Backup Server has stopped working.
How can this happen?
Does anyone know if this is an official bug or is there a patch?
Thanks to all.
Best regards.

 

[fernoliv]

Hi @wisepds, I’m having the same issues with mine PBS servers since I upgraded it to the latest version available on Debian/Proxmox repositories. There’s 5 servers where I did upgrade it and 2 of them are crazy with CPU usage going to 100% on all cores, turning the servers unusable. If the previous kernel is able to fix that weird behavior, I’m going to switch these 2 servers back to it in order to reduce the issues that I’m having for 3 days already.

 

[fernoliv]

@wisepds I realize that the issue is not with the kernel packages but with the proxmox-server-proxy bad version.

Downgrade it to the prior version and it’s back to the normal behavior.

Follow the commands below as stated by @meyergru and you can back to use your PBS normally until Proxmox team releases a new version fixing the issue:

apt-get install proxmox-backup-server=3.2.7-1
apt-mark hold proxmox-backup-server
reboot

As soon a new fixed version is available and you want to upgrade, use this command to unhold updates for proxmox-backup-server package:

apt-mark unhold proxmox-backup-server

And then you will be able to run "apt update" and "apt dist-upgrade" later in the future again. It's what you should to do, do not keep your PBS down if there's a workaround to overcome that temporary issue.

 

[wisepds]

Yeah! You'r right! Now 6.8.12-4 working!!!!
how i know when a fix is ready? Only testing?
Thank you for your help.. i think we can say now it's a bug!!!

 

[fernoliv]

Yeah! You'r right! Now 6.8.12-4 working!!!!
how i know when a fix is ready? Only testing?
Thank you for your help.. i think we can say now it's a bug!!!

View attachment 77626

That's great!

Proxmox team should work on a fix soon I think, as soon they consider it as a serious issue like I do.

I don't think it happened with people using their paid enterprise repository, possibly just on the free one. So that's normal because we act like "beta testers" when use non-subscription repositories (which it not designed for production, according with Proxmox), where looks like Proxmox releases updates firstly, before it be rock solid / production ready.

If you have a production environment like I do, I do recommend that you run apt update in the coming weeks and pay attention if there's a new version of proxmox backup server available. Just remember that your proxmox backup server package currently is in hold temporarily, so if you wants to upgrade it you to the latest version you will need to unhold the package using the command apt-mark unhold proxmox-backup-server.

 

[wisepds]

Yes yes, i got it! But for now i can update without problems... i have a production cluster with another pre-production machine where i test all updates before apply on production.
I always update on friday... because if a problem appears i have time to fix it...
About repository.. i think subscription repositorie is too expensive for now.. I would like to pay but i have 4 production machines and i think it's too expensive...
Thanks Fernilov... i'm now following you!

 

[fernoliv]

Yes yes, i got it! But for now i can update without problems... i have a production cluster with another pre-production machine where i test all updates before apply on production.
I always update on friday... because if a problem appears i have time to fix it...
About repository.. i think subscription repositorie is too expensive for now.. I would like to pay but i have 4 production machines and i think it's too expensive...
Thanks Fernilov... i'm now following you!

You’re welcome my friend

I do have the same thinking about the subscription costs, which is expensive for me too. I would like to support this amazing project that does a lot for my company and will do it as soon I can, but currently unfortunately I can’t afford for that as well.

 

[hunor]

Hi,

I had issues with 3.2.8-1
- CPU usage 100%
- The network connection sort of works, but GUI doesn't load
- disappearing from Proxmox VE

It looks like downgrading to 3.2.7-1 solves the problem. Thank you for the recommendation.

 

[fernoliv]

T

Hi,

I had issues with 3.2.8-1
- CPU usage 100%
- The network connection sort of works, but GUI doesn't load
- disappearing from Proxmox VE

It looks like downgrading to 3.2.7-1 solves the problem. Thank you for the recommendation.

I'm glad that it fixed your issue @hunor, thanks for your feedback. I was very concerned about this crazy CPU usage with no reason and now I'm alleviate that @meyergru posted a workaround to "fix" that really weird PBS behavior (https://forum.proxmox.com/threads/n...tart-nicht-mehr-erreichbar.157298/post-720144 - this is the original post where I found the right way to overcome the problem).

 

[Pcom]

@wisepds
apt-get install proxmox-backup-server=3.2.7-1
apt-mark hold proxmox-backup-server
reboot

As soon a new fixed version is available and you want to upgrade, use this command to unhold updates for proxmox-backup-server package:

apt-mark unhold proxmox-backup-server

Same problem, I'm surprised not many people are talking about this, my pbs machine keeps crashing with latest version.
Seems OK now with your temporary fix.

Thanks!

PS: to give more details my pbs runs as a VM inside Truenas 24.10.0.2, an AMD Ryzen 5600G, 8 GB ram

 

[fernoliv]

Same problem, I'm surprised not many people are talking about this, my pbs machine keeps crashing with latest version.
Seems OK now with your temporary fix.

Thanks!

You're welcome. That's funny because I do have 5 PBS servers and I did the same upgrade on all them together. I only had issues with 2 of 5 and they had exact the same version of Debian 12 and Proxmox non-subriscription repositories. Looks like these 2 specific PBS machines (bare metal as the others) triggered something on the proxmox-backup-proxy process that drained all the CPU cores in few minutes.

I don't get it either. Looks like it is not affecting everybody that upgrade their servers like we did, but specifically couple different systems under some specific conditions that I'm not sure why.