Proxmox VE 8.2 released!

[SInisterPisces]

Moved to Proxmox 8.2 yesterday and had a VM crash just now.
[deletia]

Supermicro IPMI is reporting the memory as aok. Host has been solid on 6.5.x for months.

This is a quad socket host with 4th gen Intel's.

With all the talk about lockups etc in this thread, im starting to question if 6.8.x was a good move.

I feel like we are compromising bleeding edge for stability with updates as of late.

That's a valid concern, but the odd thing about all this is that the Linux 6.8 kernel is the backbone of Ubuntu's current LTS release. It's not really supposed to be bleeding edge.

The 6.5.x kernel is now no longer officially supported by Ubuntu, so staying on it would have its own problems long term given that Proxmox's kernel development is tied to Ubuntu's kernel development cycle.

I'm not really sure what to make of this, but given reports of issues with basic functionality in Ubuntu 24.04 for some users, it seems like it's not just a Proxmox thing.

It sounds like kernel 6.8, 6.9, and probably 6.10 and 6.11 are all pushing or about to be pushing some major changes to how Linux works deep under the hood as far as how it talks to CPUs and GPUs and other low-level hardware, which is all to the good, but the reality is we might just be going through a rocky phase of hardware compatibility.

 

[fiona]

Hi,

Checking running kernel version..
WARN: unexpected running and installed kernel '6.8.4-3-pve'.

Looks like pve7to8 could use an update.

thank you for the report! A patch that should address the issue was sent to the mailing list: https://lists.proxmox.com/pipermail/pve-devel/2024-May/063886.html

 

[hvisage]

In Release Notes, I found the answer in section: Known Issues & Breaking Changes. I pinned the kernel to current 6.5 kernel and installation continued. Unfortunately, manually I need to remove the 6.8 kernel.

Yeah, why read Release notes, the devs don't know what they are talking about and we SysAdmins just rtFm when it's an Emergency :shrug:

 

[mrpops2ko]

i'm on a amd 7950x and was noticing daily random reboots, i've downgraded now to 6.5 kernel and pinned it. lets see how things go.

will report back in a week or so

and it rebooted
-- Reboot --
in the logs lol

seems downgrading the kernel didn't fix it
root@pve:~# uname -a
Linux pve 6.5.13-5-pve #1 SMP PREEMPT_DYNAMIC PMX 6.5.13-5 (2024-04-05T11:03Z) x86_64 GNU/Linux

so now i'll give reinstalling proxmox a shot, down to what i know was previously working that resulted in 100+ days of uptime

looking at my download history that was proxmox-ve_8.1-1.iso

so either my hardware has issues or something between proxmox-ve_8.1-1.iso to latest was changed that is causing issues for 7950x

edit: downgrade complete, lets see how we get on

root@pve:~# uname -a
Linux pve 6.5.11-4-pve #1 SMP PREEMPT_DYNAMIC PMX 6.5.11-4 (2023-11-20T10:19Z) x86_64 GNU/Linux
root@pve:~# pveversion -v
proxmox-ve: 8.1.0 (running kernel: 6.5.11-4-pve)
pve-manager: 8.1.3 (running version: 8.1.3/b46aac3b42da5d15)
proxmox-kernel-helper: 8.0.9
proxmox-kernel-6.5.11-4-pve-signed: 6.5.11-4
proxmox-kernel-6.5: 6.5.11-4
ceph-fuse: 17.2.7-pve1
corosync: 3.1.7-pve3
criu: 3.17.1-2
glusterfs-client: 10.3-5
ifupdown2: 3.2.0-1+pmx7
ksm-control-daemon: 1.4-1
libjs-extjs: 7.0.0-4
libknet1: 1.28-pve1
libproxmox-acme-perl: 1.5.0
libproxmox-backup-qemu0: 1.4.0
libproxmox-rs-perl: 0.3.1
libpve-access-control: 8.0.7
libpve-apiclient-perl: 3.3.1
libpve-common-perl: 8.1.0
libpve-guest-common-perl: 5.0.6
libpve-http-server-perl: 5.0.5
libpve-network-perl: 0.9.4
libpve-rs-perl: 0.8.7
libpve-storage-perl: 8.0.5
libspice-server1: 0.15.1-1
lvm2: 2.03.16-2
lxc-pve: 5.0.2-4
lxcfs: 5.0.3-pve3
novnc-pve: 1.4.0-3
proxmox-backup-client: 3.0.4-1
proxmox-backup-file-restore: 3.0.4-1
proxmox-kernel-helper: 8.0.9
proxmox-mail-forward: 0.2.2
proxmox-mini-journalreader: 1.4.0
proxmox-offline-mirror-helper: 0.6.2
proxmox-widget-toolkit: 4.1.3
pve-cluster: 8.0.5
pve-container: 5.0.8
pve-docs: 8.1.3
pve-edk2-firmware: 4.2023.08-1
pve-firewall: 5.0.3
pve-firmware: 3.9-1
pve-ha-manager: 4.0.3
pve-i18n: 3.1.2
pve-qemu-kvm: 8.1.2-4
pve-xtermjs: 5.3.0-2
qemu-server: 8.0.10
smartmontools: 7.3-pve1
spiceterm: 3.3.0
swtpm: 0.8.0+pve1
vncterm: 1.8.0
zfsutils-linux: 2.2.0-pve3

root@pve:~# uptime
 02:06:49 up  1:11,  3 users,  load average: 4.98, 5.15, 4.63

 

[johnlee]

starting apt-get update
Ign:1 https://mirrors.huaweicloud.com/debian bookworm InRelease
Ign:2 https://mirrors.ustc.edu.cn/proxmox/debian/ceph-quincy bookworm InRelease
Ign:3 https://mirrors.tuna.tsinghua.edu.cn/proxmox/debian bookworm InRelease
Ign:4 https://mirrors.huaweicloud.com/debian-security bookworm-security InRelease
Ign:5 https://mirrors.huaweicloud.com/debian bookworm-updates InRelease
Ign:6 https://mirrors.huaweicloud.com/debian bookworm-backports InRelease
Ign:1 https://mirrors.huaweicloud.com/debian bookworm InRelease
Ign:2 https://mirrors.ustc.edu.cn/proxmox/debian/ceph-quincy bookworm InRelease
Ign:3 https://mirrors.tuna.tsinghua.edu.cn/proxmox/debian bookworm InRelease
Ign:4 https://mirrors.huaweicloud.com/debian-security bookworm-security InRelease
Ign:5 https://mirrors.huaweicloud.com/debian bookworm-updates InRelease
Ign:6 https://mirrors.huaweicloud.com/debian bookworm-backports InRelease
Ign:1 https://mirrors.huaweicloud.com/debian bookworm InRelease
Ign:2 https://mirrors.ustc.edu.cn/proxmox/debian/ceph-quincy bookworm InRelease
Ign:3 https://mirrors.tuna.tsinghua.edu.cn/proxmox/debian bookworm InRelease
Ign:4 https://mirrors.huaweicloud.com/debian-security bookworm-security InRelease
Ign:5 https://mirrors.huaweicloud.com/debian bookworm-updates InRelease
Ign:6 https://mirrors.huaweicloud.com/debian bookworm-backports InRelease
Err:1 https://mirrors.huaweicloud.com/debian bookworm InRelease
Temporary failure resolving 'mirrors.huaweicloud.com'
Err:2 https://mirrors.ustc.edu.cn/proxmox/debian/ceph-quincy bookworm InRelease
Temporary failure resolving 'mirrors.ustc.edu.cn'
Err:3 https://mirrors.tuna.tsinghua.edu.cn/proxmox/debian bookworm InRelease
Temporary failure resolving 'mirrors.tuna.tsinghua.edu.cn'
Err:4 https://mirrors.huaweicloud.com/debian-security bookworm-security InRelease
Temporary failure resolving 'mirrors.huaweicloud.com'
Err:5 https://mirrors.huaweicloud.com/debian bookworm-updates InRelease
Temporary failure resolving 'mirrors.huaweicloud.com'
Err:6 https://mirrors.huaweicloud.com/debian bookworm-backports InRelease
Temporary failure resolving 'mirrors.huaweicloud.com'
Reading package lists...
W: Failed to fetch https://mirrors.huaweicloud.com/debian/dists/bookworm/InRelease Temporary failure resolving 'mirrors.huaweicloud.com'
W: Failed to fetch https://mirrors.huaweicloud.com/debian-security/dists/bookworm-security/InRelease Temporary failure resolving 'mirrors.huaweicloud.com'
W: Failed to fetch https://mirrors.huaweicloud.com/debian/dists/bookworm-updates/InRelease Temporary failure resolving 'mirrors.huaweicloud.com'
W: Failed to fetch https://mirrors.huaweicloud.com/debian/dists/bookworm-backports/InRelease Temporary failure resolving 'mirrors.huaweicloud.com'
W: Failed to fetch https://mirrors.ustc.edu.cn/proxmox/debian/ceph-quincy/dists/bookworm/InRelease Temporary failure resolving 'mirrors.ustc.edu.cn'
W: Failed to fetch https://mirrors.tuna.tsinghua.edu.cn/proxmox/debian/dists/bookworm/InRelease Temporary failure resolving 'mirrors.tuna.tsinghua.edu.cn'
W: Some index files failed to download. They have been ignored, or old ones used instead.
TASK OK
I can't upgrade to 8.2, can you show me how to fix it

 

[zzz09700]

If experience stands, trust anying from Ubuntu non-LTS release or anything from LTS release without the .x postfix is at least reckless, or I would prefer calling it plain stupid.

I still remember that Ubuntu 18.04 breaks on some Ubuntu certified laptops released in the same year, with Ubuntu certificate driver packages and people were forced to wipe the whole disk and install 16.04.x.

Oh wait the same thing happened for 16.04 too, people we're sticking with 14.04.4.

The new Gnome from 20.04 was more or less a disaster for pretty much any non-intel GPUs at launch.

I guess that's a traditional now lol.

 

[adamb]

That's a valid concern, but the odd thing about all this is that the Linux 6.8 kernel is the backbone of Ubuntu's current LTS release. It's not really supposed to be bleeding edge.

The 6.5.x kernel is now no longer officially supported by Ubuntu, so staying on it would have its own problems long term given that Proxmox's kernel development is tied to Ubuntu's kernel development cycle.

I'm not really sure what to make of this, but given reports of issues with basic functionality in Ubuntu 24.04 for some users, it seems like it's not just a Proxmox thing.

It sounds like kernel 6.8, 6.9, and probably 6.10 and 6.11 are all pushing or about to be pushing some major changes to how Linux works deep under the hood as far as how it talks to CPUs and GPUs and other low-level hardware, which is all to the good, but the reality is we might just be going through a rocky phase of hardware compatibility.

Sounds like the proxmox dev's need to make some decisions then. We need stability.

The last good kernel was 5.15 and at this point 6.5 has been the next solid kernel.

We are a real production enviroment (700 VM's running over 7 front ends), from KSM issues to stability, to just basic live migration the last few kernel releases have been terrible.

Proxmox Dev's here is a user from over a decade at this point asking you to do better. This is starting to get old.

Proxmox4, Proxmox5, Proxmox6 were all considerably more stable than 7 or 8 at this point. Its extremely disappointing and has me questioning if we will stick with proxmox for KVM.

 

[mrpops2ko]

Sounds like the proxmox dev's need to make some decisions then. We need stability.

The last good kernel was 5.15 and at this point 6.5 has been the next solid kernel.

We are a real production enviroment (700 VM's running over 7 front ends), from KSM issues to stability, to just basic live migration the last few kernel releases have been terrible.

Proxmox Dev's here is a user from over a decade at this point asking you to do better. This is starting to get old.

Proxmox4, Proxmox5, Proxmox6 were all considerably more stable than 7 or 8 at this point. Its extremely disappointing and has me questioning if we will stick with proxmox for KVM.

i think they need much better logging, i don't inherently mind that certain releases might be unstable - esxi had the same issue with some releases and it was quite simple to just not upgrade for a while

what isn't acceptable is that they don't have a robust logging process, for example with any crash on esxi no matter what it was, i'd be able to spit out and submit a report that was many pages long, detailing in exact and precise details where things stopped and why

with proxmox all i'm getting for the same event is --reboot-- and that data isn't verbose or meaningful in the slightest, which causes proxmox devs to play mystic meg and try predict why my crash happened.

after having crashes every 2-3 days i reverted back down to this and it might have solved it

 

[adamb]

i think they need much better logging, i don't inherently mind that certain releases might be unstable - esxi had the same issue with some releases and it was quite simple to just not upgrade for a while

what isn't acceptable is that they don't have a robust logging process, for example with any crash on esxi no matter what it was, i'd be able to spit out and submit a report that was many pages long, detailing in exact and precise details where things stopped and why

with proxmox all i'm getting for the same event is --reboot-- and that data isn't verbose or meaningful in the slightest, which causes proxmox devs to play mystic meg and try predict why my crash happened.

after having crashes every 2-3 days i reverted back down to this and it might have solved it

I can't debate that, at this point we are talking very basic topics here.

With kernel 5.15+ we started having KSM issues (KSM basically pointless), we stopped being able to live migrate between hosts with different generations etc. Proxmox took their stance on all of this and I do feel they could do ALOT better. Its simply not the same promxox I am used to.

I will be honest and I feel the dev's are bending over for bleeding edge topics like GPU passthrough etc instead of worrying about customers who are actually paying for the enterprise repo's (Like us!).

We are a larger client of proxmox and I do think its time for them to start making some real decisions. Stability has been set to the side for bleeding edge.

How much further are the dev's willing to go at this point. You either want a enterprise solution or you want bleeding edge, based on the issues, this is bleeding edge.

 

[zzz09700]

The 6.5.x kernel is now no longer officially supported by Ubuntu

Ubuntu 22.04 LTS is still using 6.5.x kernel as the "HWE kernel", so 6.5.x SURE IS supported by Ubuntu.

 

[mrpops2ko]

I can't debate that, at this point we are talking very basic topics here.

With kernel 5.15+ we started having KSM issues (KSM basically pointless), we stopped being able to live migrate between hosts with different generations etc. Proxmox took their stance on all of this and I do feel they could do ALOT better. Its simply not the same promxox I am used to.

I will be honest and I feel the dev's are bending over for bleeding edge topics like GPU passthrough etc instead of worrying about customers who are actually paying for the enterprise repo's (Like us!).

We are a larger client of proxmox and I do think its time for them to start making some real decisions. Stability has been set to the side for bleeding edge.

How much further are the dev's willing to go at this point. You either want a enterprise solution or you want bleeding edge, based on the issues, this is bleeding edge.

all excellent points, and i'd suggest you need both. you need a team that drives stabillity and you also need bleeding edge to drive progression forward.

you need to support both ends, businesses who need stability need stable releases that they dont update for say a major release, similar to how linux does LTS

 

[h00bi]

Having Issues after update, too. -> see EDIT below: solved(?)
I use a 10Gb/s X710 NIC single 10Gb Link only.
i tried renaming in etc/network/interfaces but i think the main problem is, that all my links are down.
I can't reach the webinterface and i can't ping anything, access only via local console or idrac console

As far as i understood comments in this thread, link down is only known to broadcom NICs.
Any suggestions?





EDIT:
i changed the following via nano /etc/network/interfaces

eno1 -> eno1np0 for the interface itself and for vmbr0


All hosts seems to run fine with these changes, after another reboot.
Any further suggestions?

 

[thesithlord]

Tried to give Proxmox a chance... Installed Prox Mox latest release and the installation freezes (showing always the big middle stinky finger), whatever I try it freezes on my HP Z 820 Workstation with 64GB RAM, nVidia Card und 2 Xeon CPUs. ESXi installation no problem, Citrix XenServer Installation no problem, MS Hyper-V installation no problem, installing latest Ubuntu LTS server with GNome no problem, installing Debian 12 with GNome no problem.

Freezes after loading driver intel_rapl_common...

Maybe proxmox will fix the problem that it will be maybe installable like the other mentioned systems I installed successfully?

 

[mrpops2ko]

i think they need much better logging, i don't inherently mind that certain releases might be unstable - esxi had the same issue with some releases and it was quite simple to just not upgrade for a while

what isn't acceptable is that they don't have a robust logging process, for example with any crash on esxi no matter what it was, i'd be able to spit out and submit a report that was many pages long, detailing in exact and precise details where things stopped and why

with proxmox all i'm getting for the same event is --reboot-- and that data isn't verbose or meaningful in the slightest, which causes proxmox devs to play mystic meg and try predict why my crash happened.

after having crashes every 2-3 days i reverted back down to this and it might have solved it

further to this, i am now probably in a position to say that downgrading resolved the issue. this level of uptime previously was not possible when running the latest version. the only bad thing though is that i can't ever upgrade now... or rather i need to stalk the forums and find written accounts of people who when they upgrade also have the same cpu and it no longer affecting them :/

 

[dsi]

I'm using Proxmox for years. The upgrade went without any issues (as usual - thanks Proxmox team!) But the latest 6.8 kernel upgrade results in unpredictable crashes (several days) on my HPE Microserver Gen8 (back to 6.5 now and stable) - nothing suspicious in the logs, just bang and reboot. The Supermicro X11SBA runs without any problems.

 

[alextsang]

Is proxmox 8.2.2 is stable version for upgrade? or upgrade to 8.2.2 with using kernel 6.5.13-7-pve is best option in this moment?

 

[golruck]

I am personally have an issue with the kernel so I am running 8.2.2 with 6.5 kernel. I have just tried to set iommu=off as advised in the documentation but I still cannot boot on the new kernel:

I edited the nano /etc/default/grub, set it and ran proxmox-boot-tool refresh



I opened another post there will all details but I could not make it to work:
https://forum.proxmox.com/threads/proxmox-upgrade-8-2-made-my-system-not-booting.147327

I assume I will remain on the 6.5 kernel until there is a fix for this. The only thing is that I am not sure how to know if this has been acknowledged as an issue and where I could follow its resolution.

Would be grateful if anyone has more info

 

[razit]

Hi there,

Server Experiencing Random Freezes After Proxmox Update to 8.2.2

System Specifications:

• CPU: Intel Core i9-9900K
• Motherboard: Gigabyte B360 HD3PLM
• Kernel: Linux 6.8.4-2-pve (compiled on 2024-04-10T17:36Z)
• Hosting: Hetzner Dedicated Server
• Virtualization: Licensed Proxmox with 12 VMs (approximately 3 running)

Issue Description:
Following the recent update of Proxmox to version 8.2.2, the dedicated server has experienced two instances of complete system freezes. Prior to this update, no such freezes were observed. The freezes occurred while the server was idle and under minimal load, with most VMs powered off. The incidents happened during overnight hours when the administrator was asleep. Seems like a broader issue based on the postings here?

lspci output:

lspci
00:00.0 Host bridge: Intel Corporation 8th/9th Gen Core 8-core Desktop Processor Host Bridge/DRAM Registers [Coffee Lake S] (rev 0d)
00:01.0 PCI bridge: Intel Corporation 6th-10th Gen Core Processor PCIe Controller (x16) (rev 0d)
00:02.0 VGA compatible controller: Intel Corporation CoffeeLake-S GT2 [UHD Graphics 630] (rev 02)
00:12.0 Signal processing controller: Intel Corporation Cannon Lake PCH Thermal Controller (rev 10)
00:14.0 USB controller: Intel Corporation Cannon Lake PCH USB 3.1 xHCI Host Controller (rev 10)
00:14.2 RAM memory: Intel Corporation Cannon Lake PCH Shared SRAM (rev 10)
00:16.0 Communication controller: Intel Corporation Cannon Lake PCH HECI Controller (rev 10)
00:17.0 SATA controller: Intel Corporation Cannon Lake PCH SATA AHCI Controller (rev 10)
00:1b.0 PCI bridge: Intel Corporation Cannon Lake PCH PCI Express Root Port #21 (rev f0)
00:1d.0 PCI bridge: Intel Corporation Cannon Lake PCH PCI Express Root Port #9 (rev f0)
00:1f.0 ISA bridge: Intel Corporation Device a308 (rev 10)
00:1f.4 SMBus: Intel Corporation Cannon Lake PCH SMBus Controller (rev 10)
00:1f.5 Serial bus controller: Intel Corporation Cannon Lake PCH SPI Controller (rev 10)
00:1f.6 Ethernet controller: Intel Corporation Ethernet Connection (7) I219-LM (rev 10)
01:00.0 Non-Volatile memory controller: Micron Technology Inc 3400 NVMe SSD [Hendrix]
02:00.0 Non-Volatile memory controller: Micron Technology Inc 3400 NVMe SSD [Hendrix]

Spoiler: CPU/RAM usage until crash

View attachment 67108

I have the same issue with:
CPU: Intel(R) Core(TM) i9-9900K CPU @ 3.60GHz
MEMORY: 64GB
Motherboard: Gigabyte B365M D3H

Server after upgrade randomly freeze

//update 10 June 2024//
I tested:
1. pve-efiboot-tool kernel pin 6.5.13-5-pve -> not solve issue
2. Upgrade BIOS - > not solve the issue


//update 13 June 2024//
intel_iommu=off solved

How to:

edit: /etc/default/grub
set: GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=off"

MAIN KEY: intel_iommu=off

Another server (also upgraded) works well, but another specs:
CPU: Intel(R) Core(TM) i5-8400 CPU @ 2.80GHz
MEMORY: 64GB
Motherboard: Gigabyte B360M D3H-CF

 

[antreos]

Hi there,

Server Experiencing Random Freezes After Proxmox Update to 8.2.2

System Specifications:

• CPU: Intel Core i9-9900K
• Motherboard: Gigabyte B360 HD3PLM
• Kernel: Linux 6.8.4-2-pve (compiled on 2024-04-10T17:36Z)
• Hosting: Hetzner Dedicated Server
• Virtualization: Licensed Proxmox with 12 VMs (approximately 3 running)

Issue Description:
Following the recent update of Proxmox to version 8.2.2, the dedicated server has experienced two instances of complete system freezes. Prior to this update, no such freezes were observed. The freezes occurred while the server was idle and under minimal load, with most VMs powered off. The incidents happened during overnight hours when the administrator was asleep. Seems like a broader issue based on the postings here?

lspci output:

lspci
00:00.0 Host bridge: Intel Corporation 8th/9th Gen Core 8-core Desktop Processor Host Bridge/DRAM Registers [Coffee Lake S] (rev 0d)
00:01.0 PCI bridge: Intel Corporation 6th-10th Gen Core Processor PCIe Controller (x16) (rev 0d)
00:02.0 VGA compatible controller: Intel Corporation CoffeeLake-S GT2 [UHD Graphics 630] (rev 02)
00:12.0 Signal processing controller: Intel Corporation Cannon Lake PCH Thermal Controller (rev 10)
00:14.0 USB controller: Intel Corporation Cannon Lake PCH USB 3.1 xHCI Host Controller (rev 10)
00:14.2 RAM memory: Intel Corporation Cannon Lake PCH Shared SRAM (rev 10)
00:16.0 Communication controller: Intel Corporation Cannon Lake PCH HECI Controller (rev 10)
00:17.0 SATA controller: Intel Corporation Cannon Lake PCH SATA AHCI Controller (rev 10)
00:1b.0 PCI bridge: Intel Corporation Cannon Lake PCH PCI Express Root Port #21 (rev f0)
00:1d.0 PCI bridge: Intel Corporation Cannon Lake PCH PCI Express Root Port #9 (rev f0)
00:1f.0 ISA bridge: Intel Corporation Device a308 (rev 10)
00:1f.4 SMBus: Intel Corporation Cannon Lake PCH SMBus Controller (rev 10)
00:1f.5 Serial bus controller: Intel Corporation Cannon Lake PCH SPI Controller (rev 10)
00:1f.6 Ethernet controller: Intel Corporation Ethernet Connection (7) I219-LM (rev 10)
01:00.0 Non-Volatile memory controller: Micron Technology Inc 3400 NVMe SSD [Hendrix]
02:00.0 Non-Volatile memory controller: Micron Technology Inc 3400 NVMe SSD [Hendrix]

Spoiler: CPU/RAM usage until crash

View attachment 67108

As written here:

• #13
  
  6.8.4-3-pve didn't fix the issue

 

[peteb]

I had to shut my proxmox server down due to instability with random crashes after the recent updates to the kernel and the base software. My server runs everything from files to web pages, to the vyos router. I tried using the 6.5 kernel and it worked for a few days and then crashed again. The instability is terrible and proxmox should come up with a way to roll back the base software as well (you can only roll back the kernel).
I'm now back on my ISP's router with proxmox shut down and will have to wait this out.