Opt-in Linux 7.0 Kernel for Proxmox VE 9 available

Hi all,

I would like to add a separate Intel X550 / ixgbe crash report on 7.0.14-20-pve. This differs from the recent Broadcom / bnxt_en reports in this thread.

We captured the full crash through pstore, which identifies the failing function as ixgbe_check_mdd_event().

Environment

  • Proxmox VE manager: 9.2.21
  • Kernel at the time of the crash: 7.0.14-20-pve
  • Mainboard: Supermicro H11DSi-NT
  • BIOS: 2.1, dated 2020-02-21
  • NIC: Intel X550 [8086:1563], revision 01
  • Subsystem: Supermicro [15d9:1563]
  • Driver: in-tree ixgbe
  • NIC firmware reported by ethtool: 0x800007f3, 1.1276.0
  • Affected interface: eno1, PCI address 0000:61:00.0
  • Interface used as a bridge uplink for VM traffic, with VLAN subinterfaces
  • Link: 10 Gbit/s
Messages immediately preceding the panic

ixgbe 0000:61:00.0: Malicious event on VF 8 tx:80000 rx:0
ixgbe 0000:61:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0004 address=0xd5b294c0 flags=0x0000]
BUG: kernel NULL pointer dereference, address: 000000000000078c
#PF: supervisor write access in kernel mode
Relevant pstore excerpt

Oops: Oops: 0002 [#1] SMP NOPTI
CPU: 5 UID: 0 PID: 0 Comm: swapper/5 Tainted: P O 7.0.14-20-pve #1 PREEMPT(lazy)
Hardware name: Supermicro Super Server/H11DSi-NT, BIOS 2.1 02/21/2020
RIP: 0010:ixgbe_check_mdd_event+0x124/0x160 [ixgbe]
RAX: 0000000000000740 RBX: 0000000000000008 RCX: 0000000000000008
CR2: 000000000000078c
The IRQ call chain includes:

ixgbe_check_mdd_event
ixgbe_msg_task
ixgbe_msix_other
__handle_irq_event_percpu
handle_irq_event
handle_edge_irq
__common_interrupt
common_interrupt
Followed by:

Kernel panic - not syncing: Fatal exception in interrupt
The host became completely unresponsive and required a reboot.

No SR-IOV VFs configured

The following was checked after reboot:

cat /sys/class/net/eno1/device/sriov_numvfs
0
The private flags are:

legacy-rx : off
vf-ipsec : off
mdd-disable-vf: off
This appears consistent with the reported issue where an MDD event causes access to adapter->vfinfo[] despite no VFs being configured. The stacktrace confirms the crash location; we have not established what triggered the MDD event or the accompanying IOMMU fault.

Potentially related reports:

Current mitigation status

We have installed 6.17.13-21-pve in preparation for a comparison test. We do not yet have a stability result and cannot claim that it fixes this issue.

The X550 interfaces are required for our infrastructure, so bypassing these NICs is not a practical workaround.

Could the Proxmox team confirm:

  1. Whether 6.17.13-21-pve contains the same unguarded MDD/VF access path.
  2. Whether a fix or backport is planned for the 7.0 kernel series, and which build should contain it.
  3. Whether there is a validated workaround for systems with sriov_numvfs=0 while retaining the X550 interfaces.
We have the full pstore dump available if additional context would help.
 
uzumo said:
Some people say that SRIOV has been deprecated in the i915 driver.
...
Xe module currently does not support MTL (Meteor Lake) and LNL (Lunar Lake) platforms. Please use i915 instead.
Click to expand...
I’m working on an experimental Xe SR-IOV patch set targeting Meteor Lake and Arrow Lake: xe-sriov-mtl-arl.

On my Linux 7.2.7 test host, I have Windows guests running with Intel VFs, and I’ve resolved the rendering glitches affecting text, transparency and hover effects in the Windows UI.

I’ve also received independent feedback from dmfrpro, who tested my patches in a downstream fork on Meteor Lake (185H). In this GitHub comment, they reported:

Video games work
Firefox youtube videos work fine
WinUI3 doesn't have artifatcs
Click to expand...
This is encouraging, but the project remains highly experimental. I’m still investigating stability and reviewing the changes.

Feedback and further test reports are welcome!
 
theleatherman said:
Could the Proxmox team confirm:

  1. Whether 6.17.13-21-pve contains the same unguarded MDD/VF access path.
  2. Whether a fix or backport is planned for the 7.0 kernel series, and which build should contain it.
  3. Whether there is a validated workaround for systems with sriov_numvfs=0 while retaining the X550 interfaces.
We have the full pstore dump available if additional context would help.
Click to expand...

Not Proxmox staff, just another X550 user on 7.0.14-23-pve who dug into this.

1) Yes, 6.17.13-21-pve has the same code. The MDD handling arrived upstream
in v6.17 and ixgbe_check_mdd_event() is unchanged through 7.0 and mainline;
the compiled function in ixgbe.ko is the same size in 6.17.13-21-pve and
7.0.14-23-pve on my host. Your 6.17 test is unlikely to help. Your CR2 fits
exactly: VF 8 * 0xe8 (entry size) + 0x4c (clear_to_send) = 0x78c.

2) A fix is posted upstream but not merged. Thomas Lamprecht (Proxmox) sent
two patches on 2026-10-04: one guards the vfinfo access, one stops enabling
MDD when there are no VFs.
https://lore.kernel.org/netdev/20261004193704.2895700-1-t.lamprecht@proxmox.com/
In bug 7874 he says he wants initial upstream feedback before putting it in
Proxmox kernels, so there is no build to name yet. 7.0.14-23-pve does not
have it. Watch: https://bugzilla.proxmox.com/show_bug.cgi?id=7874

3) The only workaround I'm confident in is a pre-6.17 kernel. 6.14 is still
in the PVE 9 repo and has no MDD code:

apt install proxmox-kernel-6.14
proxmox-boot-tool kernel pin 6.14.11-9-pve
reboot

Afterwards "grep ixgbe_check_mdd_event /proc/kallsyms" should print nothing.
Undo with "proxmox-boot-tool kernel unpin". Caveat: the last 6.14 build is
from May 2026, and I don't know if Proxmox still officially supports it.

Not workarounds: mdd-disable-vf (different mechanism), and creating a VF or
two to make vfinfo non-NULL (an event on a PF queue then writes past the end
of the array instead).

The patch author tested on an H11DSi-NT like yours but has no reproducer for
the MDD events and is asking reporters to test. If your host crashes
regularly, attaching your pstore dump to bug 7874, or testing the patches
and reporting the result there, would help get this merged.

I haven't experienced the same panic that you are having -- I am using SR-IOV with 6 VF's, which may or may not be saving me from the crashes.

I can't explain the AMD-Vi IO_PAGE_FAULT; nothing I found covers it.
 
  • Like
Reactions: Johannes S
Hi,

I have encountered an SMB access regression with ksmbd after upgrading the Proxmox VE kernel.

Kernel versions tested​

  • 7.0.14-23-pve — works
  • 7.0.14-24-pve — works
  • 7.0.14-25-pve — SMB access from Windows fails
  • 7.0.14-26-pve — SMB access from Windows fails
The ksmbd configuration has not changed between these kernel versions.

Environment​

  • Proxmox VE 9
  • SMB server: ksmbd (kernel module and userspace daemon)
  • Network interface: vmbr0
  • SMB protocol range: SMB 2.1–SMB 3.1.1
Windows clients:

  • Windows 11 Pro for Workstations, version 26H2, OS build 26300.9807
  • Windows 10 Pro for Workstations, version 22H2, OS build 19045.7058

Symptoms​

On kernels 7.0.14-25-pve and 7.0.14-26-pve, both Windows clients exhibit the same behavior:

  • Test-NetConnection 192.168.1.2 -Port 445 succeeds.
  • net view \\192.168.1.2
  • returns System error 53 has occurred. The network path was not found.
  • SMB shares are inaccessible from Windows.
  • Linux clients can still access the shares.
The ksmbd service starts successfully, the kernel module loads, and TCP port 445 is listening on vmbr0.

Kernel logs show incoming SMB connections and read requests, but some connections are closed shortly after being accepted. Other log entries show successful SMB read requests.

Windows SMB client configuration:

Get-SmbClientConfiguration |
Select-Object EnableSecuritySignature, RequireSecuritySignature,
EnableInsecureGuestLogons
All three values are False.

Relevant​

[global]
bind interfaces only = yes
interfaces = vmbr0
tcp port = 445

server min protocol = SMB2_10
server max protocol = SMB3_11
server multi channel support = yes
server signing = disabled
smb3 encryption = no

smb2 leases = no
smb2 max credits = 16384
smb2 max read = 8MB
smb2 max trans = 4MB
smb2 max write = 8MB
smbd max io size = 16MB

map to guest = never
restrict anonymous = 1
guest account = nobody
workgroup = WORKGROUP

max active sessions = 128
max open files = 10000
max connections = 1024

Workaround​

Booting kernel 7.0.14-23-pve or 7.0.14-24-pve restores SMB access.

I have pinned the working kernel:

proxmox-boot-tool kernel pin 7.0.14-24-pve
The issue is reproducible on two different Windows versions with the same server configuration. SMB access works on kernels 7.0.14-23-pve and 7.0.14-24-pve, but fails on 7.0.14-25-pve and 7.0.14-26-pve.
 
I have seen massive stability issues on one of my three Proxmox machines after changing from kernel "6.17.13-21-pve" to a v7 kernel. The machine frequently "hung", no message, no log entry. Once only I have at least seen a "kernel panic" notice on the screen, the other times not even that. The freezes seemed to occur completely at random, I had uptimes between 2,5 up to 14 days, but no more.

The failing kernels I tried back then were "7.0.14-5-pve" (possibly earlier, but this was my earliest bootlog) up to "7.0.14-12-pve". Then I stopped trying and pinned kernel "6.17.13-21-pve" which has been stable ever since.

The machine in question is a "Minisforum HM80", AMD Ryzen 7 4800U.

Now that the v6 kernels have been EOL'd the problem of course comes up again.

What would you suggest to do? There is no newer BIOS than is already installed, and I have made no other modification to the system between kernel v6 and v7. Proxmox itself has been upgraded over time, but kernel v6 is still pinned. I realize I can't stay on kernel v6 forever, but I also can't power cycle my system every two days. The tested kernel v7 versions are a while back and have been superseded since, but do you think I can expect to fare better with the current v7?
 
150d said:
The machine in question is a "Minisforum HM80", AMD Ryzen 7 4800U.

What would you suggest to do?
Click to expand...
Well, each and every cheap hardware constellation (Ram, SSD, NVMe ...) may behave differently, so take this with a grain of salt:

I have three of those HM80 in my homelab for ~four years now. I can remember that one of them has had hard freezes some years ago with unclear reasons; I remember I'd replaced some hardware then. (Especially the tiny SSD-connector-cables don't look like "very reliable".) For the VM storage I have two (refurbished) Intel SSDs in them.

All three of them are running continuously 24*7 for years now. Always with the current pve-no-sub kernel.

Note that this is just a random data-point and I do not say "everything will run on your devices".
 
  • Like
Reactions: Johannes S
UdoB said:
I have three of those HM80 in my homelab for ~four years now. I can remember that one of them has had hard freezes some years ago with unclear reasons [...]
Click to expand...
Thank you for your note. It's very interesting to hear that there isn't actually a "HM80 problem", since it runs well for you. Instead, your description gave me the idea that I may have been looking at it from the wrong side: When researching freezes like mine, the top candidates for workaround always are the Ryzen graphics driver and IOMMU. But if those were the causes for my v7-issues, then why is it running fine for you, even with the v7 kernels?

That leaves hardware: I have extended RAM to 64 G, which I have not yet run tests on. I have a SATA-SSD as a data drive that has been known to drop to a crawl when overloaded. Nothing crashed before, but maybe the v7 kernel is less forgiving than the v6 was.

That at least gives me a place to start.

PS: Which of the LAN ports do you use? The 2.5G Intel one, or the 1G Realtek one (or both?) That could also be a critical point.
 
Last edited:
150d said:
Which of the LAN ports do you use? The 2.5G Intel one, or the 1G Realtek one (or both?)
Click to expand...
In my case the 2.5 GBit/s port is exclusively used for storage (ZFS replication + NFS + PBS + experimental iSCSI). This one is "untagged" and connected to a separate, dumb 2.5 GBit/s switch.

The other NIC is "tagged only" and transports multiple networks. My chosen strategy is to have a dedicated vmbrX each, configured with the corresponding tag. (Instead of having one single "VLAN aware" bridge.)

(( Sidenote, off-topic: of course both of them are used to implement one Corosync ring each. And I can see them getting saturated during different use cases - e.g. backups respectively user-side downloads on the 1 GBit/s nic. Only once in those years I managed to saturate both lines at the same time, resulting in a partial cluster reboot - as expected. That experience is the reason why I always recommend to have an independent (third) nic exclusively for Corosync... ))

:-)
 
  • Like
Reactions: Johannes S and gfngfn256
poulpor said:
I'm experiencing exactly the same issue.
With 32 GB of RAM, I have to leave 8 to 10% free, or the Proxmox host freezes and the fans go wild. I first noticed this after rebooting following the update to kernel 7.0: the fans went wild, and I couldn’t connect to the Proxmox host. I reverted to 6.17, disabled a VM from auto-starting, rebooted to 7.0 and it booted fine. I then manually started the VM after the Proxmox host had fully booted, and it froze with the fans running wild. After tweaking other settings, I ended up reducing the RAM allocated to this VM, and it worked.

This has been happening since the first release, I'm now on kernel 7.0.14-11.
I just updated to kernel 7.0.14-14 and will check if the behavior persists.

Is it mandatory with kernel 7.0 to leave more RAM free? With the previous 6.17.13-21, I could allocate more RAM to my VMs without issues.
Click to expand...

I had the same issue. With about 192 GB RAM, I had to keep 5-10% free. That totaled to 10-20 GB of RAM, sitting around unused.

In the end, I found the culprit: kexec hand over (KHO). This is a relativley new feature. There were discussions about it on the proxmox kernel mailing list.

The solution: Turn it off, since proxmox does not use it (at the moment). Add this to your kernel paramaters:
Code: 
kho=off
 
  • Like
Reactions: santiagobiali and poulpor
systemofapwne said:
I had the same issue. With about 192 GB RAM, I had to keep 5-10% free. That totaled to 10-20 GB of RAM, sitting around unused.

In the end, I found the culprit: kexec hand over (KHO). This is a relativley new feature. There were discussions about it on the proxmox kernel mailing list.

The solution: Turn it off, since proxmox does not use it (at the moment). Add this to your kernel paramaters:
Code: 
kho=off
Click to expand...
Wow, I hadn't done the math on how much RAM that would use with 192Gb, but that's definitely concerning!
Thank you very much for sharing this information. I'm turning it off right now.
 
"Oh nooooo, has AI gotten into PVE codebase and done a numpty?" lol

Issue appeared after upgrading from 7.0.14-23-pve to 7.0.14-26-pve

Have never had this issue in all the upgrades prior. Something has broken in the underlying networking/bridge stack is what my hallucinating virtual buddy Google Gemini keeps suggesting, normally I write-off AI-slop, however in this case I'm in agreement.

This could be quite a breaking issue for countless production environments who utilize Kerio Control and possibly other similar firewall virtual appliances.

Prior to 7.0.14-26-pve, for example (7.0.14-23-pve), Kerio Control was able to rollup IPv4 and IPv6 addresses into a single dual-stack auth entry like this:

Fig1:
before.png

This is the 'Active Hosts' view in Kerio Control's webui. It basically shows the host name, user, traffic up and traffic down, mac, ipv4 and ipv6. When a user authenticates via an ipv4 web portal, Kerio authorizes both ipv4 and ipv6 protocols for that user. This is how it works in a functioning system.


After upgrading to 7.0.14-26-pve, now it seems to mess things up a little:

Fig2:
after.png

As you can see the 2 rows for the same machine, same mac, however it only authorized ipv4 stack. Normally it rolls them up together like (Fig1). Now it kind of behaves erraticaly for some virtio-net nics, and for some it totally no longer rolls up the authorization effectively leaving the ipv6 unauthorized perpetually unless you physically log into the ipv6 fqdn web-portal and then it also enables that one seperately.

The underlying setup is basically quite standard. Single ethernet nic, some vlan bridges, and an untagged traffic bridge. The one that is now perpetually stuck with no rollups/dual-stack auth is the untagged vlan bridge.

Not sure what has changed between .23 and .26, but I would advise a rollback of underlying networking changes as I think its affecting quite a few other use-cases too?
 
It's not really clear what your actual issue is, where Kerio Control runs and how PVE kernel changes interact with it and if that third party software had any changes that just got then active with a reboot.

Also, it's not fully clear if you actually tried rebooting into an older kernel fixing the effects you see or if it just worked on -23?

Some network setup and hardware details would be also good to have. New kernel version bring lots of fixes and sometimes naturally also regressions, but the range mentioned here hasn't exactly any wide spread network issues reported and is already widely in use since enough days to rule out some more general (network) regression.
 
  • Like
Reactions: gfngfn256
Fair point. I reverted to a backup rather than kernel pinning. As I do kernel coding myself I normally would have gone in and checked what is going on, the problem is the fact that like numerous other vendors of commercial virtual appliances, GFI Kerio Control is closed source, so there isnt much I have to go on, even the logs that it does provide show no errors/warnings. However functionality has altered moving from .23 proxmost host to .26 proxmox host, and indeed reverting to a backup prior to the upgrade fixed the issue.

The platform is a xeon E- 2300 series fujitsu rack server, using a RealTek 8127 10gbe nic.

One thing I did try was just globally turn off hardware offloading everywhere, as that normally tends to be a baseline debugging foundation for virtio-net, however that made no difference, so my presumption along with Gemini's internet searching and consolidation prowess concluded something has changed in the networking stack, causing possibly a timing issue or a packet/sequence order issue or whatever, however the lack of being able to debug Kerio Control as we do not have access to the source code we're kind of blind on that side so we can only look at this side and wonder what changed, Gemini reckons a fair amount has changed upstream but not quite sure how much of that is pulled into .26 since .23

I have run tcpdumps, and have not noticed anything out of the ordinary, however I have no idea of what Kerio Control does under the hood as it uses a closed source kernel module which handles all the network comms, a black box.
 
You didn't answer a single question posed by t.lamprecht!
Where is kerio control running? In a VM on over? In a separate box? On the PvE node itself?
What is your network topology? Post your /etc/network/interfaces please.
Did booting the old kernel fix the issue or only the rollback of everything including the kernel, kerio control, ...?
Have you opened a support ticket?

Always so quick with the accusations...
 
drwr24 said:
Windows clients:

  • Windows 11 Pro for Workstations, version 26H2, OS build 26300.9807
  • Windows 10 Pro for Workstations, version 22H2, OS build 19045.7058

​

The issue is reproducible on two different Windows versions with the same server configuration. SMB access works on kernels 7.0.14-23-pve and 7.0.14-24-pve, but fails on 7.0.14-25-pve and 7.0.14-26-pve.
Click to expand...
ksmbd SMB access from Windows is now working again with kernel 7.0.14-28-pve. Both Windows 10 and Windows 11 clients can access the shares successfully.
 
Last edited:
  • Like
Reactions: fabian
You must log in or register to reply here.
Top Bottom
Back