After proxmox updates: Random segmentation faults in AlmaLinux 8 VMs on Proxmox VE 9.1

[PVasileff]

Hello,

I am investigating an issue with AlmaLinux 8 virtual machines running on Proxmox VE. The symptom is not a full host freeze, but random segmentation faults inside the guest VMs. The segfaults happen in different applications and services, not only in one
specific software stack.

Initially I suspected a hardware issue on my main Proxmox server, but the problem is now observed on several different Proxmox hosts with different Intel hardware. One affected mail VM was migrated from a large Supermicro server to an Intel NUC, and aft
er some time the issue started appearing there as well.

Important: so far the issue is only observed with AlmaLinux 8 guests. The affected AlmaLinux 8 systems are updated to the latest available packages, but the issue is also observed on VMs restored from a backup dated 2026-05-10.

Main symptom:
- Random segmentation faults inside AlmaLinux 8 VMs.
- Different services/applications are affected, including:
- httpd
- mariadbd / mysqld
- different php-fpm versions
- crond
- imap
- clamd
- amavis
- systemd-journald
- python2
- coredumpctl mostly shows Signal: 11 (SEGV), sometimes Signal: 6 (ABRT).
- In some examples coredump storage is none, so a backtrace is not always available.
- Kernel log pattern includes many crashes like:
- segfault at 0 ip 0000000000000000 ... error 14
- There are also crashes inside core/shared libraries:
- ld-2.28.so
- libc-2.28.so
- libpthread-2.28.so
- libclamav.so.12.0.3
- Encode.so
- libpython2.7.so.1.0

Checks and tests performed so far:
- Migrated an affected AlmaLinux 8 mail VM from the large Supermicro host to the Intel NUC host.
- The issue reproduced on the NUC after some time.
- Tested CPU type host - issue still reproduced.
- Tested CPU type x86-64-v2 - issue still reproduced.
- Higher x86-64 CPU levels are unsupported on part of the hardware.
- Tested guest kernel boot parameter mitigations=off via GRUB - did not fix the issue.
- Tested guest kernel boot parameter transparent_hugepage=never via GRUB - did not fix the issue.
- Tested stopping ksmtuned/KSM service on the Proxmox host - did not fix the issue.
- Tested disabling ballooning in the VM settings - did not fix the issue.
- Tested disabling KSM for the VM via allow-ksm: 0 - did not fix the issue.
- Checked /etc/ld.so.preload - no output.
- Environment only shows MODULES_RUN_QUARANTINE=LD_LIBRARY_PATH LD_PRELOAD.
- ldconfig -p looks normal for /lib64 mappings such as libc, libpthread, libstdc++, openssl, pcre2, jemalloc, etc.
- Tried AlmaLinux guest kernel rollback - did not fix the issue.
- Tried glib2 rollback - did not fix the issue.
- AlmaLinux guest kernel versions seen/tested include:
- 4.18.0-553.124.4.el8_10
- 4.18.0-553.124.1.el8_10
- 4.18.0-553.123.2.el8_10
- cPanel does not look like the common root cause, because the issue is also seen on AlmaLinux 8 + ISPConfig VMs.
- KernelCare also does not look like the common root cause, because only the cPanel machine has KernelCare/kcarectl while other affected VMs do not.

Common Proxmox software pattern:
- Proxmox VE: 9.1.0
- Running kernel on the compared hosts: 7.0.2-4-pve
- pve-qemu-kvm: 11.0.0-2
- qemu-server: 9.1.10 or 9.1.12
- pve-manager: 9.1.11 or 9.1.14
- intel-microcode: 3.20251111.1~deb13u1
- pve-firmware: 3.18-3
- pve-edk2-firmware: not correctly installed
- Installed kernel packages include:
- proxmox-kernel-7.0.2-4-pve-signed
- proxmox-kernel-7.0.2-3-pve-signed
- proxmox-kernel-6.17.13-9-pve-signed
- on some hosts also proxmox-kernel-6.14.11-9-pve-signed

The issue has been observed with Proxmox kernel versions:
- 7.0.2-4-pve
- 7.0.2-3-pve
- 6.17.13-9-pve

Common VM patterns:
- Guest OS: AlmaLinux 8
- ostype: l26
- machine: q35
- virtio network
- qcow2 disks
- numa: 0
- often 2 sockets x 2 cores, but there is also an affected VM with 2 sockets x 1 core
- SCSI controller is not identical everywhere:
- virtio-scsi-pci on one affected mail VM
- virtio-scsi-single + iothread=1 on another mail VM
- virtio-scsi-single on an ISPConfig VM

Host 1 - main server:
- Motherboard: Supermicro X8DTL
- CPU: 2x Intel Xeon X5675 @ 3.07GHz
- Logical CPUs: 24
- Threads per core: 2
- Cores per socket: 6
- Sockets: 2
- Virtualization: VT-x
- RAM: 96 GB
- Running kernel: 7.0.2-4-pve
- pve-manager: 9.1.11
- pve-qemu-kvm: 11.0.0-2
- qemu-server: 9.1.10
- uname:
Linux pve 7.0.2-4-pve #1 SMP PREEMPT_DYNAMIC PMX 7.0.2-4 (2026-05-15T07:32Z) x86_64 GNU/Linux

Mail VM on Host 1 / later migrated to NUC:
- VMID: 1004
- name: Mail
- agent: 1
- ostype: l26
- machine: q35
- meta: creation-qemu=6.2.0
- cores: 2
- sockets: 2
- memory: 4096 MB
- numa: 0
- scsihw: virtio-scsi-pci
- disks: qcow2, discard=on, one 34G SSD-marked disk and one 100G disk
- net0: virtio

Host 2 - Intel NUC:
- Model: Intel NUC 7i3BNB
- CPU: Intel Core i3-7100U @ 2.40GHz
- Logical CPUs: 4
- Threads per core: 2
- Cores per socket: 2
- Sockets: 1
- Virtualization: VT-x
- RAM: 16 GB
- Running kernel: 7.0.2-4-pve
- pve-manager: 9.1.14
- pve-qemu-kvm: 11.0.0-2
- qemu-server: 9.1.12
- uname:
Linux pvemini 7.0.2-4-pve #1 SMP PREEMPT_DYNAMIC PMX 7.0.2-4 (2026-05-15T07:32Z) x86_64 GNU/Linux

Mail VM on NUC:
- VMID: 1004
- name: Mail
- agent: 1
- allow-ksm: 0
- balloon: 0
- ostype: l26
- machine: q35
- meta: creation-qemu=6.2.0
- cores: 2
- sockets: 2
- memory: 4096 MB
- numa: 0
- scsihw: virtio-scsi-pci
- disks: local qcow2, discard=on, 34G + 100G
- net0: virtio

Host 3:
- Motherboard: Supermicro X8DT3
- CPU: Intel Xeon X5675 @ 3.07GHz
- Logical CPUs: 24
- Threads per core: 2
- Cores per socket: 6
- Sockets: 2
- Virtualization: VT-x
- Running kernel: 7.0.2-4-pve
- pve-manager: 9.1.11
- pve-qemu-kvm: 11.0.0-2
- qemu-server: 9.1.10
- uname:
Linux pve 7.0.2-4-pve #1 SMP PREEMPT_DYNAMIC PMX 7.0.2-4 (2026-05-15T07:32Z) x86_64 GNU/Linux

Mail VM on Host 3:
- VMID: 1111
- name: Mail
- agent: 1
- ostype: l26
- machine: q35
- meta: creation-qemu=9.0.2
- cores: 2
- sockets: 2
- memory: 4096 MB
- numa: 0
- scsihw: virtio-scsi-single
- disks: qcow2, discard=on, iothread=1, 34G + 100G
- net0: virtio

Host 4:
- Motherboard: unknown
- CPU: Intel Celeron G530 @ 2.40GHz
- Logical CPUs: 2
- Threads per core: 1
- Cores per socket: 2
- Sockets: 1
- Virtualization: VT-x
- Running kernel: 7.0.2-4-pve
- pve-manager: 9.1.14
- pve-qemu-kvm: 11.0.0-2
- qemu-server: 9.1.12
- uname:
Linux backup 7.0.2-4-pve #1 SMP PREEMPT_DYNAMIC PMX 7.0.2-4 (2026-05-15T07:32Z) x86_64 GNU/Linux

ISPConfig VM on Host 4:
- VMID: 1000
- name: ISPConfig
- ostype: l26
- machine: q35
- meta: creation-qemu=6.1.1
- cores: 1
- sockets: 2
- memory: 2048 MB
- numa: 0
- scsihw: virtio-scsi-single
- disk: local qcow2, discard=on, size=37G
- net0: virtio, bridge=vmbr1

Questions:
1. Has anyone seen random segfaults inside AlmaLinux 8 guests on Proxmox VE 9.1 / QEMU 11?
2. Could this be related to Proxmox kernel 7.0.x / QEMU 11 / KVM interaction with AlmaLinux 8's 4.18 kernel or glibc/runtime stack?
3. Are there known issues with q35 + AlmaLinux 8 + virtio devices on current Proxmox?
4. Would you recommend testing an older Proxmox kernel, an older QEMU stack, or changing VM machine type/CPU topology?
5. Are there specific VM settings you would test first: CPU type, sockets/cores topology, machine type, scsihw, iothread, ballooning, KSM, NUMA?
6. What logs/backtraces would be most useful to collect next?

My current thinking:
This does not look like a single bad RAM module or one broken host, because the same class of problem appears across multiple Intel hosts and multiple AlmaLinux 8 VMs. It also does not look tied to only cPanel, KernelCare, or one application stack. Sinc
e unrelated daemons crash in libc/loader/pthread/language modules and sometimes at null IP, I suspect either a low-level guest runtime/package interaction or a hypervisor/virt layer interaction.

 

[PVasileff]

I would like to add one more important detail to the original post.

On the main Proxmox server where the random segmentation faults are observed in AlmaLinux 8 VMs, I also have other Linux virtual machines which do not show the same problem.

On the same Proxmox host, I am running:
- Red Hat Enterprise Linux 9 VM
- Debian 13 VMs
- Debian 12 VMs
- Debian 8 VMs

So far, these Red Hat 9 and Debian guests do not show random segmentation faults like the AlmaLinux 8 guests do.

This makes me think even more that the issue is probably not just a general host hardware problem or a general Proxmox instability issue. It looks more specific to the AlmaLinux 8 guest environment, or to an interaction between AlmaLinux 8 and the curre
nt Proxmox VE / QEMU / KVM stack.

 

[EugenMayer]

We have the very same issue, while we rather use debian12 VMs in this regard. Segmentation faults are happening for different processes.

What we have tried:

• latest PVE 7.x or 6.x kernel - happens on both - currently running 6.17.13-9-pve
• we have PVEs with and without enterprise subscription, happens for both
• disabled ksm on all VMs
• disabled ballooning on all VMs
• rebooting the VMs and the hypervisor

All affected debian VMs have:

• the latest stable linux kernel 6.1.0-48-amd64
• docker engine: Docker version 29.4.3
• all debian updates installed (bookworm as of today)
• all have qemu guest agent installed

All affected proxmox servers have:

• all PVEs have all updates installed
• qeme-server 9.1.15
• pve-manager 9.1.18
• pve-qeme-kvm 11.0.0-2
• pve-firmware: 3.18-3
• 
  
• kernel running is 6.17.13-9-pve, 6.17.13-10-pve on next reboot

Odd facts:
- Customers that run the very same VMs with the same patch level, but not on proxmox but other hypervisors, have no such issues.
- Several (means all) proxmox servers are affected (6 different), with different hardware (~128GB ram, CPU is either AMD Ryzen 9 5950X , Intel(R) Core(TM) Ultra 7 265, AMD Ryzen 9 7950X3D )

As far as we can see this started either 14 or 15 Mai.

Segfaults look like this

[Tue May 19 17:17:47 2026] supervisord[1521316]: segfault at 0 ip 0000000000573b83 sp 00007ffd1a2e9350 error 4 in python3.13[420000+31f000] likely on CPU 1 (core 1, socket 0)
[Tue May 19 17:17:48 2026] supervisord[1520370]: segfault at 0 ip 000000000061c95a sp 00007fffbe8ec560 error 4 in python3.12[420000+2e3000] likely on CPU 2 (core 2, socket 0)
[Tue May 19 17:18:36 2026] supervisord[1522412]: segfault at 0 ip 00000000005a71f5 sp 00007ffe98588610 error 4 in python3.13[420000+31f000] likely on CPU 2 (core 2, socket 0)
[Tue May 19 17:50:15 2026] gunicorn[1878169]: segfault at 0 ip 00005652c9ff874c sp 00007fffa5233730 error 6 in python3.10[5652c9ef0000+2b5000] likely on CPU 2 (core 2, socket 0)
[Tue May 19 17:50:15 2026] gunicorn[1878733]: segfault at 8 ip 00005652c9fe0fd0 sp 00007fffa5234710 error 4 in python3.10[5652c9ef0000+2b5000] likely on CPU 2 (core 2, socket 0)
[Tue May 19 17:50:25 2026] supervisord[1876648]: segfault at 8 ip 000000000050ba80 sp 00007ffd32394350 error 4 in python3.11[41f000+2b3000] likely on CPU 1 (core 1, socket 0)
[Tue May 19 17:50:31 2026] supervisord[1876465]: segfault at a9 ip 000000000054c981 sp 00007ffdae81bb60 error 4 in python3.11[41f000+2b3000] likely on CPU 2 (core 2, socket 0)

Example configuration of one VM

acpi: 1
agent: enabled=1,fstrim_cloned_disks=0,type=virtio
allow-ksm: 0
balloon: 0
bios: seabios
boot: order=virtio0;net0
cores: 2
cpu: host
keyboard: en-us
memory: 12000
meta: creation-qemu=10.1.2,ctime=1766156649
name: <redacted>
net0: virtio=<redacted>,bridge=vmbr40,firewall=0,mtu=1400
numa: 0
onboot: 1
ostype: l26
protection: 1
scsihw: virtio-scsi-pci
serial0: socket
smbios1: uuid=<redacted>
sockets: 1
startup: order=3,up=5,down=30
tablet: 1
template: 0
vga: memory=16,type=std
virtio0: local:100/vm-100-disk-0.qcow2,aio=io_uring,backup=1,cache=none,discard=ignore,iothread=0,replicate=1,size=120G
vmgenid: 700d2260-e052-4639-9288-9622bf93c49f

 

[fabian]

did you upgrade just the PVE packages, or also those inside the guest? which package updates exactly coincide with the start of the problems (keep in mind that kernel upgrades require a reboot, and pve-qemu-kvm updates will only affect newly started VM processes, e.g. live-migration or freshly started VMs, not already running ones)?

 

[fiona]

Hi,

@PVasileff are you also using the host CPU type for your VMs?

@ both, could you try with a different model like x86-64-v2-AES/x86-64-v3 or a vendor-specific one close to your physical model?

 

[EugenMayer]

@fabian we update our packages daily, within the VMs and on the proxmox server. So we updated to whatever was available to debian12 at this moment, including what is available for PVE.

The reason that this is cleary proxmox and not 'vm software' related is, that we run the exact same VM, with the excact same payloads on other hypervisors (hundreds) and none of those have this issue. Only on our proxmox those errors seem to happen.

We have to use host-cpu due to MongDB (AVX). When you refer to 'x86-64-v2-AES/x86-64-v3' is this even possible when using 'host CPU'

 

[fiona]

We have to use host-cpu due to MongDB (AVX). When you refer to 'x86-64-v2-AES/x86-64-v3' is this even possible when using 'host CPU'

The v3 model does include flags for avx, use v4 if you need the avx512 flags too, and a vendor-specific one close to your physical model will support those flags too.

 

[EugenMayer]

Are you expected that 'non-host' / x86-64-v2-AES/x86-64-v3 should be less error prone than host? I would have suggested, that this is unexpected, but i follow your advice, if that is the case

 

[fiona]

When host is used, there can be quirks that are not present with virtual models. I'm not betting on it resolving the issue, since multiple physical models are affected, but there is a chance and it is a valuable data point for the investigation.

 

[PVasileff]

Thanks for the replies.

I want to add another datapoint.

I gave a backup of one of my affected AlmaLinux 8 VMs to a colleague, and he restored/tested it on two of his own Proxmox hosts.

Also, on the problematic AlmaLinux 8 servers, fsck has been run inside the guests against the filesystems and no filesystem errors were found. So a broken guest filesystem does not currently look like the likely explanation.

To answer the update question explicitly: all available updates were applied both on the Proxmox hosts and inside the affected virtual servers. The affected AlmaLinux 8 guests were fully updated, and the Proxmox hosts were also updated to the currently
available packages when the issue was observed.

Independent restore test 1 - already updated Intel Proxmox host

My colleague first restored the VM on an already updated Intel Proxmox host:

- Hardware: Supermicro X9DR3-F
- CPU: 2x Intel Xeon E5-2667 v2 @ 3.30GHz
- Logical CPUs: 32
- Virtualization: VT-x
- Proxmox VE: 9.1.0
- Running kernel: 7.0.2-4-pve
- pve-manager: 9.1.14
- pve-qemu-kvm: 11.0.0-2
- qemu-server: 9.1.12
- pve-firmware: 3.18-3
- pve-edk2-firmware: not correctly installed
- intel-microcode: 3.20251111.1~deb13u1

After restoring and starting the provided AlmaLinux 8 VM, the issue reproduced almost immediately: segmentation faults started appearing inside the guest.

Independent restore test 2 - AMD Proxmox host before and after update

He then restored the same backup on a second Proxmox host with AMD CPU. Before updating this host, the restored AlmaLinux 8 VM did not show the segmentation fault behavior.

Host before update:

proxmox-ve: 9.1.0 (running kernel: 7.0.0-3-pve)
pve-manager: 9.1.9 (running version: 9.1.9/ee7bad0a3d1546c9)
proxmox-kernel-helper: 9.0.4
proxmox-kernel-7.0: 7.0.0-3
proxmox-kernel-7.0.0-3-pve-signed: 7.0.0-3
proxmox-kernel-6.17: 6.17.13-6
proxmox-kernel-6.17.13-6-pve-signed: 6.17.13-6
proxmox-kernel-6.17.13-2-pve-signed: 6.17.13-2
proxmox-kernel-6.17.13-1-pve-signed: 6.17.13-1
proxmox-kernel-6.17.9-1-pve-signed: 6.17.9-1
proxmox-kernel-6.17.4-2-pve-signed: 6.17.4-2
proxmox-kernel-6.17.4-1-pve-signed: 6.17.4-1
pve-edk2-firmware: not correctly installed
pve-firmware: 3.18-3
pve-qemu-kvm: 10.1.2-7
qemu-server: 9.1.9

Linux amdhv 7.0.0-3-pve #1 SMP PREEMPT_DYNAMIC PMX 7.0.0-3 (2026-04-21T22:56Z) x86_64 GNU/Linux

Hardware:

- CPU: AMD Ryzen 7 8845HS w/ Radeon 780M Graphics
- Logical CPUs: 16
- Cores/socket: 8
- Threads/core: 2
- Virtualization: AMD-V

VM config on this AMD host:

agent: 1,fstrim_cloned_disks=1
boot: order=ide2;scsi0;net0
cores: 1
ide2: none,media=cdrom
machine: q35
memory: 2048
meta: creation-qemu=6.1.1,ctime=1645945861
name: ISPConfig
net0: virtio=...,bridge=vmbr0
numa: 0
ostype: l26
scsi0: storage:1000/vm-1000-disk-0.qcow2,discard=on,iothread=1,size=37G,ssd=1
scsihw: virtio-scsi-single
sockets: 2

At that point, this AMD host had pending updates. The most relevant ones were:

proxmox-kernel-7.0: 7.0.0-3 -> 7.0.2-4
proxmox-kernel-6.17: 6.17.13-6 -> 6.17.13-9
pve-manager: 9.1.9 -> 9.1.16
pve-qemu-kvm: 10.1.2-7 -> 11.0.0-2
qemu-server: 9.1.9 -> 9.1.14
lxc-pve: 6.0.5-4 -> 7.0.0-1
lxcfs: 6.0.4-pve1 -> 7.0.0-pve1
libpve-access-control: 9.0.7 -> 9.1.1
libpve-cluster-api-perl: 9.1.2 -> 9.1.5
libpve-cluster-perl: 9.1.2 -> 9.1.5
libpve-common-perl: 9.1.11 -> 9.1.12
libpve-guest-common-perl: 6.0.2 -> 6.0.3
libpve-network-api-perl: 1.3.0 -> 1.6.1
libpve-network-perl: 1.3.0 -> 1.6.1
libpve-notify-perl: 9.1.2 -> 9.1.5
libpve-rs-perl: 0.13.0 -> 0.15.1
libpve-storage-perl: 9.1.2 -> 9.1.5
pve-cluster: 9.1.2 -> 9.1.5
pve-container: 6.1.5 -> 6.1.9
pve-docs: 9.1.2 -> 9.1.5
pve-ha-manager: 5.2.0 -> 5.2.3
pve-xtermjs: 5.5.0-3 -> 6.0.0-1
pve-yew-mobile-gui: 0.6.4 -> 0.7.0
proxmox-widget-toolkit: 5.1.9 -> 5.2.1
zfsutils-linux: 2.4.1-pve1 -> 2.4.2-pve1
zfs-zed: 2.4.1-pve1 -> 2.4.2-pve1
libzfs7linux: 2.4.1-pve1 -> 2.4.2-pve1
libzpool7linux: 2.4.1-pve1 -> 2.4.2-pve1
libnvpair3linux: 2.4.1-pve1 -> 2.4.2-pve1
libuutil3linux: 2.4.1-pve1 -> 2.4.2-pve1
frr/frr-pythontools: 10.4.1-1+pve1 -> 10.6.1-1+pve2

There were also normal Debian 13 package updates pending on that host, including systemd 257.9 -> 257.13, libc6 2.41-12+deb13u2 -> 2.41-12+deb13u3, OpenSSL 3.5.5 -> 3.5.6, e2fsprogs 1.47.2-3+b10 -> 1.47.2-3+b11, initramfs-tools 0.148.3 -> 0.148.4, opens
sh 10.0p1-7+deb13u2 -> 10.0p1-7+deb13u4, and related libraries.

After updating the AMD Proxmox host, the same restored VM started showing the same segmentation fault behavior.

So in this specific test, the guest VM backup was the same. The important variable was the Proxmox host update. Before the update on the AMD host, the restored VM was not showing the issue. After the update, it started showing it.

Regarding CPU type

To answer Fiona's CPU type question: on my main/big Proxmox server I tested CPU type host with the affected VM. That host is:

- CPU: 2x Intel Xeon X5675 @ 3.07GHz
- Proxmox host: the large Supermicro server mentioned in the first post

Using CPU type host on that server did not fix the issue.

On my own affected VMs I have also tested:

- CPU type x86-64-v2 - issue still reproduced

On that older Xeon X5675 host I do not have x86-64-v2-AES/x86-64-v3 available as options. I can try moving the VM back to the Intel NUC again, or I can ask my colleague to start the restored VM with x86-64-v2-AES, x86-64-v3, or a vendor-specific CPU mod
el on hardware where those options are available.

In my colleague's AMD restore test, the VM config does not contain an explicit cpu line, so that test was not using host CPU explicitly.

This now looks less likely to be one broken physical host, because the same class of issue reproduced on my hosts and on a colleague's separate Intel host. The AMD test is the most interesting datapoint, because the same restored AlmaLinux 8 VM was fine
before the Proxmox update and started showing the same segmentation faults after the host update.

The most suspicious change from that AMD A/B test is the move from:

- kernel 7.0.0-3-pve to 7.0.2-4-pve
- pve-qemu-kvm 10.1.2-7 to 11.0.0-2
- qemu-server 9.1.9 to 9.1.14
- plus the related PVE libraries and Debian 13 updates listed above

I can provide the full apt list output from before the AMD host update if needed.

 

[fabian]

please provide the "/var/log/apt/history.log" entries that correlate with the start of the issues.

 

[PVasileff]

Here are the relevant `/var/log/apt/history.log` entries from the AMD Proxmox host `amdhv`.

This is the host where the same restored AlmaLinux 8 VM did not show the issue before the update, but started showing segmentation faults immediately after the Proxmox host update.

Start-Date: 2026-05-19  10:14:47
Commandline: apt install pve-edk2-firmware
Install: pve-edk2-firmware:amd64 (4.2025.05-2)
End-Date: 2026-05-19  10:14:48

Start-Date: 2026-05-19  10:20:38
Commandline: apt dist-upgrade
Install: proxmox-kernel-6.17.13-9-pve-signed:amd64 (6.17.13-9, automatic), wireguard-tools:amd64 (1.0.20210914-3, automatic), proxmox-kernel-7.0.2-4-pve-signed:amd64 (7.0.2-4, automatic), proxmox-enterprise-support-keyring:amd64 (1.0, automatic)
Upgrade: pve-docs:amd64 (9.1.2, 9.1.5), initramfs-tools-core:amd64 (0.148.3, 0.148.4), udev:amd64 (257.9-1~deb13u1, 257.13-1~deb13u1), python3.13:amd64 (3.13.5-2, 3.13.5-2+deb13u2), libnghttp2-14:amd64 (1.64.0-1.1, 1.64.0-1.1+deb13u1), openssh-client:amd64 (1:10.0p1-7+deb13u2, 1:10.0p1-7+deb13u4), proxmox-widget-toolkit:amd64 (5.1.9, 5.2.1), libpve-rs-perl:amd64 (0.13.0, 0.15.1), frr:amd64 (10.4.1-1+pve1, 10.6.1-1+pve2), gpg:amd64 (2.4.7-21+deb13u1+b2, 2.4.7-21+deb13u1+b3), tzdata:amd64 (2026a-0+deb13u1, 2026b-0+deb13u1), zfs-zed:amd64 (2.4.1-pve1, 2.4.2-pve1), libpam-systemd:amd64 (257.9-1~deb13u1, 257.13-1~deb13u1), busybox:amd64 (1:1.37.0-6+b7, 1:1.37.0-6+b8), libzfs7linux:amd64 (2.4.1-pve1, 2.4.2-pve1), sed:amd64 (4.9-2, 4.9-2+deb13u1), pve-qemu-kvm:amd64 (10.1.2-7, 11.0.0-2), libnvpair3linux:amd64 (2.4.1-pve1, 2.4.2-pve1), xsltproc:amd64 (1.1.35-1.2+deb13u2, 1.1.35-1.2+deb13u3), libcap2-bin:amd64 (1:2.75-10+b8, 1:2.75-10+deb13u1+b1), gir1.2-glib-2.0:amd64 (2.84.4-3~deb13u2, 2.84.4-3~deb13u3), libpve-cluster-api-perl:amd64 (9.1.2, 9.1.5), libext2fs2t64:amd64 (1.47.2-3+b10, 1.47.2-3+b11), pve-ha-manager:amd64 (5.2.0, 5.2.3), lxcfs:amd64 (6.0.4-pve1, 7.0.0-pve1), libpython3.13-stdlib:amd64 (3.13.5-2, 3.13.5-2+deb13u2), initramfs-tools-bin:amd64 (0.148.3, 0.148.4), libuutil3linux:amd64 (2.4.1-pve1, 2.4.2-pve1), libpve-storage-perl:amd64 (9.1.2, 9.1.5), openssl-provider-legacy:amd64 (3.5.5-1~deb13u2, 3.5.6-1~deb13u1), libsystemd0:amd64 (257.9-1~deb13u1, 257.13-1~deb13u1), linux-cpupower:amd64 (6.12.85-1, 6.12.88-1), libnss-systemd:amd64 (257.9-1~deb13u1, 257.13-1~deb13u1), libpve-guest-common-perl:amd64 (6.0.2, 6.0.3), libbson-1.0-0t64:amd64 (1.30.4-1+deb13u1, 1.30.4-1+deb13u2), openssh-server:amd64 (1:10.0p1-7+deb13u2, 1:10.0p1-7+deb13u4), proxmox-kernel-7.0:amd64 (7.0.0-3, 7.0.2-4), libpython3.13-minimal:amd64 (3.13.5-2, 3.13.5-2+deb13u2), pve-cluster:amd64 (9.1.2, 9.1.5), libglib2.0-data:amd64 (2.84.4-3~deb13u2, 2.84.4-3~deb13u3), libzpool7linux:amd64 (2.4.1-pve1, 2.4.2-pve1), systemd:amd64 (257.9-1~deb13u1, 257.13-1~deb13u1), libudev1:amd64 (257.9-1~deb13u1, 257.13-1~deb13u1), gpg-agent:amd64 (2.4.7-21+deb13u1+b2, 2.4.7-21+deb13u1+b3), libpostproc58:amd64 (7:7.1.3-0+deb13u1, 7:7.1.4-0+deb13u1), libcurl3t64-gnutls:amd64 (8.14.1-2+deb13u2, 8.14.1-2+deb13u3), libcom-err2:amd64 (1.47.2-3+b10, 1.47.2-3+b11), lxc-pve:amd64 (6.0.5-4, 7.0.0-1), libarchive13t64:amd64 (3.7.4-4, 3.7.4-4+deb13u1), novnc-pve:amd64 (1.6.0-4, 1.7.0-1), libjxl0.11:amd64 (0.11.1-4, 0.11.2-0.1~deb13u1), libcap2:amd64 (1:2.75-10+b8, 1:2.75-10+deb13u1+b1), libc6:amd64 (2.41-12+deb13u2, 2.41-12+deb13u3), locales:amd64 (2.41-12+deb13u2, 2.41-12+deb13u3), libmongoc-1.0-0t64:amd64 (1.30.4-1+deb13u1, 1.30.4-1+deb13u2), proxmox-kernel-6.17:amd64 (6.17.13-6, 6.17.13-9), libpython3.13:amd64 (3.13.5-2, 3.13.5-2+deb13u2), libavcodec61:amd64 (7:7.1.3-0+deb13u1, 7:7.1.4-0+deb13u1), pve-xtermjs:amd64 (5.5.0-3, 6.0.0-1), qemu-server:amd64 (9.1.9, 9.1.14), gpgv:amd64 (2.4.7-21+deb13u1+b2, 2.4.7-21+deb13u1+b3), libpve-access-control:amd64 (9.0.7, 9.1.1), bash:amd64 (5.2.37-2+b8, 5.2.37-2+b9), pve-container:amd64 (6.1.5, 6.1.9), shim-helpers-amd64-signed:amd64 (1+15.8+1+pmx1, 1+16.1+1+pmx1), libavfilter10:amd64 (7:7.1.3-0+deb13u1, 7:7.1.4-0+deb13u1), base-files:amd64 (13.8+deb13u4, 13.8+deb13u5), frr-pythontools:amd64 (10.4.1-1+pve1, 10.6.1-1+pve2), gnutls-bin:amd64 (3.8.9-3+deb13u2, 3.8.9-3+deb13u3), libavutil59:amd64 (7:7.1.3-0+deb13u1, 7:7.1.4-0+deb13u1), rsync:amd64 (3.4.1+ds1-5+deb13u1, 3.4.1+ds1-5+deb13u2), libcurl4t64:amd64 (8.14.1-2+deb13u2, 8.14.1-2+deb13u3), gpgsm:amd64 (2.4.7-21+deb13u1+b2, 2.4.7-21+deb13u1+b3), libxslt1.1:amd64 (1.1.35-1.2+deb13u2, 1.1.35-1.2+deb13u3), libunbound8:amd64 (1.22.0-2+deb13u1, 1.22.0-2+deb13u2), libswscale8:amd64 (7:7.1.3-0+deb13u1, 7:7.1.4-0+deb13u1), libpve-network-api-perl:amd64 (1.3.0, 1.6.1), libgnutls-dane0t64:amd64 (3.8.9-3+deb13u2, 3.8.9-3+deb13u3), distro-info-data:amd64 (0.66+deb13u1, 0.66+deb13u2), smartmontools:amd64 (7.4-pve1, 7.5-pve2), libopenjp2-7:amd64 (2.5.3-2.1~deb13u1, 2.5.3-2.1~deb13u2), pve-manager:amd64 (9.1.9, 9.1.16), libpve-common-perl:amd64 (9.1.11, 9.1.12), openssh-sftp-server:amd64 (1:10.0p1-7+deb13u2, 1:10.0p1-7+deb13u4), libpve-network-perl:amd64 (1.3.0, 1.6.1), nano:amd64 (8.4-1, 8.4-1+deb13u1), libc-l10n:amd64 (2.41-12+deb13u2, 2.41-12+deb13u3), python3.13-minimal:amd64 (3.13.5-2, 3.13.5-2+deb13u2), sudo:amd64 (1.9.16p2-3+deb13u1, 1.9.16p2-3+deb13u2), libc-bin:amd64 (2.41-12+deb13u2, 2.41-12+deb13u3), libsystemd-shared:amd64 (257.9-1~deb13u1, 257.13-1~deb13u1), libswresample5:amd64 (7:7.1.3-0+deb13u1, 7:7.1.4-0+deb13u1), libpve-notify-perl:amd64 (9.1.2, 9.1.5), initramfs-tools:amd64 (0.148.3, 0.148.4), libssl3t64:amd64 (3.5.5-1~deb13u2, 3.5.6-1~deb13u1), libgnutls30t64:amd64 (3.8.9-3+deb13u2, 3.8.9-3+deb13u3), dirmngr:amd64 (2.4.7-21+deb13u1+b2, 2.4.7-21+deb13u1+b3), jq:amd64 (1.7.1-6+deb13u1, 1.7.1-6+deb13u2), libharfbuzz0b:amd64 (10.2.0-1+b1, 10.2.0-1+deb13u1), systemd-sysv:amd64 (257.9-1~deb13u1, 257.13-1~deb13u1), gnupg-utils:amd64 (2.4.7-21+deb13u1+b2, 2.4.7-21+deb13u1+b3), libavformat61:amd64 (7:7.1.3-0+deb13u1, 7:7.1.4-0+deb13u1), pve-yew-mobile-gui:amd64 (0.6.4, 0.7.0), curl:amd64 (8.14.1-2+deb13u2, 8.14.1-2+deb13u3), logsave:amd64 (1.47.2-3+b10, 1.47.2-3+b11), libcpupower1:amd64 (6.12.85-1, 6.12.88-1), gpg-wks-client:amd64 (2.4.7-21+deb13u1+b2, 2.4.7-21+deb13u1+b3), libjq1:amd64 (1.7.1-6+deb13u1, 1.7.1-6+deb13u2), liblcms2-2:amd64 (2.16-2, 2.16-2+deb13u2), libpng16-16t64:amd64 (1.6.48-1+deb13u4, 1.6.48-1+deb13u5), shim-signed:amd64 (1.47+pmx1+15.8-1+pmx1, 1.48+pmx1+16.1-1+pmx1), libpq5:amd64 (17.9-0+deb13u1, 17.10-0+deb13u1), gpgconf:amd64 (2.4.7-21+deb13u1+b2, 2.4.7-21+deb13u1+b3), libss2:amd64 (1.47.2-3+b10, 1.47.2-3+b11), shim-signed-common:amd64 (1.47+pmx1+15.8-1+pmx1, 1.48+pmx1+16.1-1+pmx1), zfsutils-linux:amd64 (2.4.1-pve1, 2.4.2-pve1), shim-unsigned:amd64 (15.8-1+pmx1, 16.1-1+pmx1), openssl:amd64 (3.5.5-1~deb13u2, 3.5.6-1~deb13u1), libglib2.0-0t64:amd64 (2.84.4-3~deb13u2, 2.84.4-3~deb13u3), e2fsprogs:amd64 (1.47.2-3+b10, 1.47.2-3+b11), libpve-cluster-perl:amd64 (9.1.2, 9.1.5)
End-Date: 2026-05-19  10:21:42

 

[fabian]

when you say "immediately after the update" do you mean after those packages got upgraded *and the host got rebooted*? if so, which kernel version was booted before/after?

 

[PVasileff]

Yes. Immediately after the package upgrade, the Proxmox host was rebooted, and then the same restored VM was started again.

Before the upgrade/reboot, `amdhv` was running:

7.0.0-3-pve

After the reboot, it was running:

7.0.2-4-pve

 

[fabian]

if it is possible for you to do this in a non-production environment (this build does not contain the most recent wave of security fixes, and has no built-in ZFS module!), could you try the Ubuntu mainline kernel build of 7.0.2:

https://kernel.ubuntu.com/mainline/v7.0.2/amd64/

as well as the intermediate builds of our kernels (comes with ZFS, but same caveat about security fixes applies):

proxmox-kernel-7.0.2-1-pve
proxmox-kernel-7.0.2-2-pve
proxmox-kernel-7.0.2-3-pve

 

[fiona]

It could also be a QEMU regression. Could you also try downgrading the QEMU package (independently of the kernel)? VMs have to be started fresh to be running with the downgraded QEMU binary (shutdown+start or using the Reboot button in the UI, reboot inside the guest is not enough).

 

[fever_wits]

Hello,

The amdhv + Supermicro X9DR3-F servers are mine. Both are currently in production.
I have a third server on which we are currently doing tests with a restored VM from backup.
The VM still had errors.

The server has the following data:
CPU: Intel(R) Atom(TM) CPU C3758 @ 2.20GHz
RAM: 2 x 16GB ECC sodim
Disk: 2 x SSD 200GB

This is another minipc:
Qoton Q20332G9 S20

OS: 37G
Storage - ZFS.
If I install a kernel without ZFS support, I will not be able to do tests.
What I can suggest is to mount external storage via NFS from my large server (Supermicro X9DR3-F) and the VM disk to be on an NFS array.

But I will be able to do this in a few hours.

I would like to ask for confirmation that you agree with this proposal.
If so, I would like to ask for precise instructions on what and how to do.
This is in order to make the correct diagnosis.

Regards
A.H.

 

[fiona]

Hi @fever_wits,
on the third server, could you first test the older kernel builds with ZFS support:

proxmox-kernel-7.0.0-3-pve
proxmox-kernel-7.0.2-1-pve
proxmox-kernel-7.0.2-2-pve
proxmox-kernel-7.0.2-3-pve

For each kernel, you can run (assuming you have the -signed variant for secure boot):

apt install proxmox-kernel-7.0.0-3-pve-signed
proxmox-boot-tool kernel pin 7.0.2-3-pve --next-boot

and then reboot and test.

After the kernel tests, reboot into 7.0.2-4-pve and then try the QEMU downgrade:

apt install pve-qemu-kvm=10.1.2-7

only then start the VM fresh.

And then stop the VM, run

apt install pve-qemu-kvm=10.2.1-2

and again, start the VM fresh.

Could you also share excerpts from the host system logs/journal from around the time the segfaults are happening within the guest?

 

[fever_wits]

Hello,

I spoke to @PVasileff , he tested older kernel versions. That doesn't help. As far as I can tell it's related to disk creation and management.
On minihv (Qoton Q20332G9 S20) today we created 2 VMs:
- Alma 9.7, then updated - after the last reboot there are no segmentation faults
- Alma 10v2, then updated - after the last reboot there are no segmentation faults

There are 3 VMs on minihv
- Alma 8 - which has a disk made with an old version of pve-qemu-kvm (The file system was created around February 1, 2025), and we cannot say with certainty which version of pve-qemu-kvm the disk was created with.
- Alma 9.7 - with pve-qemu-kvm: 11.0.0-2 and no segmentation faults
- Alma 10 v2 - with pve-qemu-kvm: 11.0.0-2 and no segmentation faults

We have currently done this experiment.
VM with Alma 8 is stopped. Then from the "Hardware" menu select the disk > Disk Action > Move Storage
- Target Storage - The same storage is selected
- Format - Raw disk image (raw) is selected - i.e. we convert the disk from qcow2 to raw format, with the server stopped.
- Delete source: Yes

This was done about 2 hours ago. Since then there has been no problem. But as far as I can tell it is good to wait a few hours before confirming.
In parallel with this @PVasileff , do the same on his VM, and we are also waiting to see if there will be segmentation faults.

Unfortunately, we cannot find a way to intentionally cause segmentation faults and that is why we are waiting.

My suggestion is: Let's wait a few more hours, then I'll restore the VM from backup. Then I'll downgrade the version of pve-qemu-kvm and test.
But again, it will have to wait.

I'm in a meeting about what to help with and how to proceed.

Best regards,
A.H.

 

[EugenMayer]

Thanks for all the informations and work already done above!

I'am currently on the way to test the downgrade of pve-qemu-kvm to 10.1.2 to see if that helps. In my book, the kernel of the host is less likely the cause here (this is only a bet). I guess this aligns with the existing experiments above.

I will report back tomorrow after I can observe the effect of this downgrade