Repeated host-level outage during PBS backups after recent kernel updates — what evidence should I collect?

[jbhorner]

I am troubleshooting repeated Proxmox VE host-level outages that appear temporally correlated with recent Proxmox kernel/runtime updates. I am not trying to assign blame prematurely, but I would like guidance on what evidence would be most useful if this happens again. These circumstances began after I restarted the server a couple of days ago; prior to that, the last restart was June.

Summary​

I have had multiple incidents where the PVE host suffers a host-level outage during scheduled PBS backups. The host remains pingable and disk activity continues, but SSH/console login becomes inaccessible and all VMs stop functioning. This does not look like a total power loss, but it is also not merely degraded performance: from a service perspective, the host and its VMs are down.

The common factor so far is that the failures occurred during PVE-side vzdump/QEMU backups to PBS storage pbs-tn.

​

Environment​

PVE host: pve
Local storage: single Samsung 990 PRO 4TB NVMe
Root/storage pool: ZFS rpool
Backup path: PVE -> PBS -> datastore on NFS/TrueNAS
PVE backup storage: pbs-tn
PBS datastore: TRUENASNFS
Two affected VMs are materially different:

VM101: Home Assistant OS
- Disk on local-zfs
- Has USB passthrough for Z-Wave/Zigbee

VM106: Docker VM
- Disk on local qcow2
- No USB passthrough

That makes me doubt this is VM101-specific, HAOS-specific, USB-specific, or local-zfs-specific.

Recent VM106 incident​

VM106 backup started normally:
INFO: starting new backup job: vzdump 106 ... --storage pbs-tn ... --mode suspend
INFO: Starting Backup of VM 106 (qemu)
INFO: Backup started at 2026-08-02 02:30:03
INFO: VM Name: dockervm
INFO: include disk 'virtio0' 'local:106/vm-106-disk-0.qcow2' 253G
INFO: creating Proxmox Backup Server archive 'vm/106/2026-08-02T09:30:03Z'
INFO: starting backup via QMP command
INFO: started backup task '470792a0-f4f0-4edb-8873-6486b205ee5a'
INFO: resuming VM again after 1 seconds
INFO: virtio0: dirty-bitmap status: created new
INFO: 41% (103.9 GiB of 253.0 GiB) in 3m 36s, read: 495.2 MiB/s, write: 819.2 KiB/s

At/after that point, the PVE host became inaccessible and all VMs stopped functioning.

Evidence capture limitation​

One difficulty is that evidence capture appears to stop when the issue begins. Local logging and remote logging both cease to provide useful new entries at or near the onset of the outage.

For example, the VM106 task log stops at normal backup progress, and the PVE kernel journal for the incident window does not show storage, NVMe, PCIe, or obvious kernel error messages.

Because the host remains pingable but SSH/console login and VM workloads stop functioning, I am not sure whether the best next capture method would be netconsole, serial console, SysRq task dump, hung-task output, or something else.

PBS / TrueNAS observations​

I considered whether PBS, NFS, or TrueNAS was blocking. So far I do not see evidence for that.

PBS journal during the VM106 incident window showed the backup starting, but I did not find PBS-side log evidence of:
NFS timeout
stale file handle
datastore lock wait
I/O error
blocked task
hung task
storage-path failure
The only relevant PBS journal lines I found were:
Aug 02 02:30:04 pbs proxmox-backup-proxy[744]: starting new backup on datastore 'TRUENASNFS' from ::ffff:192.168.70.5: "vm/106/2026-08-02T09:30:03Z"
Aug 02 03:21:43 pbs proxmox-backup-proxy[744]: TASK ERROR: verification failed - please check the log for details

So at the moment, I do not have positive evidence that PBS/TrueNAS initiated the PVE host outage.

Local disk checks​

The PVE host has only one local storage disk: a Samsung 990 PRO NVMe backing rpool.

zpool status -v reports:
rpool ONLINE
nvme-eui.002538424141c0a2-part3 ONLINE 0 0 0
errors: No known data errors
The most recent scrub repaired 0B with 0 errors. Pool capacity is around 23%.

I also checked the kernel journal for the VM106 incident window and did not find NVMe timeouts, controller resets, PCIe/AER errors, blk_update_request errors, Buffer I/O errors, failed read/write commands, or other local disk error indicators.

So far, I do not see evidence of local disk retries or local storage faults during the incident window.

Kernel/update concern​

What concerns me is the recent Proxmox kernel cadence. In July I saw these kernel update events (date of email messages stating there were updates):
2026.07.08
proxmox-kernel-6.17 6.17.13-13 -> 6.17.13-15
proxmox-kernel-7.0 7.0.6-2 -> 7.0.14-4

2026.07.20
proxmox-kernel-6.17 6.17.13-16 -> 6.17.13-18
proxmox-kernel-7.0 7.0.14-4 -> 7.0.14-5

2026.07.28
proxmox-kernel-6.17 6.17.13-18 -> 6.17.13-19
proxmox-kernel-7.0 7.0.14-5 -> 7.0.14-6

2026.07.31
proxmox-kernel-6.17 6.17.13-19 -> 6.17.13-21
proxmox-kernel-7.0 7.0.14-6 -> 7.0.14-8
I understand that security fixes and urgent regressions can justify rapid releases. But from an operations perspective, four hypervisor kernel update events in a month, including two in the final week, is concerning. A bad hypervisor kernel has a large blast radius.

My current working hypothesis is:
PVE-side kernel/runtime/backup-path failure causing a host-level outage during QEMU/PBS backups
rather than:
PBS, TrueNAS, VM101, HAOS, USB passthrough, or local disk failure

Current mitigation​

I have rolled back/pinned to an older kernel (that is apparently no longer supported) and am watching whether the same backup workload stabilizes.

Questions​

1. Are there known issues in recent proxmox-kernel-7.0 builds involving host-level outages, blocked tasks, QEMU backup, PBS backup, storage I/O, or network stalls?
2. Since local and remote logging appear to stop when the issue begins, what is the most useful evidence capture method for this failure mode: netconsole, serial console, SysRq task dump, hung-task output, or something else?
3. Has anyone else seen PVE host-level outages during PBS backups after recent kernel updates?
4. Is there a recommended conservative kernel stream for production hosts right now, especially ZFS-root systems using PBS backups?

I am mainly looking for a disciplined way to collect useful evidence if this recurs. I have a gut-level hunch about this. I believe it is a problem with the Intel IGC drivers (I'm using a i225 NIC) and the kernel. I've researched this a bit and see some level of correlation back to Ubuntu. That said, gut-level hunches are about as good at a ten-cent stamp.

One thing I will say is that I'm not really impressed with Proxmox's release notes. They make problem isolation much harder. For example, I had to review each CVE and see if there might be some correlation to the CVE fixes to what is happening. This review is what led me to Ubuntu, as I did see that they are a downstream from Ubuntu.

Thanks!

 

[fabian]

the current upstream kernel release cadence is a few hundred commits a week, where often about half of them have a CVE assigned, with usually no information until a public exploit or PoC drops which of them are easily,reliably exploitable. this affects every Linux distribution out there.

there are no known reports matching your symptoms, so yeah, the best next bet would be to try to collect logs, and narrow down in which exact kernel version it regressed for your system.

 

[jbhorner]

the current upstream kernel release cadence is a few hundred commits a week, where often about half of them have a CVE assigned, with usually no information until a public exploit or PoC drops which of them are easily,reliably exploitable. this affects every Linux distribution out there.

there are no known reports matching your symptoms, so yeah, the best next bet would be to try to collect logs, and narrow down in which exact kernel version it regressed for your system.

Thank you for the reply. I did note the CVE cadence. My comment about release notes ties to them having information like:

• update submodules and patches to Ubuntu-7.0.0-28.28i3 <- Which?
• upstream stable changes <- Like?
• various CVE fixes

There isn't much in there to help one understand what was actually changed. CVEs are always are purposely vague for sure. The others, however, feel like there should be more substance.

When the issue arises the logging stops. There is nothing in the logs pre or post incident that indicate any problem. As much as I'd like to collect logs, the system simply stops any valuable logging. is there something I should do pre or post incident?

At this point I am on the older kernel. I have moved backups from the IGC ports to a non-IGC port for the purpose of isolation. I will restart with the newer kernel to see if that backs my theory that this is a kernel/Intel driver problem. I did see that there was a driver update. The IGC module differs between the currently running 6.17.13-21-pve kernel and the 7.0.14-8-pve kernel. `modinfo -k` shows different srcversion values:

6.17.13-21-pve: A2ADB5346D9DC1140279E95
7.0.14-8-pve: 636F2CE0326BE17364514C6

 

[fabian]

no logs on disk does not mean no logs generated. a serial or netconsole, or checking pstore can often help.

w.r.t. the changelogs:

* update submodules and patches to Ubuntu-7.0.0-28.28i3
    - upstream stable changes from 6.18.39 and 7.1.4
    - various CVE fixes

this means that we updated the kernel baseline from Ubuntu-7.0.0-28.28i2 (noted in the previous versions changelog entry) to Ubuntu-7.0.0-28.28i3. this baseline updated included the stable changes from 6.18.39 and 7.1.4 - they are too many to list in the changelog, it's roughly 600 commits. of those 600 commits, at the time our kernel upload was being prepared, 257 of those had a CVE associated (again, that's too many to sensibly list in the changelog, and the number is very often a lot lower at the time the upload is cut compared to just a week later - the CVE assignment happens after the fact).

if there are very prominent issues we try to call them out separately, but at this point you can assume that every kernel update likely contains fixes for exploitable issues.

 

[jbhorner]

no logs on disk does not mean no logs generated. a serial or netconsole, or checking pstore can often help.

w.r.t. the changelogs:

* update submodules and patches to Ubuntu-7.0.0-28.28i3
    - upstream stable changes from 6.18.39 and 7.1.4
    - various CVE fixes

this means that we updated the kernel baseline from Ubuntu-7.0.0-28.28i2 (noted in the previous versions changelog entry) to Ubuntu-7.0.0-28.28i3. this baseline updated included the stable changes from 6.18.39 and 7.1.4 - they are too many to list in the changelog, it's roughly 600 commits. of those 600 commits, at the time our kernel upload was being prepared, 257 of those had a CVE associated (again, that's too many to sensibly list in the changelog, and the number is very often a lot lower at the time the upload is cut compared to just a week later - the CVE assignment happens after the fact).

if there are very prominent issues we try to call them out separately, but at this point you can assume that every kernel update likely contains fixes for exploitable issues.

There was no output to the local console. Netconsole was also implemented and tested successfully, but no messages are emitted through netconsole (or on the physical console) when the failure occurs.

Thank you for the explanation regarding the updates.

At this point, I have moved critical VMs off this Proxmox host. I have personally spent many hours isolating variables, changing backup paths, moving workloads, testing netconsole/pstore, moving the installation to different hardware, and adding boot/storage redundancy.

Ultimately, what matters most to me is reliable infrastructure and effective support when something goes wrong. That was the primary reason I had no issue paying for the enterprise repository and support subscription. I also believe Proxmox is a solution that is worth a reasonable fee.

However, in this case, I do not feel the support process materially helped isolate or advance the issue. The problem remains unexplained, and I have had to reduce my dependency on this host as a result.

I do not know where the issue originated or what changed to expose it, but the experience has changed my view of the practical value of the support subscription.

 

[J-Rod]

What hardware do you have?
What is shown on the display when the system halts?
Have you run hardware stress tests? Does the system offer built-in hardware diagnostics?
If nothing is logged to disk, the single, consumer-grade SSD could be locking up, overheating or intermittently faulty.

 

[fabian]

However, in this case, I do not feel the support process materially helped isolate or advance the issue. The problem remains unexplained, and I have had to reduce my dependency on this host as a result.

I do not know where the issue originated or what changed to expose it, but the experience has changed my view of the practical value of the support subscription.

this is not the enterprise support portal, but the community forum. if you want enterprise support, open an enterprise support ticket.

 

[jbhorner]

What hardware do you have?
What is shown on the display when the system halts?
Have you run hardware stress tests? Does the system offer built-in hardware diagnostics?
If nothing is logged to disk, the single, consumer-grade SSD could be locking up, overheating or intermittently faulty.

Hi thanks for the reply.

It's a Lenovo i7 M920 with 128GB of RAM. I did a RAM stress test and ran diagnostics on the NVME which came up clean. When the system ceases to respond (it doesn't "halt" in the usual sense), there are no messages on the screen. At the console I can press enter and still see the # prompt (if I remain logged in prior to the problem). If I enter any commands, it doesn't respond. It is a head-scratcher to be sure.

It could be (and most likely is) a hardware fault somewhere. The odd thing is that it was working flawlessly until I updated to the newest kernel and software updates. Literally the next day backups commenced. It's a wild coincidence if it is hardware. But then again, the reboot may have triggered something with the hardware. I would have expected some level of a diagnostic message. I've had other Linux drive/SSD/NVME failures in the past (long ago) and there were messages complaining about an inability to write to storage.

I've since moved the NVME to a different box. I also implemented ZFS mirroring. I'll see how that goes.

 

[jbhorner]

this is not the enterprise support portal, but the community forum. if you want enterprise support, open an enterprise support ticket.

Interesting. When I'd sent an email to the support email address I was told that the level of support available for that subscription was forum-based support, and was assured it was very good support. There was no link in the reply.

 

[Johannes S]

Then you propably have the community subscription. It's main point is access to the enterprise repo and removal of the nag screen. The enterprise repo gets updated packages after the others so it's more unlikely to run into bugs. Please note "unlikely": While packages from Debian stable or the stuff from the Proxmox Developers get most quirks ironed out that way this is not really possible with the kernel. Either you get the new kernel with its fixes or you are more likely to get attacked due to an open exploit. Yes, this sucks, welcome to the year 2026 in the age of AI assisted security research.

 

[leesteken]

Interesting. When I'd sent an email to the support email address I was told that the level of support available for that subscription was forum-based support, and was assured it was very good support.

Support here from volunteering strangers on the internet is indeed pretty good. Proxmox staff also visits but I feel that their time here is limited and sometimes spread thin, so I also think about them as volunteers.
EDIT: If you need guaranteed response times and the like, you'll need a more expensive subscription with a certain number of tickets per year and/or use a Proxmox partner (in your area).