Hello
I've a serious problem with a PVE/PBS Node. 2 Windows VM that continuously log problems with Storage and sometimes they crash.
Hardware/Software:
Source: vioscsi Event 129 - Reset to device, \Device\RaidPort1, was issued.
Source: storahci Event 129 - Reset to device, \Device\RaidPort0, was issued.
Source: disk Event 153 - The IO operation ... was retried.
Source: ESENT Event 508 - ...took an abnormally long time (25 seconds) to be serviced by the OS.
Sometimes followed by a BSOD:
0x000000A0 INTERNAL_POWER_ERROR
0x0000007A KERNEL_DATA_INPAGE_ERROR
Host Logs : On the Proxmox host, at the same time, dmesg/journal shows ZFS txg_sync and io_uring workers stuck in D-state for minutes:
INFO: task txg_sync:2664 blocked for more than 245 seconds.
Tainted: P O 6.8.12-8-pve #1
task:txg_sync state
stack:0 pid:2664
Call Trace:
__schedule+0x42b/0x1500
schedule_timeout+0x95/0x170
io_schedule_timeout+0x51/0x80
__cv_timedwait_io+0x19/0x30 [spl]
zio_wait+0x13a/0x2c0 [zfs]
dsl_pool_sync+0xce/0x4e0 [zfs]
spa_sync+0x578/0x1050 [zfs]
txg_sync_thread+0x207/0x3a0 [zfs]
...
INFO: task iou-wrk-4306:568360 blocked for more than 122 seconds.
task:iou-wrk-4306 state
Call Trace:
io_schedule+0x46/0x80
folio_wait_writeback+0x2b/0xa0
__filemap_fdatawait_range+0x90/0x100
blkdev_fsync+0x39/0x70
io_fsync+0x3d/0x60
io_wq_submit_work+0xe2/0x360
...
Repeated multiple times over ~50 minutes on the same evening, with txg_sync blocking for 122–245+ seconds each time
The Zpool's appear OK
I've ended my ideas.
Can you help me ?
I've a serious problem with a PVE/PBS Node. 2 Windows VM that continuously log problems with Storage and sometimes they crash.
Hardware/Software:
- Dell R440, PERC H730P (Non-RAID mode / true HBA mode confirmed via iDRAC — Controller Mode: HBA)
- Controller firmware: 25.5.9.0001 (confirmed latest available from Dell)
- Proxmox VE 8.3.4, kernel 6.8.12-8-pve
- 2 ZFS pools on this node:
- sasvol: RAIDZ1, 4x SAS disks
- ssdvol: mirror, 2x SATA SSD (Samsung MZ7LM960HCHP)
- rpool: mirror, 2x NVMe (boot)
- PBS also running on the same node, backing up to a NAS mounted via NFS
- 2x Windows Server 2022 VMs (virtio-scsi), latest virtio drivers (100.101.104.28500)
Source: vioscsi Event 129 - Reset to device, \Device\RaidPort1, was issued.
Source: storahci Event 129 - Reset to device, \Device\RaidPort0, was issued.
Source: disk Event 153 - The IO operation ... was retried.
Source: ESENT Event 508 - ...took an abnormally long time (25 seconds) to be serviced by the OS.
Sometimes followed by a BSOD:
0x000000A0 INTERNAL_POWER_ERROR
0x0000007A KERNEL_DATA_INPAGE_ERROR
Host Logs : On the Proxmox host, at the same time, dmesg/journal shows ZFS txg_sync and io_uring workers stuck in D-state for minutes:
INFO: task txg_sync:2664 blocked for more than 245 seconds.
Tainted: P O 6.8.12-8-pve #1
task:txg_sync state
stack:0 pid:2664Call Trace:
__schedule+0x42b/0x1500
schedule_timeout+0x95/0x170
io_schedule_timeout+0x51/0x80
__cv_timedwait_io+0x19/0x30 [spl]
zio_wait+0x13a/0x2c0 [zfs]
dsl_pool_sync+0xce/0x4e0 [zfs]
spa_sync+0x578/0x1050 [zfs]
txg_sync_thread+0x207/0x3a0 [zfs]
...
INFO: task iou-wrk-4306:568360 blocked for more than 122 seconds.
task:iou-wrk-4306 state
Call Trace:
io_schedule+0x46/0x80
folio_wait_writeback+0x2b/0xa0
__filemap_fdatawait_range+0x90/0x100
blkdev_fsync+0x39/0x70
io_fsync+0x3d/0x60
io_wq_submit_work+0xe2/0x360
...
Repeated multiple times over ~50 minutes on the same evening, with txg_sync blocking for 122–245+ seconds each time
The Zpool's appear OK
I've ended my ideas.
Can you help me ?