Windows VMs crashing / BSOD due to storage stalls

P

paolone

Member
May 6, 2024
31
2
13
Hello

I've a serious problem with a PVE/PBS Node. 2 Windows VM that continuously log problems with Storage and sometimes they crash.

Hardware/Software:
  • Dell R440, PERC H730P (Non-RAID mode / true HBA mode confirmed via iDRAC — Controller Mode: HBA)
  • Controller firmware: 25.5.9.0001 (confirmed latest available from Dell)
  • Proxmox VE 8.3.4, kernel 6.8.12-8-pve
  • 2 ZFS pools on this node:
    • sasvol: RAIDZ1, 4x SAS disks
    • ssdvol: mirror, 2x SATA SSD (Samsung MZ7LM960HCHP)
    • rpool: mirror, 2x NVMe (boot)
  • PBS also running on the same node, backing up to a NAS mounted via NFS
  • 2x Windows Server 2022 VMs (virtio-scsi), latest virtio drivers (100.101.104.28500)
Symptoms:Windows guests intermittently log:

Source: vioscsi Event 129 - Reset to device, \Device\RaidPort1, was issued.
Source: storahci Event 129 - Reset to device, \Device\RaidPort0, was issued.
Source: disk Event 153 - The IO operation ... was retried.
Source: ESENT Event 508 - ...took an abnormally long time (25 seconds) to be serviced by the OS.

Sometimes followed by a BSOD:

0x000000A0 INTERNAL_POWER_ERROR
0x0000007A KERNEL_DATA_INPAGE_ERROR
Host Logs : On the Proxmox host, at the same time, dmesg/journal shows ZFS txg_sync and io_uring workers stuck in D-state for minutes:

INFO: task txg_sync:2664 blocked for more than 245 seconds.
Tainted: P O 6.8.12-8-pve #1
task:txg_sync state:D stack:0 pid:2664
Call Trace:
__schedule+0x42b/0x1500
schedule_timeout+0x95/0x170
io_schedule_timeout+0x51/0x80
__cv_timedwait_io+0x19/0x30 [spl]
zio_wait+0x13a/0x2c0 [zfs]
dsl_pool_sync+0xce/0x4e0 [zfs]
spa_sync+0x578/0x1050 [zfs]
txg_sync_thread+0x207/0x3a0 [zfs]
...

INFO: task iou-wrk-4306:568360 blocked for more than 122 seconds.
task:iou-wrk-4306 state:D
Call Trace:
io_schedule+0x46/0x80
folio_wait_writeback+0x2b/0xa0
__filemap_fdatawait_range+0x90/0x100
blkdev_fsync+0x39/0x70
io_fsync+0x3d/0x60
io_wq_submit_work+0xe2/0x360
...

Repeated multiple times over ~50 minutes on the same evening, with txg_sync blocking for 122–245+ seconds each time

The Zpool's appear OK

I've ended my ideas.
Can you help me ?
 
Hello @paolone

thanks for posting on the forum!

There were some issues with LSI based controllers in the past but they should only affect kernel 6.17.
Still most of them were fixed with kernel 7 which is default for PVE 9, so maybe you could try updating the server a little bit earlier.
Since PVE 8 is EOL this August [1] this should be on your roadmap anyway.
See the upgrade guides for PVE [2] and PBS [3] accordingly.

Furthermore it is recommended to use version 0.1.271 of VirtIO drivers, since there are known issues with the current 0.1.285 [4].
They are mostly affecting Windows Server 2025, but there were also issues observed on Win Server 2022.

Best regards
Jonas

[1] https://pve.proxmox.com/pve-docs/chapter-pve-faq.html#faq-support-table
[2] https://pve.proxmox.com/wiki/Upgrade_from_8_to_9
[3] https://pbs.proxmox.com/wiki/Upgrade_from_3_to_4
[4] https://pve.proxmox.com/wiki/Windows_VirtIO_Drivers
 
I can personally vouch for version 266 as I've been running that with some 2022 VMs (domain controllers) for a couple years and have zero issues to report.
 
Hello

Updated node to last 9.2.5 but in the volume with sas disk I've a big latency in RAID Z1 (raid5) sometimes it arrives to 500ms and the vm crash.
Disks are ok, Volume ok.
Downgrade Virtio to 0.1.271 .

Do you have other suggestion?

Best regards
 
paolone said:
Disks are ok, Volume ok.
Click to expand...
Controller OK? Try swapping it out for another to test. I believe you can also check the controller logs themselves via iDRAC & perform HW diagnostics.
What is the history of this controller?
 
Hello
gfngfn256 said:
Controller OK? Try swapping it out for another to test. I believe you can also check the controller logs themselves via iDRAC & perform HW diagnostics.
What is the history of this controller?
Click to expand...
This is a H730P updated no errors on idrac (apparently).
This controller was in RAID mode than when I've installed Proxmox in HBA Mode
 
I would expect ZFS in this configuration to perform very poorly, and even worse since it doesn't appear you have a special device attached to the HDD array. Assuming the H730P has a battery-backed cache, I would suggest setting up the RAID array on the controller and not using ZFS to handle the RAID features. Enable write-back on the array and disable the disk cache. Writes will now be acknowledged as soon as they land in the controller cache, instead of after they are written to disk, and this should greatly increase performance.
 
  • Like
Reactions: leesteken
Hello
H730P
J-Rod said:
I would expect ZFS in this configuration to perform very poorly, and even worse since it doesn't appear you have a special device attached to the HDD array. Assuming the H730P has a battery-backed cache, I would suggest setting up the RAID array on the controller and not using ZFS to handle the RAID features. Enable write-back on the array and disable the disk cache. Writes will now be acknowledged as soon as they land in the controller cache, instead of after they are written to disk, and this should greatly increase performance.
Click to expand...
H730P is on HBA mode. I think no cache.
Forgive me, but I'm not very experienced: do you think the problem is the controller (I repeat, it's in HBA mode and I can see all the individual disks on the node) or the RAIDZ1?

Best regards
 
You must log in or register to reply here.
Top Bottom
Back