SQL Server 2022 database corruption

C

carver.it

New Member
Aug 12, 2026
8
0
1
Some background.

PVE 9.1.7
Windows 2022 with SQL Server 2022
Have got 3 x hosts, ZFS data stores with replication of VM to another host currently hourly
PBS running daily snapshot backup to a drive on third cluster node
Not currently got support arrangement with ProMox, on the list to get signed for and done!
PBS drive became full and so backups not running properly
Cove backup within the Windows OS also multiple times daily. Currently 4 hourly
Each night getting a disk I/O error within Windows think related to when backup was kicking in --> "Error detail: ID=153, Source=Disk, Message: The IO operation at logical block address 0x4ddf6e8 for Disk 0 (PDO name: \Device\0000001f) was retried." few of those over maybe 1 second in total.

SQL DBCC checks complaining of some corruption few weeks ago and ended up restoring from BAK files and replaying logs and has seemed OK since then and Cove backups have been reporting 100%
Yesterday consistency check failed again and issues with corrupt log files.

Physical drives are stable
RAID controller stable
ZFS drives stable
Windows VM no complaints except the , CHKDSK when run before was stable

I am looking around here but can't seem to search nicely yet for other SQL best practice with ProxMox beyond https://192.168.10.11:8006/pve-docs/chapter-qm.html#qm_qga_fsfreeze etc.


Any advice appreciated!

Steve
 
One long thread, trying to work out relevant parts still so if there is particular thing pointing at in there? Coming from a VMWARE history where things mainly "just work" and had someone set this up for us.

I have turned off PBS backups for now. Replication is still on running hourly to another host and shows up in SQL logs for instance as a <1 second freeze.

As PBS which wasn't intended to be a permanent solution it was only an "extra" while things were being changed had filled it's drive and erroring I presume it was causing the VM to wait while it took ages, failed and the I/O to retry.

This is config of one of the machines.

1786544158824.png
 
When you perform a backup with PBS, PVE/QEMU integrates the backup operation into the VM's I/O path. Data that is being modified while the backup is running has to be coordinated with the backup process. Depending on the storage, network, PBS performance, and workload, this can introduce additional I/O pressure and potentially affect VM performance.

There is also an option for staging backup data on local storage rather than sending it directly to PBS, which can reduce the impact on the VM when the PBS connection or backup target is slow. However, this obviously does not protect you from problems caused by the PBS datastore itself being full.

Theoretically, a failed backup should not affect the health of the VM. However, if the underlying storage runs out of space, applications at higher layers can behave unpredictably. Running out of storage is never a good situation.

So, if disabling the PBS backup resolves your I/O issue, that is a indication that the backup path may be contributing to the problem.

The thread I referenced deals with IO issues in Windows that can happen in various conditions, including backups. However, the issue discussed there is scoped to the VIRTIO windows driver, maintained by RedHat/IBM (but universally used in QEMU ecosystem).

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox
 
Thank you. Yes I have disabled that and will see if it helps Meanwhile we have attempts of SQL restore of one DB and not sure whether we can actually trust the I/O to work. It suggested various driver version errors.

The backup is only to storage within the cluster not to internet but only 1Gbit server interconnect.

Hopefully then if we can stablise SQL the problem may have been caused by the Backup only.
 
Thanks. Two HDD RAID6 arrays with slower and faster discs on each host. ZFS on top of those with each vm replicating to another host on repurposed VMware hardware
 
ZFS on top of hardware RAID6 means ZFS cannot see or manage the individual disks. It can detect checksum errors, but with a single RAID LUN as a vdev it generally cannot repair corrupted data from another ZFS replica, while the RAID controller itself has no knowledge of ZFS checksums.

Are those two RAID6 arrays separate ZFS pools, or are both RAID6 LUNs vdevs in the same ZFS pool?
 
  • Like
Reactions: Kingneutron
One pool per array. Hardware side is off course monitored at iDRAC level due any reports from controllers and physical and virtual discs.
 
If you're going to use ZFS, the FIRST thing you need to do is implement it properly. It's well known for years that running ZFS on top of hardware RAID is bad, who set that mess up?

Backup everything and rearchitect your ZFS pool using professional advice. I suspect the "SQL database corruption" may fix itself, but at least you'll have fixed one major problem.
 
You must log in or register to reply here.
Top Bottom
Back