Reset to device, \Device\RaidPort2, was issued

[davemcl]

I have a SQL Server that started spitting out these errors in the event log - MS SQL completely locked up and required a hard reboot

Source: vioscsi
Event ID: 129
Reset to device, \Device\RaidPort2, was issued.

At the first appearance of the above event I got this in the pve node syslog:

Apr 24 07:13:40 pve01 QEMU[1258450]: kvm: Desc next is 3

IO wasnt particularly high at the time and there dont seem to be any other logs with errors/warnings.

Windows Server 2019
VirtIO drivers are 0.1.229

Package Versions
proxmox-ve: 7.4-1 (running kernel: 6.1.15-1-pve)
pve-manager: 7.4-3 (running version: 7.4-3/9002ab8a)
pve-kernel-5.15: 7.4-1
pve-kernel-6.1: 7.3-6
pve-kernel-6.1.15-1-pve: 6.1.15-1
pve-kernel-5.15.104-1-pve: 5.15.104-2
pve-kernel-5.15.102-1-pve: 5.15.102-1
pve-kernel-5.15.74-1-pve: 5.15.74-1
ceph-fuse: 15.2.17-pve1
corosync: 3.1.7-pve1
criu: 3.15-1+pve-1
glusterfs-client: 9.2-1
ifupdown2: 3.1.0-1+pmx3
ksm-control-daemon: 1.4-1
libjs-extjs: 7.0.0-1
libknet1: 1.24-pve2
libproxmox-acme-perl: 1.4.4
libproxmox-backup-qemu0: 1.3.1-1
libproxmox-rs-perl: 0.2.1
libpve-access-control: 7.4-2
libpve-apiclient-perl: 3.2-1
libpve-common-perl: 7.3-4
libpve-guest-common-perl: 4.2-4
libpve-http-server-perl: 4.2-3
libpve-rs-perl: 0.7.5
libpve-storage-perl: 7.4-2
libspice-server1: 0.14.3-2.1
lvm2: 2.03.11-2.1
lxc-pve: 5.0.2-2
lxcfs: 5.0.3-pve1
novnc-pve: 1.4.0-1
proxmox-backup-client: 2.4.1-1
proxmox-backup-file-restore: 2.4.1-1
proxmox-kernel-helper: 7.4-1
proxmox-mail-forward: 0.1.1-1
proxmox-mini-journalreader: 1.3-1
proxmox-widget-toolkit: 3.6.5
pve-cluster: 7.3-3
pve-container: 4.4-3
pve-docs: 7.4-2
pve-edk2-firmware: 3.20230228-2
pve-firewall: 4.3-1
pve-firmware: 3.6-4
pve-ha-manager: 3.6.0
pve-i18n: 2.12-1
pve-qemu-kvm: 7.2.0-8
pve-xtermjs: 4.16.0-1
qemu-server: 7.4-3
smartmontools: 7.2-pve3
spiceterm: 3.2-2
swtpm: 0.8.0~bpo11+3
vncterm: 1.7-1
zfsutils-linux: 2.1.9-pve1

qm config

agent: 1
bios: ovmf
boot: order=scsi0;ide2;net0
cores: 16
cpu: Skylake-Server
description: #### Windows Server 2019 ####
efidisk0: SSD-R10:vm-102-disk-0,efitype=4m,pre-enrolled-keys=1,size=528K
ide2: none,media=cdrom
machine: pc-q35-7.1
memory: 49152
meta: creation-qemu=7.1.0,ctime=1676025539
name: LUNA
net0: virtio=0E:AC:28:EA:A7:20,bridge=vmbr0,firewall=1,tag=25
numa: 0
onboot: 1
ostype: win10
scsi0: SSD-R10:vm-102-disk-1,cache=writeback,discard=on,iothread=1,size=80G
scsi1: SSD-R10:vm-102-disk-2,cache=writeback,discard=on,iothread=1,size=320G
scsi2: SSD-R10:vm-102-disk-3,cache=writeback,discard=on,iothread=1,size=128G
scsi3: SSD-R10:vm-102-disk-4,cache=writeback,discard=on,iothread=1,size=64G
scsihw: virtio-scsi-single
smbios1: uuid=6fd7bb99-ef24-40e6-92b9-f4e15da8fc8f
sockets: 1
tablet: 0
tags:

Hardware is Dell R740 - Intel Gold Xeon 6152

 

[davemcl]

This happened again on a different Windows Server VM (2016) / SQL 2019.
Host is Dell R640 this time - Intel Gold Xeon 6136

Host syslog reports
Jun 09 19:09:50 pve03 QEMU[2389459]: kvm: virtio: zero sized buffers are not allowed

Then the VM just logs lots of

Event 129, Source vioscsi

Reset to device, \Device\RaidPort2, was issued.

At this point SQL is unusable and the vm requires a hard reset.

Ive rolled back to kernel 5.15.107-2-pve from 6.1 to see if that will make a difference.

Any ideas?

 

[fiona]

Please don't double post, that just makes it hard to keep track of things. Discussion will be continued here: https://forum.proxmox.com/threads/what-is-kvm-desc-next-is-3-indicative-of.128739/