[SOLVED] Crash VM with io-error

[carcella]

Good Evening,
som month ago I installed little Proxmox VE to testing affidability and functionality.
I created 2 vm one Windows 2019 Server and one Linux server.
About two month the VMs crasched with io-error yellow Triangle. I dowgrade the kernel version to 5.11 but it's same.
Can you help me?

 

[mira]

Please provide the output of pveversion -v and qm config <VMID> for one of the affected VMs.
Also check the syslog/journal for any errors regarding the disk those VMs are on.

 

[carcella]

This is pveversion:
root@pve:~# pveversion -v
proxmox-ve: 7.3-1 (running kernel: 5.11.22-7-pve)
pve-manager: 7.3-3 (running version: 7.3-3/c3928077)
pve-kernel-5.15: 7.2-14
pve-kernel-helper: 7.2-14
pve-kernel-5.13: 7.1-9
pve-kernel-5.15.74-1-pve: 5.15.74-1
pve-kernel-5.15.64-1-pve: 5.15.64-1
pve-kernel-5.13.19-6-pve: 5.13.19-15
pve-kernel-5.13.19-2-pve: 5.13.19-4
pve-kernel-5.11.22-7-pve: 5.11.22-12
ceph-fuse: 15.2.15-pve1
corosync: 3.1.7-pve1
criu: 3.15-1+pve-1
glusterfs-client: 9.2-1
ifupdown2: 3.1.0-1+pmx3
ksm-control-daemon: 1.4-1
libjs-extjs: 7.0.0-1
libknet1: 1.24-pve2
libproxmox-acme-perl: 1.4.2
libproxmox-backup-qemu0: 1.3.1-1
libpve-access-control: 7.2-5
libpve-apiclient-perl: 3.2-1
libpve-common-perl: 7.3-1
libpve-guest-common-perl: 4.2-3
libpve-http-server-perl: 4.1-5
libpve-storage-perl: 7.3-1
libspice-server1: 0.14.3-2.1
lvm2: 2.03.11-2.1
lxc-pve: 5.0.0-3
lxcfs: 4.0.12-pve1
novnc-pve: 1.3.0-3
proxmox-backup-client: 2.2.7-1
proxmox-backup-file-restore: 2.2.7-1
proxmox-mini-journalreader: 1.3-1
proxmox-offline-mirror-helper: 0.5.0-1
proxmox-widget-toolkit: 3.5.3
pve-cluster: 7.3-1
pve-container: 4.4-2
pve-docs: 7.3-1
pve-edk2-firmware: 3.20220526-1
pve-firewall: 4.2-7
pve-firmware: 3.5-6
pve-ha-manager: 3.5.1
pve-i18n: 2.8-1
pve-qemu-kvm: 7.1.0-4
pve-xtermjs: 4.16.0-1
qemu-server: 7.3-1
smartmontools: 7.2-pve3
spiceterm: 3.2-2
swtpm: 0.8.0~bpo11+2
vncterm: 1.7-1
zfsutils-linux: 2.1.6-pve1
root@pve:~#

 

[carcella]

root@pve:~# qm config 100
boot: order=ide0;ide2;net0
cores: 2
description: administrator / AzPan!c02020
ide0: local-lvm:vm-100-disk-0,size=62G
ide2: cdrom,media=cdrom
machine: pc-i440fx-6.1
memory: 6144
meta: creation-qemu=6.1.0,ctime=1642289339
name: Win19srv
net0: e1000=A2:4F:4C:1D:A12,bridge=vmbr0,firewall=1
numa: 0
ostype: win10
parent: win2019_18122022
scsihw: virtio-scsi-pci
smbios1: uuid=ecd0cefd-2aa5-45c1-b123-f3c00946843d
sockets: 3
usb0: spice,usb3=1
vga: qxl
vmgenid: 284cb1d3-c3bc-4298-8495-4606211b5504

 

[carcella]

I attach my syslog, the problem was about Nov 28 at 01.00 a.m. , but I can't find any disk problem.
Thank's for help me..

 

[mira]

It seems the syslog didn't make it.
Can you try attaching it again?

 

[carcella]

is it....
thank's

 

[carcella]

It seems the syslog didn't make it.
Can you try attaching it again?

 

[mira]

Thank you for the syslog!

It would seem that your `local-lvm` storage is simply full:

Nov 28 12:38:16 pve kernel: [864872.499140] device-mapper: thin: 253:4: switching pool to out-of-data-space (queue IO) mode
Nov 28 12:38:24 pve lvm[445]: WARNING: Thin pool pve-data-tpool data is now 100.00% full.
Nov 28 12:39:18 pve kernel: [864934.283889] device-mapper: thin: 253:4: switching pool to out-of-data-space (error IO) mode

Did you overprovision it or create snapshots?
But your VMs can no longer write since there's no space available and you'll see the `io-error` status then.

Free up some space on your `local-lvm` storage and it should be fine again. And make sure to always have some spare space, since running out of space can corrupt the filesystems inside the VMs.

 

[carcella]

Thank you for the syslog!

It would seem that your `local-lvm` storage is simply full:

Nov 28 12:38:16 pve kernel: [864872.499140] device-mapper: thin: 253:4: switching pool to out-of-data-space (queue IO) mode
Nov 28 12:38:24 pve lvm[445]: WARNING: Thin pool pve-data-tpool data is now 100.00% full.
Nov 28 12:39:18 pve kernel: [864934.283889] device-mapper: thin: 253:4: switching pool to out-of-data-space (error IO) mode

Did you overprovision it or create snapshots?
But your VMs can no longer write since there's no space available and you'll see the `io-error` status then.

Free up some space on your `local-lvm` storage and it should be fine again. And make sure to always have some spare space, since running out of space can corrupt the filesystems inside the VMs.

Auch!!!
I did some snapshot and this increase size more time.....
How I can change snapshot disk destination??

 

[mira]

You can't, those are part of the storage itself.
But you can remove some of those and it should free up space again.

 

[carcella]

OK, Thank You very much for your disponibility!!!