[SOLVED] VM state goes to "io-error" - due to qemu-server <9.0.8 - bug with qcow and LVM

Jul 11, 2025
6
0
1
Hey,

on a shared storage with hundreds of VMs, I have one VM that seem to constantly switch to the "io-error" state.

My setup:

I have a non-thin LVM, that is connected via FibreChannel to the host.
The physical volume is 8TB big. Only ~3TB is assigned to VMs, so I don't think it's an issue of a full host disk.
Also, there are hundreds of other VMs on the same storage - they don't experience any issues.

What I checked so far:

echo '{"execute": "qmp_capabilities"}{"execute": "query-block", "arguments": {}}' | socat - /var/run/qemu-server/107.qmp | jq .
{
"QMP": {
"version": {
"qemu": {
"micro": 2,
"minor": 0,
"major": 10
},
"package": "pve-qemu-kvm_10.0.2-4"
},
"capabilities": []
}
}
{
"return": {}
}
{
"return": [
{
"io-status": "ok",
"device": "",
"locked": true,
"removable": true,
"inserted": {
"iops_rd": 0,
"detect_zeroes": "off",
"active": true,
"image": {
"backing-image": {
"virtual-size": 8388608,
"filename": "/dev/storage1-preview-1/vm-107-cloudinit",
"format": "raw",
"actual-size": 0,
"dirty-flag": false
},
"virtual-size": 8388608,
"filename": "json:{\"throttle-group\": \"throttle-drive-ide2\", \"driver\": \"throttle\", \"file\": {\"driver\": \"raw\", \"file\": {\"driver\": \"host_device\", \"filename\": \"/dev/storage1-preview-1/vm-107-cloudinit\"}}}",
"format": "throttle",
"actual-size": 0,
"dirty-flag": false
},
"iops_wr": 0,
"ro": true,
"children": [
{
"node-name": "fad7a45e2bca0b0efb685abc8d9b2ef",
"child": "file"
}
],
"node-name": "drive-ide2",
"backing_file_depth": 1,
"drv": "throttle",
"iops": 0,
"bps_wr": 0,
"write_threshold": 0,
"encrypted": false,
"bps": 0,
"bps_rd": 0,
"cache": {
"no-flush": false,
"direct": false,
"writeback": true
},
"file": "json:{\"throttle-group\": \"throttle-drive-ide2\", \"driver\": \"throttle\", \"file\": {\"driver\": \"raw\", \"file\": {\"driver\": \"host_device\", \"filename\": \"/dev/storage1-preview-1/vm-107-cloudinit\"}}}"
},
"qdev": "ide2",
"tray_open": false,
"type": "unknown"
},
{
"io-status": "nospace",
"device": "",
"locked": false,
"removable": false,
"inserted": {
"iops_rd": 0,
"detect_zeroes": "on",
"active": true,
"image": {
"backing-image": {
"virtual-size": 10737418240,
"filename": "/dev/storage1-preview-1/vm-107-disk-0.qcow2",
"cluster-size": 65536,
"format": "qcow2",
"actual-size": 0,
"format-specific": {
"type": "qcow2",
"data": {
"compat": "1.1",
"compression-type": "zlib",
"lazy-refcounts": false,
"refcount-bits": 16,
"corrupt": false,
"extended-l2": false
}
},
"dirty-flag": false
},
"virtual-size": 10737418240,
"filename": "json:{\"throttle-group\": \"throttle-drive-scsi0\", \"driver\": \"throttle\", \"file\": {\"driver\": \"qcow2\", \"file\": {\"driver\": \"host_device\", \"filename\": \"/dev/storage1-preview-1/vm-107-disk-0.qcow2\"}}}",
"cluster-size": 65536,
"format": "throttle",
"actual-size": 0,
"dirty-flag": false
},
"iops_wr": 0,
"ro": false,
"children": [
{
"node-name": "f8b934fb1f6846ccfef3594d3efebdb",
"child": "file"
}
],
"node-name": "drive-scsi0",
"backing_file_depth": 1,
"drv": "throttle",
"iops": 0,
"bps_wr": 0,
"write_threshold": 0,
"encrypted": false,
"bps": 0,
"bps_rd": 0,
"cache": {
"no-flush": false,
"direct": false,
"writeback": true
},
"file": "json:{\"throttle-group\": \"throttle-drive-scsi0\", \"driver\": \"throttle\", \"file\": {\"driver\": \"qcow2\", \"file\": {\"driver\": \"host_device\", \"filename\": \"/dev/storage1-preview-1/vm-107-disk-0.qcow2\"}}}"
},
"qdev": "scsi0",
"type": "unknown"
},
{
"io-status": "ok",
"device": "",
"locked": false,
"removable": false,
"inserted": {
"iops_rd": 0,
"detect_zeroes": "on",
"active": true,
"image": {
"backing-image": {
"virtual-size": 4294967296,
"filename": "/dev/storage1-preview-1/vm-107-disk-2.qcow2",
"cluster-size": 65536,
"format": "qcow2",
"actual-size": 0,
"format-specific": {
"type": "qcow2",
"data": {
"compat": "1.1",
"compression-type": "zlib",
"lazy-refcounts": false,
"refcount-bits": 16,
"corrupt": false,
"extended-l2": false
}
},
"dirty-flag": false
},
"virtual-size": 4294967296,
"filename": "json:{\"throttle-group\": \"throttle-drive-scsi2\", \"driver\": \"throttle\", \"file\": {\"driver\": \"qcow2\", \"file\": {\"driver\": \"host_device\", \"filename\": \"/dev/storage1-preview-1/vm-107-disk-2.qcow2\"}}}",
"cluster-size": 65536,
"format": "throttle",
"actual-size": 0,
"dirty-flag": false
},
"iops_wr": 0,
"ro": false,
"children": [
{
"node-name": "f842cc8aee5c625dfe52be2caf346a0",
"child": "file"
}
],
"node-name": "drive-scsi2",
"backing_file_depth": 1,
"drv": "throttle",
"iops": 0,
"bps_wr": 0,
"write_threshold": 0,
"encrypted": false,
"bps": 0,
"bps_rd": 0,
"cache": {
"no-flush": false,
"direct": false,
"writeback": true
},
"file": "json:{\"throttle-group\": \"throttle-drive-scsi2\", \"driver\": \"throttle\", \"file\": {\"driver\": \"qcow2\", \"file\": {\"driver\": \"host_device\", \"filename\": \"/dev/storage1-preview-1/vm-107-disk-2.qcow2\"}}}"
},
"qdev": "scsi2",
"type": "unknown"
},
{
"io-status": "ok",
"device": "",
"locked": false,
"removable": false,
"inserted": {
"iops_rd": 0,
"detect_zeroes": "on",
"active": true,
"image": {
"backing-image": {
"virtual-size": 54760833024,
"filename": "/dev/storage1-preview-1/vm-107-disk-1.qcow2",
"cluster-size": 65536,
"format": "qcow2",
"actual-size": 0,
"format-specific": {
"type": "qcow2",
"data": {
"compat": "1.1",
"compression-type": "zlib",
"lazy-refcounts": false,
"refcount-bits": 16,
"corrupt": false,
"extended-l2": false
}
},
"dirty-flag": false
},
"virtual-size": 54760833024,
"filename": "json:{\"throttle-group\": \"throttle-drive-scsi3\", \"driver\": \"throttle\", \"file\": {\"driver\": \"qcow2\", \"file\": {\"driver\": \"host_device\", \"filename\": \"/dev/storage1-preview-1/vm-107-disk-1.qcow2\"}}}",
"cluster-size": 65536,
"format": "throttle",
"actual-size": 0,
"dirty-flag": false
},
"iops_wr": 0,
"ro": false,
"children": [
{
"node-name": "fa1c283d5f1584b679fce04d2b3145c",
"child": "file"
}
],
"node-name": "drive-scsi3",
"backing_file_depth": 1,
"drv": "throttle",
"iops": 0,
"bps_wr": 0,
"write_threshold": 0,
"encrypted": false,
"bps": 0,
"bps_rd": 0,
"cache": {
"no-flush": false,
"direct": false,
"writeback": true
},
"file": "json:{\"throttle-group\": \"throttle-drive-scsi3\", \"driver\": \"throttle\", \"file\": {\"driver\": \"qcow2\", \"file\": {\"driver\": \"host_device\", \"filename\": \"/dev/storage1-preview-1/vm-107-disk-1.qcow2\"}}}"
},
"qdev": "scsi3",
"type": "unknown"
}
]
}

lvdisplay /dev/storage1-preview-1/vm-107-disk-0.qcow2
root@n-1:~# lvdisplay /dev/storage1-preview-1/vm-107-disk-0.qcow2
--- Logical volume ---
LV Path /dev/storage1-preview-1/vm-107-disk-0.qcow2
LV Name vm-107-disk-0.qcow2
VG Name storage1-preview-1
LV UUID 3tYkgm-C0Af-2YKD-6Dfo-Wtph-3BZ0-dPJ7f3
LV Write Access read/write
LV Creation host, time n-1, 2025-07-23 14:50:14 +0200
LV Status available
# open 1
LV Size 10.00 GiB
Current LE 2561
Segments 1
Allocation inherit
Read ahead sectors auto
- currently set to 16384
Block device 252:7

cat /proc/partitions (inside the VM)
major minor #blocks name

8 0 10485760 sda
8 1 1024 sda1
8 2 10482688 sda2
8 16 4194304 sdb
8 32 53477376 sdc
8 33 52427759 sdc1
11 0 8192 sr0

root@n-1:~# pvdisplay /dev/mapper/storage1-preview-1
--- Physical volume ---
PV Name /dev/mapper/pure3_b-preview-1
VG Name pure3_b-preview-1
PV Size 8.00 TiB / not usable 4.00 MiB
Allocatable yes
PE Size 4.00 MiB
Total PE 2097151
Free PE 1318762
Allocated PE 778389
PV UUID jJCHNo-NHUf-tjTO-0nFd-6xj2-MYDJ-Yk80A1

root@n-1:~# pvdisplay --columns --all
PV VG Fmt Attr PSize PFree
/dev/mapper/pure3_b-preview-1 pure3_b-preview-1 lvm2 a-- <8.00t 5.03t
/dev/sda2 --- 0 0
/dev/sda3 pve lvm2 a-- <446.07g 16.00g


root@n-1:~# vgdisplay storage1-preview-1
--- Volume group ---
VG Name storage1-preview-1
System ID
Format lvm2
Metadata Areas 1
Metadata Sequence No 804
VG Access read/write
VG Status resizable
MAX LV 0
Cur LV 511
Open LV 100
Max PV 0
Cur PV 1
Act PV 1
VG Size <8.00 TiB
PE Size 4.00 MiB
Total PE 2097151
Alloc PE / Size 778389 / <2.97 TiB
Free PE / Size 1318762 / 5.03 TiB
VG UUID ZqF5d2-M2Sr-DxGT-shls-1Lfg-qYyk-znWthJ

My VMs config file:
root@n-1:~# cat /etc/pve/qemu-server/107.conf
agent: enabled=1
autostart: 1
balloon: 0
boot: order=scsi0
cicustom: user=pve-cloudinit:snippets/testvm.yaml
cores: 1
cpu: x86-64-v2-AES
hotplug: network,disk,cpu,memory,usb,cloudinit
ide2: storage1-preview-1:vm-107-cloudinit,media=cdrom
ipconfig0: gw=10.10.10.1,ip=10.10.10.10/20
memory: 9216
meta: creation-qemu=10.0.2,ctime=1753274457
migrate_downtime: 0.05
name: vmname.com
nameserver: 10.0.0.2
net0: virtio=BC:24:11:EA:04:50,bridge=testnet
numa: 1
ostype: l26
scsi0: storage1-preview-1:vm-107-disk-0.qcow2,aio=threads,iothread=1,size=10G
scsi2: storage1-preview-1:vm-107-disk-2.qcow2,aio=threads,iothread=1,size=4G
scsi3: storage1-preview-1:vm-107-disk-1.qcow2,aio=threads,iothread=1,size=51G
scsihw: virtio-scsi-single
searchdomain: example.com
smbios1: uuid=091c13c1-f47f-474d-a38c-62418d60a710
sockets: 4
tags: ops;ubuntu_24.04_lts
vmgenid: 430b01f0-5a90-4701-8bf2-240bae377af0


So at least for me, it looks like there would be plenty of free space left on the host.
To me, the VM disk size and the LV size seem to match.

Is there anything I am missing here?

Thanks a lot in advance
 
Nothing in the logs (dmesg, journalctl, /var/log) that catched our attention while investigating
  • The VM (Ubuntu 24.04) itself logs nothing related, it just stopps
  • The nodes (PVE 9.0.6) do not tell us anything we considered related, we also did not really find any logs telling us the reason/task to set the node to "io-error" we only see some of this:
    • Code:
      Sep 24 11:50:40 n-1 pvedaemon[567027]: VM 107 qmp command failed - VM 107 qmp command 'guest-ping' failed - got timeout
      Sep 24 11:50:57 n-1 pvedaemon[580190]: VM 107 qmp command failed - VM 107 qmp command 'guest-ping' failed - got timeout
      Sep 24 11:50:59 n-1 pvedaemon[582973]: VM 107 qmp command failed - VM 107 qmp command 'guest-ping' failed - got timeout
      Sep 24 11:51:18 n-1 pvedaemon[567027]: VM 107 qmp command failed - VM 107 qmp command 'guest-ping' failed - got timeout
  • If we reset the VM, it runs for a random ammount of time (ranging from minutes to hours) and goes to "io-error" state again
    • If we cause some disk activity we can likely force it to break earlier
  • If we move the VM to another node, its the same behaviour.
  • We guess its something around the root disk/lvm of the VM? We have not yet tried to resized the affected disk as we are curious to undersand the reason behind issue.
  • The VM itself is pretty unimportant and is idling around (it does not have any noteworthy workload/load/diskusage etc)
 
Last edited:
If I understand correctly, the VM crashes or freezes. If yes, it may not be able to record an error in the logs. But maybe it writes some error on its (virtual) terminal.

So my suggestion is to log in to the VM's virtual serial terminal from the PVE's shell and wait. Maybe the VM will write some error on its console.

Some instructions on how to configure serial terminal in VM:

https://pve.proxmox.com/wiki/Serial_Terminal (a little dated, doesn't describe configuration of systemd, but this is at https://0pointer.de/blog/projects/serial-console.html )

https://codecisions.com/connecting-a-serial-terminal-to-a-proxmox-virtual-machine/

https://devopstales.github.io/virtualization/proxmox-xtermjs-enable/
 
Last edited:
  • Like
Reactions: nwad
@nwad , to ensure that you don't miss the output from the console you can execute these commands via SSH terminal:
script serial.log
qm terminal [vmid]

and leave it running.

Another option that we use in our testing is to add char device:
--args -chardev file,id=char0,mux=on,path=/tmp/serial."$vmid".log,signal=off -serial chardev:char0

This will log anything to /tmp/serial.vmid.log

I also noticed that you are using the new Preview feature of QCOW/LVM. Is there anything particular about traffic pattern for this VM? Are you taking/keeping snapshots?
Have you tried to move it to traditional RAW LVM?


Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox
 
  • Like
Reactions: nwad
Hey, thank you for your replies,

unfortunately there is no output on the serial terminal when it goes to "io-error" state.

We enabled the snapshots feature with v9, but are not using it actively/automated.
This machine does not have any snapshots. And no other machine on this datastore has some.

This VM was the first VM we deployed to this cluster and served as a test to verify our provisioning of VMs to this cluster. Since then it was just lying around. No load on it, just a basic Ubuntu 24.04 with some basic configuration.

We then deployed >100 more VMs on that cluster that do some real work when we some weeks later noticed that this VM went into "io-error" state.
The other VMs are fine, but we are a bit nervous what we might have hit here.
 
Hi,
please share the output of pveversion -v as well as
Code:
qemu-img info /dev/storage1-preview-1/vm-107-disk-0.qcow2 --output json
qemu-img check -f qcow2 /tmp/disk.qcow2

Was the image initially copied/moved from another storage?
 
  • Like
Reactions: nwad
Hey,
thank you for your answer.
no, the image was created on that datastore from a template like all other VMs as well.

Bash:
root@n-1:~# pveversion -v
proxmox-ve: 9.0.0 (running kernel: 6.14.8-2-pve)
pve-manager: 9.0.6 (running version: 9.0.6/49c767b70aeb6648)
proxmox-kernel-helper: 9.0.4
proxmox-kernel-6.14.11-2-pve-signed: 6.14.11-2
proxmox-kernel-6.14: 6.14.11-2
proxmox-kernel-6.14.8-2-pve-signed: 6.14.8-2
proxmox-kernel-6.14.8-1-pve-signed: 6.14.8-1
proxmox-kernel-6.8.12-12-pve-signed: 6.8.12-12
proxmox-kernel-6.8: 6.8.12-12
proxmox-kernel-6.8.12-11-pve-signed: 6.8.12-11
proxmox-kernel-6.8.12-9-pve-signed: 6.8.12-9
ceph-fuse: 19.2.3-pve1
corosync: 3.1.9-pve2
criu: 4.1.1-1
frr-pythontools: 10.3.1-1+pve4
ifupdown2: 3.3.0-1+pmx10
intel-microcode: 3.20250512.1
ksm-control-daemon: 1.5-1
libjs-extjs: 7.0.0-5
libproxmox-acme-perl: 1.7.0
libproxmox-backup-qemu0: 2.0.1
libproxmox-rs-perl: 0.4.1
libpve-access-control: 9.0.3
libpve-apiclient-perl: 3.4.0
libpve-cluster-api-perl: 9.0.6
libpve-cluster-perl: 9.0.6
libpve-common-perl: 9.0.10
libpve-guest-common-perl: 6.0.2
libpve-http-server-perl: 6.0.4
libpve-network-perl: 1.1.7
libpve-rs-perl: 0.10.10
libpve-storage-perl: 9.0.13
libspice-server1: 0.15.2-1+b1
lvm2: 2.03.31-2+pmx1
lxc-pve: 6.0.5-1
lxcfs: 6.0.4-pve1
novnc-pve: 1.6.0-3
openvswitch-switch: 3.5.0-1+b1
proxmox-backup-client: 4.0.14-1
proxmox-backup-file-restore: 4.0.14-1
proxmox-backup-restore-image: 1.0.0
proxmox-firewall: 1.1.2
proxmox-kernel-helper: 9.0.4
proxmox-mail-forward: 1.0.2
proxmox-mini-journalreader: 1.6
proxmox-offline-mirror-helper: 0.7.1
proxmox-widget-toolkit: 5.0.5
pve-cluster: 9.0.6
pve-container: 6.0.11
pve-docs: 9.0.8
pve-edk2-firmware: 4.2025.02-4
pve-esxi-import-tools: 1.0.1
pve-firewall: 6.0.3
pve-firmware: 3.16-4
pve-ha-manager: 5.0.4
pve-i18n: 3.6.0
pve-qemu-kvm: 10.0.2-4
pve-xtermjs: 5.5.0-2
qemu-server: 9.0.21
smartmontools: 7.4-pve1
spiceterm: 3.4.1
swtpm: 0.8.0+pve2
vncterm: 1.9.1
zfsutils-linux: 2.3.4-pve1

Bash:
root@n-1:~# qemu-img info /dev/storage1-preview-1/vm-107-disk-0.qcow2 --output json
{
    "children": [
        {
            "name": "file",
            "info": {
                "children": [
                ],
                "virtual-size": 10741612544,
                "filename": "/dev/storage1-preview-1/vm-107-disk-0.qcow2",
                "format": "host_device",
                "actual-size": 0,
                "format-specific": {
                    "type": "file",
                    "data": {
                    }
                },
                "dirty-flag": false
            }
        }
    ],
    "virtual-size": 10737418240,
    "filename": "/dev/storage1-preview-1/vm-107-disk-0.qcow2",
    "cluster-size": 65536,
    "format": "qcow2",
    "actual-size": 0,
    "format-specific": {
        "type": "qcow2",
        "data": {
            "compat": "1.1",
            "compression-type": "zlib",
            "lazy-refcounts": false,
            "refcount-bits": 16,
            "corrupt": false,
            "extended-l2": false
        }
    },
    "dirty-flag": false
}

Bash:
root@n-1:~# qemu-img check -f qcow2 /dev/storage1-preview-1/vm-107-disk-0.qcow2
No errors were found on the image.
155662/163840 = 95.01% allocated, 6.83% fragmented, 0.00% compressed clusters
Image end offset: 10741612544
 
I just noticed that in your qmp_capabilities output, the block related to disk-0 contains:

"io-status":"nospace"

That’s a important clue. My guess is that the LV is potentially not properly sized. You have a 10 GiB LV with a 10 GiB QCOW image. Small enough that most of the clusters (≈95%) are allocated, leaving the QCOW with no space for its metadata.

If I remember correctly, QCOW metadata is updated as you write to new blocks, not existing ones. This could explain why your VM runs for a while before crashing.

If my theory is correct, the short-term fix is to extend your LV manually. The longer-term discussion would focus on:
  • How much overprovisioning to allow when creating new disks
  • How to adequately resize your existing infrastructure to prevent similar issues

P.S. we created a small technical overview of the LVM/QCOW that may be helpful: https://kb.blockbridge.com/technote/proxmox-qcow-snapshots-on-lvm/


Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox
 
Last edited:
  • Like
Reactions: nwad
Bash:
root@n-1:~# qemu-img check -f qcow2 /dev/storage1-preview-1/vm-107-disk-0.qcow2
No errors were found on the image.
155662/163840 = 95.01% allocated, 6.83% fragmented, 0.00% compressed clusters
Image end offset: 10741612544
The issue is here. Because of the fragmentation, more space is needed than the LV is prepared to handle. This was fixed a while ago: https://bugzilla.proxmox.com/show_bug.cgi?id=6543 Could it be that the image was created while you were still running a version without the fix, i.e. qemu-server < 9.0.8? You can check /var/log/apt/history.log and rotated friends .1.gz, .2.gz etc.

Otherwise please share the configuration of the template and the underlying storage of the template and the steps you created the clone with.

You should be able to remove the fragmentation by moving the image to another storage and then back again.
 
Holy .. wow yeah!
The installation of his cluster was finished under 9.0.4. on 23.09.2025 and this VM was created as first VM.
We upgraded this cluster to 9.0.8 a week later and all other VMs were created after that.

Thank you @fiona for pointing us there!

Thanks a lot to all people looking into this!
 
  • Like
Reactions: fiona