VM freezes with io-error

KingTChoka

Member
Sep 4, 2021
39
2
13
28
Hi,

I have a VM running Rocky Linux and it was working for a while (2 years), but now it's starting to freeze with io-error. The HDD itself is getting pretty full, but in the VM where the HDD is hosting the image, there's still some space left. S.M.A.R.T. scan shows no issues. When the VM boots it works fine for a few minutes then freezes with the error. Running on PVE 7.0-8

I'm not sure where to start to confirm where the issue lies and how to fix. Could anyone guide me to help me get this resolved? Thanks!
 
Last edited:
Hi,
I have a VM running Rocky Linux and it was working for a while (2 years), but now it's starting to freeze with io-error. The HDD itself is getting pretty full,
how full? What kind of storage is it?
but in the VM where the HDD is hosting the image, there's still some space left.
It doesn't matter if the VM thinks there is space when there actually is none ;)
S.M.A.R.T. scan shows no issues. When the VM boots it works fine for a few minutes then freezes with the error. Running on PVE 7.0-8
Not related to your issue, but it's highly recommended to upgrade to the latest version of 7.4 (or even 8) to get the latest (security) updates and fixes.
I'm not sure where to start to confirm where the issue lies and how to fix. Could anyone guide me to help me get this resolved? Thanks!
Anything interesting in the system logs/journal around the time the issue happens?

You can install jq and after the issue happens, run the following (replacing 123 with the ID of your VM!):
Code:
echo '{"execute": "qmp_capabilities"}{"execute": "query-block", "arguments": {}}' | socat - /var/run/qemu-server/123.qmp | jq
 
how full? What kind of storage is it?
Pretty full haha
Code:
# df -hT
Filesystem           Type      Size  Used Avail Use% Mounted on
/dev/sda1            ext4      7.3T  7.3T     0 100% /hdd-wd-8tb.1

Not related to your issue, but it's highly recommended to upgrade to the latest version of 7.4 (or even 8) to get the latest (security) updates and fixes.
Would this command be safe to do so?
[B]# apt-get update && apt-get dist-upgrade[/B]

Code:
# echo '{"execute": "qmp_capabilities"}{"execute": "query-block", "arguments": {}}' | socat - /var/run/qemu-server/111.qmp | jq

{
  "QMP": {
    "version": {
      "qemu": {
        "micro": 0,
        "minor": 0,
        "major": 6
      },
      "package": "pve-qemu-kvm_6.0.0"
    },
    "capabilities": []
  }
}
{
  "return": {}
}
{
  "return": [
    {
      "io-status": "ok",
      "device": "drive-ide2",
      "locked": true,
      "removable": true,
      "inserted": {
        "iops_rd": 0,
        "detect_zeroes": "off",
        "image": {
          "virtual-size": 1980760064,
          "filename": "/var/lib/vz/template/iso/Rocky-8.4-x86_64-minimal.iso",
          "format": "raw",
          "actual-size": 1980764160,
          "dirty-flag": false
        },
        "iops_wr": 0,
        "ro": true,
        "node-name": "#block116",
        "backing_file_depth": 0,
        "drv": "raw",
        "iops": 0,
        "bps_wr": 0,
        "write_threshold": 0,
        "encrypted": false,
        "bps": 0,
        "bps_rd": 0,
        "cache": {
          "no-flush": false,
          "direct": false,
          "writeback": true
        },
        "file": "/var/lib/vz/template/iso/Rocky-8.4-x86_64-minimal.iso"
      },
      "qdev": "ide2",
      "tray_open": false,
      "type": "unknown"
    },
    {
      "io-status": "nospace",
      "device": "drive-scsi0",
      "locked": false,
      "removable": false,
      "inserted": {
        "iops_rd": 0,
        "detect_zeroes": "on",
        "image": {
          "virtual-size": 8053063680000,
          "filename": "/hdd-wd-8tb.1/images/111/vm-111-disk-0.raw",
          "format": "raw",
          "actual-size": 7937285545984,
          "dirty-flag": false
        },
        "iops_wr": 0,
        "ro": false,
        "node-name": "#block380",
        "backing_file_depth": 0,
        "drv": "raw",
        "iops": 0,
        "bps_wr": 0,
        "write_threshold": 0,
        "encrypted": false,
        "bps": 0,
        "bps_rd": 0,
        "cache": {
          "no-flush": false,
          "direct": true,
          "writeback": true
        },
        "file": "/hdd-wd-8tb.1/images/111/vm-111-disk-0.raw"
      },
      "qdev": "scsi0",
      "type": "unknown"
    },
    {
      "io-status": "ok",
      "device": "drive-sata1",
      "locked": false,
      "removable": false,
      "inserted": {
        "iops_rd": 0,
        "detect_zeroes": "on",
        "image": {
          "virtual-size": 7696581394432,
          "filename": "/hdd-wd-8tb.2/images/111/vm-111-disk-0.raw",
          "format": "raw",
          "actual-size": 1126254546944,
          "dirty-flag": false
        },
        "iops_wr": 0,
        "ro": false,
        "node-name": "#block549",
        "backing_file_depth": 0,
        "drv": "raw",
        "iops": 0,
        "bps_wr": 0,
        "write_threshold": 0,
        "encrypted": false,
        "bps": 0,
        "bps_rd": 0,
        "cache": {
          "no-flush": false,
          "direct": true,
          "writeback": true
        },
        "file": "/hdd-wd-8tb.2/images/111/vm-111-disk-0.raw"
      },
      "qdev": "sata1",
      "type": "unknown"
    }
  ]
}
 
Pretty full haha
Code:
# df -hT
Filesystem           Type      Size  Used Avail Use% Mounted on
/dev/sda1            ext4      7.3T  7.3T     0 100% /hdd-wd-8tb.1
Well, that explains it. As for why this can happen even though the VM "sees" more space, please see:
https://en.wikipedia.org/wiki/Sparse_file (which your .raw VM images are)
https://pve.proxmox.com/pve-docs/chapter-pvesm.html#_thin_provisioning

Would this command be safe to do so?
[B]# apt-get update && apt-get dist-upgrade[/B]
Yes, always use dist-upgrade or full-upgrade when updating Proxmox VE. Make sure you have the correct repositories configured (enterprise if you have a subscription and no-subscription otherwise): https://pve.proxmox.com/wiki/Package_Repositories#_proxmox_ve_7_x_repositories

Code:
      "io-status": "nospace",
      "device": "drive-scsi0",
This status is expected when the underlying storage is full.
 
Is there any safe way to create more space so my VM stops crashing? In my VM it sees that there should be around 150GB space left
 
Is there any safe way to create more space so my VM stops crashing?
You need to find out what is using the space (on the host). I assume /hdd-wd-8tb.1/ is a separate device/mount point? If yes, you need to look there.
In my VM it sees that there should be around 150GB space left
Again, it doesn't matter if the VM thinks there is space when in reality there is none, i.e. on the underlying storage where the sparse image file is.
 
You need to find out what is using the space (on the host). I assume /hdd-wd-8tb.1/
It only contains the .raw VM file, which is currently taking up the whole space. Is it possible to reduce the .raw image file somehow?
 
It only contains the .raw VM file, which is currently taking up the whole space. Is it possible to reduce the .raw image file somehow?
If the space is actually in-use then no. What you can try is enabling discard on the disk (requires starting the VM anew or Reboot via web UI - reboot within the guest is not enough - so it picks up the change) and issuing an fstrim. What is the output of qemu-img info /path/to/the/raw/file.raw?
 
What is the output of qemu-img info /path/to/the/raw/file.raw?
Code:
# qemu-img info vm-111-disk-0.raw
image: vm-111-disk-0.raw
file format: raw
virtual size: 7.32 TiB (8053063680000 bytes)
disk size: 7.22 TiB

issuing an fstrim
How do I issue this fstrim? Do i run it on the proxmox host, or within the VM?
 
Last edited:
Code:
# qemu-img info vm-111-disk-0.raw
image: vm-111-disk-0.raw
file format: raw
virtual size: 7.32 TiB (8053063680000 bytes)
disk size: 7.22 TiB
This means that 7.22 TiB are actually in-use by the file representing the disk image.
How do I issue this fstrim? Do i run it on the proxmox host, or within the VM?
Inside the VM. You need to enable the discard option on the VM disk first.
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!