VM freezes with io-error

KingTChoka · Jan 28, 2024

Hi,

I have a VM running Rocky Linux and it was working for a while (2 years), but now it's starting to freeze with io-error. The HDD itself is getting pretty full, but in the VM where the HDD is hosting the image, there's still some space left. S.M.A.R.T. scan shows no issues. When the VM boots it works fine for a few minutes then freezes with the error. Running on PVE 7.0-8

I'm not sure where to start to confirm where the issue lies and how to fix. Could anyone guide me to help me get this resolved? Thanks!

fiona · Jan 29, 2024

Hi,

KingTChoka said:
I have a VM running Rocky Linux and it was working for a while (2 years), but now it's starting to freeze with io-error. The HDD itself is getting pretty full,

how full? What kind of storage is it?

KingTChoka said:
but in the VM where the HDD is hosting the image, there's still some space left.

It doesn't matter if the VM thinks there is space when there actually is none

KingTChoka said:
S.M.A.R.T. scan shows no issues. When the VM boots it works fine for a few minutes then freezes with the error. Running on PVE 7.0-8

Not related to your issue, but it's highly recommended to upgrade to the latest version of 7.4 (or even 8) to get the latest (security) updates and fixes.

KingTChoka said:
I'm not sure where to start to confirm where the issue lies and how to fix. Could anyone guide me to help me get this resolved? Thanks!

Anything interesting in the system logs/journal around the time the issue happens?

You can install jq and after the issue happens, run the following (replacing 123 with the ID of your VM!):

Code:

echo '{"execute": "qmp_capabilities"}{"execute": "query-block", "arguments": {}}' | socat - /var/run/qemu-server/123.qmp | jq

KingTChoka · Feb 2, 2024

how full? What kind of storage is it?

Pretty full haha

Code:

# df -hT
Filesystem           Type      Size  Used Avail Use% Mounted on
/dev/sda1            ext4      7.3T  7.3T     0 100% /hdd-wd-8tb.1

Not related to your issue, but it's highly recommended to upgrade to the latest version of 7.4 (or even 8) to get the latest (security) updates and fixes.

Would this command be safe to do so?
[B]# apt-get update && apt-get dist-upgrade[/B]

Code:

# echo '{"execute": "qmp_capabilities"}{"execute": "query-block", "arguments": {}}' | socat - /var/run/qemu-server/111.qmp | jq

{
  "QMP": {
    "version": {
      "qemu": {
        "micro": 0,
        "minor": 0,
        "major": 6
      },
      "package": "pve-qemu-kvm_6.0.0"
    },
    "capabilities": []
  }
}
{
  "return": {}
}
{
  "return": [
    {
      "io-status": "ok",
      "device": "drive-ide2",
      "locked": true,
      "removable": true,
      "inserted": {
        "iops_rd": 0,
        "detect_zeroes": "off",
        "image": {
          "virtual-size": 1980760064,
          "filename": "/var/lib/vz/template/iso/Rocky-8.4-x86_64-minimal.iso",
          "format": "raw",
          "actual-size": 1980764160,
          "dirty-flag": false
        },
        "iops_wr": 0,
        "ro": true,
        "node-name": "#block116",
        "backing_file_depth": 0,
        "drv": "raw",
        "iops": 0,
        "bps_wr": 0,
        "write_threshold": 0,
        "encrypted": false,
        "bps": 0,
        "bps_rd": 0,
        "cache": {
          "no-flush": false,
          "direct": false,
          "writeback": true
        },
        "file": "/var/lib/vz/template/iso/Rocky-8.4-x86_64-minimal.iso"
      },
      "qdev": "ide2",
      "tray_open": false,
      "type": "unknown"
    },
    {
      "io-status": "nospace",
      "device": "drive-scsi0",
      "locked": false,
      "removable": false,
      "inserted": {
        "iops_rd": 0,
        "detect_zeroes": "on",
        "image": {
          "virtual-size": 8053063680000,
          "filename": "/hdd-wd-8tb.1/images/111/vm-111-disk-0.raw",
          "format": "raw",
          "actual-size": 7937285545984,
          "dirty-flag": false
        },
        "iops_wr": 0,
        "ro": false,
        "node-name": "#block380",
        "backing_file_depth": 0,
        "drv": "raw",
        "iops": 0,
        "bps_wr": 0,
        "write_threshold": 0,
        "encrypted": false,
        "bps": 0,
        "bps_rd": 0,
        "cache": {
          "no-flush": false,
          "direct": true,
          "writeback": true
        },
        "file": "/hdd-wd-8tb.1/images/111/vm-111-disk-0.raw"
      },
      "qdev": "scsi0",
      "type": "unknown"
    },
    {
      "io-status": "ok",
      "device": "drive-sata1",
      "locked": false,
      "removable": false,
      "inserted": {
        "iops_rd": 0,
        "detect_zeroes": "on",
        "image": {
          "virtual-size": 7696581394432,
          "filename": "/hdd-wd-8tb.2/images/111/vm-111-disk-0.raw",
          "format": "raw",
          "actual-size": 1126254546944,
          "dirty-flag": false
        },
        "iops_wr": 0,
        "ro": false,
        "node-name": "#block549",
        "backing_file_depth": 0,
        "drv": "raw",
        "iops": 0,
        "bps_wr": 0,
        "write_threshold": 0,
        "encrypted": false,
        "bps": 0,
        "bps_rd": 0,
        "cache": {
          "no-flush": false,
          "direct": true,
          "writeback": true
        },
        "file": "/hdd-wd-8tb.2/images/111/vm-111-disk-0.raw"
      },
      "qdev": "sata1",
      "type": "unknown"
    }
  ]
}

fiona · Feb 2, 2024

KingTChoka said:

Well, that explains it. As for why this can happen even though the VM "sees" more space, please see:
https://en.wikipedia.org/wiki/Sparse_file (which your .raw VM images are)
https://pve.proxmox.com/pve-docs/chapter-pvesm.html#_thin_provisioning

KingTChoka said:
Would this command be safe to do so?
[B]# apt-get update && apt-get dist-upgrade[/B]

Yes, always use dist-upgrade or full-upgrade when updating Proxmox VE. Make sure you have the correct repositories configured (enterprise if you have a subscription and no-subscription otherwise): https://pve.proxmox.com/wiki/Package_Repositories#_proxmox_ve_7_x_repositories

KingTChoka said:
Code:

"io-status": "nospace", "device": "drive-scsi0",

This status is expected when the underlying storage is full.

KingTChoka · Feb 3, 2024

Is there any safe way to create more space so my VM stops crashing? In my VM it sees that there should be around 150GB space left

fiona · Feb 5, 2024

KingTChoka said:
Is there any safe way to create more space so my VM stops crashing?

You need to find out what is using the space (on the host). I assume /hdd-wd-8tb.1/ is a separate device/mount point? If yes, you need to look there.

KingTChoka said:
In my VM it sees that there should be around 150GB space left

Again, it doesn't matter if the VM thinks there is space when in reality there is none, i.e. on the underlying storage where the sparse image file is.

KingTChoka · Feb 20, 2024

fiona said:
You need to find out what is using the space (on the host). I assume /hdd-wd-8tb.1/

It only contains the .raw VM file, which is currently taking up the whole space. Is it possible to reduce the .raw image file somehow?

fiona · Feb 20, 2024

KingTChoka said:
It only contains the .raw VM file, which is currently taking up the whole space. Is it possible to reduce the .raw image file somehow?

If the space is actually in-use then no. What you can try is enabling discard on the disk (requires starting the VM anew or Reboot via web UI - reboot within the guest is not enough - so it picks up the change) and issuing an fstrim. What is the output of qemu-img info /path/to/the/raw/file.raw?

KingTChoka · Feb 25, 2024

fiona said:
What is the output of qemu-img info /path/to/the/raw/file.raw?

Code:

# qemu-img info vm-111-disk-0.raw
image: vm-111-disk-0.raw
file format: raw
virtual size: 7.32 TiB (8053063680000 bytes)
disk size: 7.22 TiB

fiona said:
issuing an fstrim

How do I issue this fstrim? Do i run it on the proxmox host, or within the VM?

fiona · Feb 26, 2024

KingTChoka said:

This means that 7.22 TiB are actually in-use by the file representing the disk image.

KingTChoka said:
How do I issue this fstrim? Do i run it on the proxmox host, or within the VM?

Inside the VM. You need to enable the discard option on the VM disk first.

Search

Search

VM freezes with io-error

KingTChoka

Member

fiona

Proxmox Staff Member

KingTChoka

Member

fiona

Proxmox Staff Member

KingTChoka

Member

fiona

Proxmox Staff Member

KingTChoka

Member

fiona

Proxmox Staff Member

KingTChoka

Member

fiona

Proxmox Staff Member