Getting "Status: io-error" on Windows 10 VM

[themanicotti]

Hello,

I'm getting an io-error on a Windows VM almost as soon as it starts. The VM will lock up and I cannot access it after that. The disk isn't full (none of the disks are) and I can't find any errors in /var/log/syslog (attached). This VM was running fine for weeks until this afternoon. I haven't done any updates (until today, while troubleshooting) or made any hardware changes.

Any help appreciated!

pveversion: pve-manager/7.1-7/df5740ad (running kernel: 5.13.19-2-pve)

 

[bbgeek17]

Did you check "dmesg" output?

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[fiona]

Hi,

I'm getting an io-error on a Windows VM almost as soon as it starts. The VM will lock up and I cannot access it after that. The disk isn't full (none of the disks are) and I can't find any errors in /var/log/syslog (attached). This VM was running fine for weeks until this afternoon. I haven't done any updates (until today, while troubleshooting) or made any hardware changes.

what kind of storage are the VM's disks on? Please use a tool like smartctl to check your physical disk's health.

pveversion: pve-manager/7.1-7/df5740ad (running kernel: 5.13.19-2-pve)

Likely not related to the issue, but the installation is rather old. Please only proceed with updating after checking that the disks are fine!
https://pve.proxmox.com/pve-docs/chapter-sysadmin.html#system_software_updates
https://pve.proxmox.com/wiki/Package_Repositories#_proxmox_ve_7_x_repositories
and Proxmox VE 7.4 will be end of life in a few months too, so:
https://pve.proxmox.com/wiki/Upgrade_from_7_to_8

 

[themanicotti]

Did you check "dmesg" output?

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

I did, but I didn't really know what to look for. I didn't see anything that seemed related to that error, but I definitely don't have as much experience with Linux logs as some. Here is the output of dmesg

 

[themanicotti]

Hi,

what kind of storage are the VM's disks on? Please use a tool like smartctl to check your physical disk's health.


Likely not related to the issue, but the installation is rather old. Please only proceed with updating after checking that the disks are fine!
https://pve.proxmox.com/pve-docs/chapter-sysadmin.html#system_software_updates
https://pve.proxmox.com/wiki/Package_Repositories#_proxmox_ve_7_x_repositories
and Proxmox VE 7.4 will be end of life in a few months too, so:
https://pve.proxmox.com/wiki/Upgrade_from_7_to_8

The VM's boot disk is on local-lvm (a 256gb SSD). The VM also has 6 additional HDDs passed through to it.

I recently had an issue and had to restore this VM from a backup. I'm realizing today that I restored it to local-lvm instead of the dedicated SSD in that machine that I was using previously. The VM's boot partition is only 120GB, so it fits on both disks fine... Would sharing local-lvm cause problems? Should I try restoring it back to the other disk?

ETA: The reason I'm hesitant to move the VM back is the 6 additional HDDs. There is about 16TB of data between them and the last time I restored from a backup, all of those disks were wiped. I use PBS and only the boot disk of the guest is set to backup. Is there a way to restore the VM to the other disk and then reassign the HDDs without wiping them?

 

[themanicotti]

what kind of storage are the VM's disks on? Please use a tool like smartctl to check your physical disk's health.

The disks are being passed through a RAID controller, so I'm not able to get SMART information

 

[fiona]

The VM's boot disk is on local-lvm (a 256gb SSD). The VM also has 6 additional HDDs passed through to it.

I recently had an issue and had to restore this VM from a backup. I'm realizing today that I restored it to local-lvm instead of the dedicated SSD in that machine that I was using previously. The VM's boot partition is only 120GB, so it fits on both disks fine... Would sharing local-lvm cause problems? Should I try restoring it back to the other disk?

What is the output of lvs? Multiple disks in a thin pool don't usually cause problems, that's the intended use case after all. If you have multiple disks, you can check which disks got the error (after it happened). Install jq and socat and then use

echo '{"execute": "qmp_capabilities"}{"execute": "query-block", "arguments": {}}' | socat - /var/run/qemu-server/123.qmp | jq

replacing 123 with the actual ID of your VM.

Then you can see if the boot disk is actually the one with the issue. If it's not, there is no reason to move it.

ETA: The reason I'm hesitant to move the VM back is the 6 additional HDDs. There is about 16TB of data between them and the last time I restored from a backup, all of those disks were wiped. I use PBS and only the boot disk of the guest is set to backup. Is there a way to restore the VM to the other disk and then reassign the HDDs without wiping them?

Hmm, that really shouldn't happen. You will need to re-attach (passthrough) disks not included in the backup, but they shouldn't be wiped. Only disks that are also in the backup will overwrite VM disks during restore, e.g. if there is scsi0 in the backup, the current scsi0 disk will be removed. If you really want to make sure, just detach the disks from the VM before restore and attach afterwards.

But you don't even need to re-assing. You can move the boot disk to the other storage while the VM is running using Disk Action > Move Storage in the UI.

 

[themanicotti]

Fiona,

Thanks for getting back to me. I figured out the solution - one of my HDDs that was passed through had died. I found a warning message on the display on the physical server. I'm not sure if the RAID controller hid the information from Proxmox or if I just didn't know where to look, but I haven't had any issues with the VM since pulling the drive (6 hours or so).

Thank you for your help!

Additionally, I went to upgrade my Proxmox and I'm getting some errors that I believe to be linked to my repositories. When upgrading from 7.1-7 should I use the bullseye repos before using the bookworm repos? I'm not able to run pve7to8, which makes me believe I need to do some more upgrades with bullseye. Running apt dist-upgrade returns an error:

W: (pve-apt-hook) !! WARNING !!
W: (pve-apt-hook) You are attempting to remove the meta-package 'proxmox-ve'!
W: (pve-apt-hook)
W: (pve-apt-hook) If you really want to permanently remove 'proxmox-ve' from your system, run the following command
W: (pve-apt-hook) touch '/please-remove-proxmox-ve'
W: (pve-apt-hook) run apt purge proxmox-ve to remove the meta-package
W: (pve-apt-hook) and repeat your apt invocation.
W: (pve-apt-hook)
W: (pve-apt-hook) If you are unsure why 'proxmox-ve' would be removed, please verify
W: (pve-apt-hook) - your APT repository settings
W: (pve-apt-hook) - that you are using 'apt full-upgrade' to upgrade your system
E: Sub-process /usr/share/proxmox-ve/pve-apt-hook returned an error code (1)
E: Failure running script /usr/share/proxmox-ve/pve-apt-hook

Happy to make this another ticket if that's easier

 

[fiona]

Fiona,

Thanks for getting back to me. I figured out the solution - one of my HDDs that was passed through had died. I found a warning message on the display on the physical server. I'm not sure if the RAID controller hid the information from Proxmox or if I just didn't know where to look, but I haven't had any issues with the VM since pulling the drive (6 hours or so).

Thank you for your help!

Glad to hear (well, except about the dead disk of course).

Additionally, I went to upgrade my Proxmox and I'm getting some errors that I believe to be linked to my repositories. When upgrading from 7.1-7 should I use the bullseye repos before using the bookworm repos? I'm not able to run pve7to8, which makes me believe I need to do some more upgrades with bullseye. Running apt dist-upgrade returns an error:

Yes, you need to upgrade to the latest version of 7.4 first. Please follow the upgrade guide closely: https://pve.proxmox.com/wiki/Upgrade_from_7_to_8#Introduction