VM Status stuck on io-error

[wahmed]

One of my Proxmox node has 4 Ubuntu VMs. All virtual disks in these VMs local, not on shared storage. They have been working without issue for several months now. Few hours ago 3 of the VM showing me io-error on the status. Shutdown/Stop/Reset none of them made it go away.
Any help to figure out the issue would be much appreciated!

 

[dietmar]

Any hint in /var/log/syslog (maybe local disk gets dying)?

 

[davlaw]

I have started to see this as well, but I feel its my SAN environment I'm connected to (48TB Scale Computing) being rather cranky when it starts to become full. When I free up some room its ok. Adding 16TB soon so I should get a better idea.

 

[wahmed]

Any hint in /var/log/syslog (maybe local disk gets dying)?

I checked the log. Shows business as usual.

I have started to see this as well, but I feel its my SAN environment I'm connected to (48TB Scale Computing) being rather cranky when it starts to become full. When I free up some room its ok. Adding 16TB soon so I should get a better idea.

The node still had 20GB free space. So i did not think space was the issue. But just to make sure, i moved the VMs in question to another node and they started working right away. The node i moved to actually have 15GB space left.

As dietmar mentioned, may be the hard drive is dying. I cant think of any other reason.

 

[davlaw]

Good luck, hope my io-error it not some other issue... but could be tho. Keeping watchful eye...