KVM disk goes readonly

[Guy]

So I'm happily working away in my KVM image of Fedora and suddenly it reports a journaling error and changes the drive to read only.

I've then tried to reboot this KVM and it comes back read/write and then a short time later same issue again. Ok maybe this KVM is corrupted.

I create a new KVM and install Fedora from the ISO DVD and shortly after the install completes and I start using it, it again locks up in to read only.

Any ideas?

Cheers,
--Guy

 

[Dave]

I've also been experiencing this exact same issue on all 5 of my Gentoo Hardened VM's since rebuilding Proxmox VE with the latest release. Tried various things but still haven't gotten to the bottom of it, was on the verge of rebuilding them all but if someone else is getting this then it might be related to the new version of KVM and not the result of my server being possessed by evil spirits as i suspected...

 

[Dave]

From memory just doing a warm reboot of the VM did not fix the issue, I had to Stop and Start the VM from the web interface to bring the filesystem back up in read/write.

 

[Guy]

well I've reboot the proxmox server and it still swaps to readonly shortly later.

--Guy

 

[Dave]

Ok, it happened just now when I was compiling something.

I can see the following in the VM's dmesg:

end_request: I/O error, dev vda, sector 159135576
end_request: I/O error, dev vda, sector 159134584
end_request: I/O error, dev vda, sector 159135080
REISERFS: abort (device vda3): Journal write error in flush_commit_list
REISERFS: Aborting journal for filesystem on vda3

Nothing obvious in any of the base Proxmox VE server's logs.

I've just done a warm 'reboot' from the command line on the VM and it has come back up as read/write this time, but I have had previous times where it hasn't and a stop/start was required from the web interface.

 

[Dave]

My VM's are all running ReiserFS on VIRTIO disks and Gentoo Hardened kernel 2.6.26-hardened-r9. Proxmox VE is a fresh build of the latest release fully patched and up to date.

Guy, what filesystem and kernel version are you using on your Fedora VM? Also, are you using VIRTIO disks?

 

[Guy]

I think I might have solved the issue...

It seems that I was really running out of disk space.

proxmox seems to allow you to create KVM and specify the drives space and does not check to see if you have enough space to expand into it. Thus you can have 100Gb real disk, but still create 4 KVMs each of 100Gb without complaint. Until you fail because a KVM starts to try and take space which no longer exists.

I think this is a BIG bug!

--Guy

 

[Dave]

Hmm, that's not the case for me, I still have over 300Gb left. Let us know if that sorts the problem for you though.

Also one important point that I forgot to mention was that before doing a rebuild with the latest release the VM's had been working fine. I can't remember which version I was running other than that it was the latest fully patched pre-1.1 (build 3664) release.