VM hard disk corruption

[jordank]

My Setup
4x Virtual Hosts w/ 2x Quad core and 30gb ram running Proxmov VE
1x RHEL5 file server w/ 6X 73gb 15000rpm scsi drives raid6
1x RHEL6 backup file server 2x 2tb drives software raid0

I have a few virtual machines running on this cluster and they are mixed windows and linux vm's. I have been having some weird issues with the linux vm's, the hard drive image on a few of the seems to get corrupted every once in a while.

I had a RHEL6 vm and had given access to a user and every few weeks he reports to me that the vm filesystem is read only. At that time it was only that vm doing it, so I fixed the filesystem but booting a live cd, mounting the system and running a disk check. Unfortunately the problem kept comming back and could not be easily fixed.

So I changed this vm to Ubuntu 10.04 server thinking that it might work better. The problem is back again, but this time it can be fixed with a reboot and it checks the disks and fixes "orphaned inodes". I am now having this issue on another RHEL vm and my Backtrack VM. What is weird about this is that this problem only affects the linux vm's, not any of my Windows vm's.

All of my vm hard disk are in raw format, could changing to qcow2 help? What are the major advantages of qcow2? Does anyone have anythoughts and things to look for to fix this issue? I have tried looking for things in the log files and have not had much luck(probably because the filesystems are read only, lol).

Thanks
Jordank

 

[udo]

Hi Jordank,
on what kind of storage are the disk-files? Are the disks on the file-server? How mounted? NFS?
Does the same happens with local-storage?

I would not recommend qcow2 - you have one more layer between VM and real disk.

Ubuntu have an extra kernel to be an good virtual guest - do you use them?
Perhaps you can try to modify your cache-parameter in the vm-config (append in the vm-config in the storageline ",cache=writethrough").

Udo

 

[jordank]

on what kind of storage are the disk-files? Are the disks on the file-server? How mounted? NFS?
Does the same happens with local-storage?

The vm's are all stored on my RHEL5 fileserver with the stats listed in my first post. The fileserver has an llvm setup and runs either ext4 or more likely ext3 and I have this all exported to the network via nfs. The network that I have this on is a few nice managed gigibit switched. I have not tried local storage on these because I needed the ability to live migrate the vm's and to be able to easily bring the vm's up in case of virtual host failure, I can test it out though.

Ubuntu have an extra kernel to be an good virtual guest - do you use them?

I had not thought of that but it is easy enough to setup. Any recommendations for the RHEL vm's?

Perhaps you can try to modify your cache-parameter in the vm-config (append in the vm-config in the storageline ",cache=writethrough").

I can give that a try.