I/O stalls with one lvm machine, only restart solves

[obrienmd]

I have a single virtual machine in proxmox, which was originally a qcow2 file but was subsequently converted to an LVM volume (to a raw file first, the dd'd to the lvm volume).

Ever since the conversion to lvm, the machine will stop responding. CPU stays super-low, but iostat shows 100% usage of the LVM volume with < 1kbps read and/or write. I/O on every other device associated with that PV/VG also slows to a crawl.

Attempting to kill the kvm process, even kill -9, just results in a kvm <defunct> process, and I/O stall remains. Only a full reboot remedies the issue (until it happens again).

Converting the LV back to a qcow2 file solves the problem, but I much prefer LVM (I've used it with other proxmox hosts/guests) for performance and backup reasons.

The host is:
Proxmox 1.4
Q6600
16GB DDR2
hw RAID10 8x640GB SATA
2x e1000 nics

The guest is:
Ubuntu 8.04.3 LTS
Virtio /dev/vda = 42GB (I've also tried IDE/SCSI)
e1000 NIC

Thoughts? Absolutely nothing in logs (dmesg, kern, messages, syslog, lvm2, etc)

 

[dietmar]

Is there a way to reproduce that problem?

 

[obrienmd]

It's very easy to reproduce - Start the VM in question after moving it back to the lvm volume for HD (from qcow2 or raw file), then just wait... It's not consistent, but usually in less than an hour any and all i/o on that pv/vg will completely stall, with 100% usage on the VM in question for <1kbps transactions in iostat.

 

[hisaltesse]

Our of curiosity, what kind of bandwidth do you get with your hw RAID10 8x640GB SATA?

What do you get running pveperf on no load and with your current load?

 

[dietmar]

It's very easy to reproduce - Start the VM in question after moving it back to the lvm volume for HD (from qcow2 or raw file), then just wait... It's not consistent, but usually in less than an hour any and all i/o on that pv/vg will completely stall, with 100% usage on the VM in question for <1kbps transactions in iostat.

Sorry, I can't reproduce that here.