Stability issues with KVM-83 is rollback to 75 available?

[bitbud]

While I am glad to see movement with the KVM versions, I am finding issues with the latest update. KVM-83 has some reported issues with networking on SMP systems (which I have a lot of). I am seeing the same issues, with network drops and strange ping response times. There has been some discussion on a fix in a future release, as this is a regression and it should be easy enough to fix.

To work around, I am installing from the Ver 1.0 ISO of Proxmox. The KVM version (75) with it works fine.

So, it leads me to a few questions:

- I'd like to move to 1.1, as you have some kernel updates in there along with other fixes, but KVM-83 isn't stabilized enough yet. Is their an easy/ recommended way to do that? To move to 1.1, but stay in KVM-75?

- I considered installing 1.0, then updating the 1.1 version and blocking the pve-kvm package from loading, or install 1.1 fresh, remove KVM altogether, and recompiling from source to the version I need. My question is that I note you have customized the kernel. Are you compiling KVM into the Kernel, or using any other configuration that would cause an issue if I used the 1.1 release of Proxmox, but compiled KVM-75 (or other version) from source?

- As KVM is a fast moving project, these types of issues are sure to come up. It would be great to have a simple way to pick from the available releases. Perhaps compiling from source is the best option? Your thought?

Any suggestions are appreciated.

And keep up the great work. Proxmox VE is an important project.

 

[tom]

While I am glad to see movement with the KVM versions, I am finding issues with the latest update. KVM-83 has some reported issues with networking on SMP systems (which I have a lot of). I am seeing the same issues, with network drops and strange ping response times. There has been some discussion on a fix in a future release, as this is a regression and it should be easy enough to fix.

To work around, I am installing from the Ver 1.0 ISO of Proxmox. The KVM version (75) with it works fine.

So, it leads me to a few questions:

- I'd like to move to 1.1, as you have some kernel updates in there along with other fixes, but KVM-83 isn't stabilized enough yet. Is their an easy/ recommended way to do that? To move to 1.1, but stay in KVM-75?

no easy way.

- I considered installing 1.0, then updating the 1.1 version and blocking the pve-kvm package from loading, or install 1.1 fresh, remove KVM altogether, and recompiling from source to the version I need. My question is that I note you have customized the kernel. Are you compiling KVM into the Kernel, or using any other configuration that would cause an issue if I used the 1.1 release of Proxmox, but compiled KVM-75 (or other version) from source?

You can try to compile the new Kernel with the KVM-75. All our sources are here:
ftp://pve.proxmox.com/sources/. but there are a lot of dependencies and it will be a very long way to get this running. Why not just running with the 1.0? Whats the issue here?

- As KVM is a fast moving project, these types of issues are sure to come up. It would be great to have a simple way to pick from the available releases. Perhaps compiling from source is the best option? Your thought?

Any suggestions are appreciated.

And keep up the great work. Proxmox VE is an important project.

KVM is not 100 % stable in all situations - but getting better and better with every release - And we will release newer KVM packages as soon as we think the improvements are important enough to update Proxmox VE - and after we tested these new packages.

 

[tog]

While I am glad to see movement with the KVM versions, I am finding issues with the latest update. KVM-83 has some reported issues with networking on SMP systems (which I have a lot of). I am seeing the same issues, with network drops and strange ping response times. There has been some discussion on a fix in a future release, as this is a regression and it should be easy enough to fix.

Hmm, I can't say my network traffic is really really heavy on my two KVM VMs but they are working nicely. One's FreeBSD 6.3 and one's Win2k Server. Neither are heavily loaded but they are fairly steadily loaded.

They are both set to use only one CPU and the host is an 8-way Xeon. So far so good over the last 6 days with KVM-83. You aren't trying to use more than one CPU in your KVMs, are you?

 

[bitbud]

Yes - this is an SMP issue

The issue is specifically when running SMP guests - that is, with multiple CPUs. It looks like it may just be a regressions on AMD processors.

It is a KVM issue, and will get eventually get resolved. I really only posted the question here as I prefer to use Proxmox due to the interface and features it adds, and as such was looking at recommendations on using alternate versions of KVM on Proxmox, as opposed to just the released version.

 

[tog]

Ah well, as Tom said, it is difficult to use a different version of KVM just because the Proxmox VE kernel goes along with the rest of the packages.

I definitely would NOT recommend using an SMP guest in KVM, just stick to using one CPU and enjoy the stability. When I tried two CPUs for my Win2k server guest I actually froze the entire PVE host once and had to go reset it

...and that was with KVM-75 which you seem to believe should be stable with SMP guests.

 

[dietmar]

yes, kvm-75 is not stable with windows SMP guest. kvm-83 is better, but still not perfect.

 

[bitbud]

No issues here

We don't run SMP guests without a reason, it is a requirement, so just switching to a single core is not an option.In fact, I am running more that 10 Win 2003 and 2008 SMP guests. Both 32 and 64bit with 2 or 4 processors. This is on Proxmox with KVM-75.I am curious what stability issues you have had.Thanks.

 

[tog]

I messed around with it for about 20 minutes with KVM-75 with a Win2k server guest and it was all kinds of weird. When I used two CPUs for the KVM, Win2k would take a looooong time to boot and finally even hung the PVE host when trying to restart the guest and required a hard reset of the physical server.

I changed back to single CPU and it was perfectly fine and has been since.

I think when I briefly tried a WindowsXP 32-bit guest with 2 CPUs it seemed to boot very very slowly but worked all right once it got booted up... have not tried it with KVM-83, though.

If you have to use 2 and 4 CPU guests and it was working with KVM-75 it's probably easiest and best to just stay back with Proxmox VE 1.0 for now until more KVM releases happen. I was perfectly happy running Proxmox VE 1.0 and I wouldn't be too sad if I had to run it for 6 more months or whatever without updating.

 

[bitbud]

Found solution

I found the solution.

The issue is specific with Windows SMP and AMD processors. To resolve, you simply need to make a small change to the VM Guest BOOT.INI file.

For the OS you select, add /usepmtimer to the line in the BOOT.INI file (on Windows computer c:\).

This fixes the issue. Now using 4 CPU guests with 4 GB RAM, no problems.

 

[mangoo]

KVM is not 100 % stable in all situations - but getting better and better with every release - And we will release newer KVM packages as soon as we think the improvements are important enough to update Proxmox VE - and after we tested these new packages.

What options do you normally use to compile kvm?

Do you apply any patches to qemu/kvm?

I tried to compile kvm-84, as it potentially solves one issue I have.

Unfortunately, it doesn't seem to work with Proxmox VE interface - when I want to start a guest, it fails with:

/usr/bin/kvm: invalid option -- '-id'

 

[dietmar]

Our kvm build is inside this package:

ftp://pve.proxmox.com/sources/pve-kernel-2.6.24_2009-01-15.tar.gz

But I will build a kvm-84 version next week anyways.

- Dietmar

 

[mangoo]

Our kvm build is inside this package:

ftp://pve.proxmox.com/sources/pve-kernel-2.6.24_2009-01-15.tar.gz

But I will build a kvm-84 version next week anyways.

I see, and there are KVM patches there as well.
Two of the fail to apply cleanly; I didn't check if it's just some code differences that need to be re-edited, or something more serious.

 

[mangoo]

Our kvm build is inside this package:

ftp://pve.proxmox.com/sources/pve-kernel-2.6.24_2009-01-15.tar.gz

But I will build a kvm-84 version next week anyways.

Hi,

Did you manage to build kvm-84?

 

[dietmar]

No, sorry - there are too many bug reports for this versions, so we decided to wait for the next release.

- Dietmar

 

[mangoo]

No, sorry - there are too many bug reports for this versions, so we decided to wait for the next release.

Or, it means that KVM is getting more and more popular, and just more user reports come to the surface

Anyway, I'll try to modify and apply your patches to KVM-84 and see if it works stable for me or not.

 

[dietmar]

Or, it means that KVM is getting more and more popular, and just more user reports come to the surface

Ah, yes - hopefully thats the reason ;-)

Anyway, I'll try to modify and apply your patches to KVM-84 and see if it works stable for me or not.

Please do (I am currently on vacation) and tell us the result.

- Dietmar

 

[mangoo]

Ah, yes - hopefully thats the reason ;-)

Please do (I am currently on vacation) and tell us the result.

- Dietmar

I'm running kvm-84 for a little bit over a week now.

It runs stable for me with 9 Linux guests with:
- different kernels
- virtio drivers
- scsi + e1000
- IO/CPU loaded to a different degree


It fixes one severe issue I had with kvm-83: after running for a couple of days, network in my some of the guests was getting extremely slow. Restarting the guest didn't help (i.e., "reboot" when logged in the guest); the only thing that helped was restarting kvm process (i.e. stop the guest, start it again).
I think the problem affects some AMD CPUs, so not everyone will see this issue (I've seen some complaints about network performance on the Proxmox VE list though).


If you're reading kvm-devel list, see this thread:

http://thread.gmane.org/gmane.comp.emulators.kvm.devel/29132/focus=29329


I hoped kvm-84 will allow me to use cpufreq/ondemand, but when I enable it, after a couple of days, some guest's network gets slow - the same issue I had with kvm-83 without even enabling cpufreq/ondemand. With kvm-83 and enabling cpufreq/ondemand, guest's clock is totally unreliable - time in one of my guests jumped back to year 1953:

http://thread.gmane.org/gmane.comp.emulators.kvm.devel/28549


So from my perspective, kvm-84 is an improvement when compared to kvm-83 - I have no more network problems with guests.

 

[amravit]

could u advice how to build the kvm or a link you used while doing it
thanks

 

[mangoo]

I'm running kvm-84 for a little bit over a week now.

It runs stable for me with 9 Linux guests with:
- different kernels
- virtio drivers
- scsi + e1000
- IO/CPU loaded to a different degree


It fixes one severe issue I had with kvm-83: after running for a couple of days, network in my some of the guests was getting extremely slow. Restarting the guest didn't help (i.e., "reboot" when logged in the guest); the only thing that helped was restarting kvm process (i.e. stop the guest, start it again).
I think the problem affects some AMD CPUs, so not everyone will see this issue (I've seen some complaints about network performance on the Proxmox VE list though).


If you're reading kvm-devel list, see this thread:

http://thread.gmane.org/gmane.comp.emulators.kvm.devel/29132/focus=29329


I hoped kvm-84 will allow me to use cpufreq/ondemand, but when I enable it, after a couple of days, some guest's network gets slow - the same issue I had with kvm-83 without even enabling cpufreq/ondemand. With kvm-83 and enabling cpufreq/ondemand, guest's clock is totally unreliable - time in one of my guests jumped back to year 1953:

http://thread.gmane.org/gmane.comp.emulators.kvm.devel/28549


So from my perspective, kvm-84 is an improvement when compared to kvm-83 - I have no more network problems with guests.

BTW, this issue (slow network after some time) is a weird bug in virtio_net and it happens with both kvm-83 and kvm-84 (and prehaps older releases); virtio_blk is not affected.