Removal of KSM in 1.6 was a bad idea (my opinion)

[yatesco]

Hi Dietmar and Tom,

First off - I am not moaning - this is meant as constructive criticism. I was going to email you but I think this forum (given the recent posts) is a more appropriate place.

First off, I think Proxmox is great. The work you have done is fantastic and I am very appreciative. I use it for production (but no, I haven't given you guys any money yet )

Anyway, my point is that I think the removal of KSM in 1.6 was a very very bad idea for a couple of reasons:

- I imagine there are a lot of people out there like myself who use KVM exclusively and make use of KSM
- Those who are using KSM may now find out that their servers have inadequate physical memory. For example I utilise KSM quite heavily as most of my VMs are Ubuntu 10.4. I need to think very carefully and calculate the memory usage before I decide whether I can upgrade or not.
- Essentially you have taken a step backwards in terms of functionality of the product. I am using 1.5 very happily, now I don't know if I can use 1.6 (because of the removal of KSM). This is strategically a big no-no. Products should be perceived to improve, not step backwards.

I think a better solution would have been to *not* release this to production *or* to split into three streams - a KVM (and KSM) one, an OpenVZ one and a mixed, without KSM one (i.e. the one you have currently).

I entirely understand why this isn't feasible - who wants to maintain that many branches - I know I wouldn't .

I wonder if you should do another poll to identify how many people:

- use only OpenVZ
- use only KVM with KSM
- use only KVM without KSM
- use both *on the same host*

If there aren't that many in the last one then happy days, two kernels (a KVM with KSM one and an OpenVZ one) would be sufficient.

Please don't take this the wrong way, I just wanted to give you an honest opinion of how your actions have been perceived (by me at least).

Flame away - I hope you don't, and I really don't want anyone to reply to this thread adding their complaints. I hope you guys make a sensible response and this will be the end of all the related moaning on the forums.

Alternatively, just ignore this and keep working on 2.0 - I want that now!

 

[m.ardito]

they already stated on the forums that there will be a 1.6/2.6.32/kvm only (with ksm) kernel later...
if you NOW already use ksm then you are already not using openvz on the same server, so for now just keep 1.5/2.6.32 AND ksm. otherwise bty more ram, having also quite cpu horsepower back, too.

who runs openvz (& maybe kvm but has ram enough) NEEDED a newer kernel than 2.6.24, and pve being able to merge two worlds (kvm and openvz) a distintive and quite unique feature, pve staff aimed at that. great!

i really don't understand pushing these great people just 2 days after 1.6 release...

Marco

 

[Lantizia]

I'm thrilled we finally have 2.6.32 out with both KVM/OpenVZ support as it should fix the fake swap issues that OpenVZ had for anyone ever having Java memory overcommitment issues.

Edit: Nevermind on everything else... I just caught up.

 

[m.ardito]

Openvz DOES NOT SUPPORT ksm, period. So they say, you guys, read a few posts before asking?

Marco

 

[yatesco]

if you NOW already use ksm then you are already not using openvz on the same server, so for now just keep 1.5/2.6.32 AND ksm.

That is my point - if I cannot use the next release of a product because it has removed working functionality then that, by definition, is a step backwards. For example, I would be pretty miffed if the next version of Microsoft Word could no longer open any documents created by a previous version of Word. If they did do that then they should at least make it a major version jump and stick lots and lots of very obvious warnings.

I understand the technical requirements for the various releases - my point was that I don't think they handled this situation very well. How many people really read release notes? By convention; "a minor release (i.e. 1.X to 1.X+Y) should be "safe" - if it worked before it should work now, unless the reason it worked before was because of a bug which has now been fixed". This is the industry standard definition which will lead to some people doing an upgrade and then realising their VMs no longer work because the server doesn't have enough memory. This isn't a bug, it is by design(!). Minor upgrades should *do no harm*. That is not the case here.

i really don't understand pushing these great people just 2 days after 1.6 release...

My post wasn't intended to "push" them (I assume you mean "having a go") - as should be clear from the multiple statements of praise in that post (and go and read my other posts as well - I love these guys! ). But, I do think they dropped the ball on this one.

My goal is to prevent this kind of thing happening again, that is all.

Go go go proxmox dudes (and duderinos ).

 

[Lantizia]

I don't think the release notes should have said KSM was removed... not because KSM should have been included but because stating that simply makes no sense.

Proxmox VE have time and again said that the kernels are branched off in to the...

KVM Only
KVM/OpenVZ

So the kernel that is bundled with 1.6 is a KVM/OpenVZ kernel... and the last KVM/OpenVZ kernel didn't have KSM either. So it's not really removed.

The most current KVM Only kernel did have KSM, and by the sounds of it the next KSM Only kernel will too.

So it's not really removed at all... it's just a different kernel branch that this product can use.

Personally I can't wait to see a version of Proxmox VE with LXC support - and finally ditch this OpenVZ shit that isn't in the mainline kernel.

Roll on version 2.

 

[yatesco]

However we skin it, I still think they should have handled this better.

I would argue that it is a different *product* not a different version of the same product. This might be considered semantics, but it is pretty important. I don't expect to apt-get upgrade to a different product, but I absolutely do expect an apt-get upgrade to be safe, which in this case it wasn't.

It is a question of definition and it would make more sense (and would ensure consistency with apt-get and standard terminology) if the team treated them as two separate products, i.e. proxmox-KVM and proxmox-KVMOpenVZ.

Goodness help those poor people who have enabled automatic updates...

 

[dietmar]

I understand the technical requirements for the various releases - my point was that I don't think they handled this situation very well. How many people really read release notes? By convention; "a minor release (i.e. 1.X to 1.X+Y) should be "safe" - if it worked before it should work now, unless the reason it worked before was because of a bug which has now been fixed".

Ok, so let me ask you something. You have done any benchmarks that clearly show that a host with KSM perform any better than a host using a bit more SWAP? If so, please post your results.

 

[meto]

Ok, so let me ask you something. You have done any benchmarks that clearly show that a host with KSM perform any better than a host using a bit more SWAP? If so, please post your results.

I don't think there are those benchmarks, and you are aware of it. I could bet any money that those 10-20% RAM saved thanks to KSM would be MUCH faster than the same amount of swap on disks without hardware raid (not even talking about writeback-cache). Disk's IO is just sooooo slow.
I'm using virtualized and very busy LAMP stack and have SWAP disabled because it killed performance in minutes. Also I don't use swap in VMs. If it's not in ram then it's not going to work

 

[yatesco]

Ok, so let me ask you something. You have done any benchmarks that clearly show that a host with KSM perform any better than a host using a bit more SWAP? If so, please post your results.

Hi Dietmar,

Nope, I haven't for a couple of reasons:

- my servers have loads of free CPU capacity and I expect (although haven't verified!) that KSM performance being CPU and memory bound will be much faster than utilising swap on the host which is disk IO bound
- my servers have a tiny hard disk and all the storage is off the SAN so I couldn't give it more than a couple of GBs of swap anyway. I certainly don't want the host using the SAN for swap space.....

Whether the performance was better or not isn't really the point .

 

[svenr]

There is a really very simple solution for this: Wait with the upgrade untill the kvm-only-kernel is available, there are no critical bugs in 1.5 which need an upgrade ...
so there is no reason for this discussion, because there will be a kernel with ksm and i´m sure that this kernel will be released when ready ;-)

sven

btw. on production systems you always wait some days with the upgrade after a new release

 

[yatesco]

so there is no reason for this discussion, because there will be a kernel with ksm and i´m sure that this kernel will be released when ready ;-)
btw. on production systems you always wait some days with the upgrade after a new release

Please read my post again (sigh). The point is that if I did an upgrade now (or in a few days whenever) my VMs would not start because there is not enough memory or swap. That cannot correct - upgrades should do no harm.

If I have to wait a different variation of the product has been released (which I do in this case) then that breaks the convention of apt and release management.

 

[meto]

Please read my post again (sigh). The point is that if I did an upgrade now (or in a few days whenever) my VMs would not start because there is not enough memory or swap. That cannot correct - upgrades should do no harm.

If I have to wait a different variation of the product has been released (which I do in this case) then that breaks the convention of apt and release management.

I agree default repo should contain version number such as 1.5 or 1.6 (they do for older releases) so there is no such mess doing normal maintenance.

 

[l.mierzwa]

I agree with yatesco. If someone did instaled proxmox-ve-2.6.32 then he is using 2.6.32 kernel and he runs only kvm virtual machines as 2.6.32-2 does not support openvz, so only people who run 2.6.18 or 2.6.24 needs to upgrade. The best way would be to add new kernel and drop 2.6.18 and 2.6.24 by proving proxmox-ve-2.6.32-openvz or something like that. You would end with 2 kernels instead of 3, people who use 2.6.32 already would not be affected by removing "unsupported" KSM, only those who run 2.6.18 and 2.6.24 would need to upgrade to new kernel. Also You don't need to provide "kvm-only kernel in the future", all You need to do is restore 2.6.32-2 for kvm people.
I don't understand why do You stick to "removing KSM is a feature not a bug", this is very important feature, way more efficient then swap, cpu power is cheaper and faster then disk I/O.
The only solution to get kvm and KSM right now is to compile Your own kernel (which is easy) but I'm not sure if proxmox does not require fairsched to be included in the kernel (I think it was required some time ago).

 

[svenr]

Please read my post again (sigh). The point is that if I did an upgrade now (or in a few days whenever) my VMs would not start because there is not enough memory or swap. That cannot correct - upgrades should do no harm.

If I have to wait a different variation of the product has been released (which I do in this case) then that breaks the convention of apt and release management.

thats another story, maybe waiting for the ksm-version before releasing 1.6 as stable would have been the better way

I agree default repo should contain version number such as 1.5 or 1.6 (they do for older releases) so there is no such mess doing normal maintenance.

thats something i would also like to see

 

[dietmar]

Please read my post again (sigh). The point is that if I did an upgrade now (or in a few days whenever) my VMs would not start because there is not enough memory or swap. That cannot correct - upgrades should do no harm.

You really have such a tight system that you totally depend on KSM? Or is "my VMs would no start" a theoretical issue? (I am just trying to understand the issue).

 

[dietmar]

Whether the performance was better or not isn't really the point .

My point is that you won't even notice any difference an a normal system - that is why we released it. Worst case is that the system gets a bit slower because it starts to use some swap?

 

[yatesco]

You really have such a tight system that you totally depend on KSM? Or is "my VMs would no start" a theoretical issue? (I am just trying to understand the issue).

Yep, I have one production box whose VMs use more than the physical RAM + swap space.

The others don't, but ironically I was planning on doing a re-gig of the VMs to utilise KSM - I have lots and lots of small mostly-bored Ubuntu VMs (web-servers, one-per-client). There are spread onto multiple boxes at the moment but I was going to shovel them all on one. Not now, obviously

 

[yatesco]

My point is that you won't even notice any difference an a normal system - that is why we released it. Worst case is that the system gets a bit slower because it starts to use some swap?

Nope - worst case is that the VMs won't start because the total memory required by the VMs is greater than the physical + swap. This is the case on my server - I have tiny hard disk but lots of SAN.

 

[dietmar]

I don't understand why do You stick to "removing KSM is a feature not a bug", this is very important feature

Nobody claimed such thing here. We simply do not have enough resources to maintain an unlimited number of kernels.