Proxmox, OpenVZ, Debian and Ubuntu Kernel Issues

[Ray]

Hello @all,

I want to discuss some issues I have right now, first I have stability problems with 2.6.32 and lvm not removing backup-snapshots, hanging uninterrubtable and the KVM in question inaccassable, leaving as only solution a reboot of the system -> Inaccaptable (see other tread of mine regarding this issue).

Going back to 2.6.18 helped and I do not experience this severe problem, but one of the problems with 2.6.18 is the missing fd_signal stuff, which in the OpenVZ 2.6.18 kernel is backported but up until now proxmox was not able to incorporate this kernel due to some kvm incompatibilities ?!

Point is, I am very worried about the future, especially since it seems that Ubuntu (rumors are debian as well) will not support OpenVZ kernels officially in the next upcoming release, which for Ubunut means the LTS Release ...

There rumoring about dropping OpenVZ in favor of LXC.

Which, for me, would be fine if they would do it after this LTS Release, leaving us with an upgrade path of around 2-3 years and, more importantly, gives some time that LXC could mature.

LXC is nice, and best of all, is integrated in the kernel, but misses critical features like stable checkpointing and therefore live-migration.

So, what does this mean for "Container"ization in 2010??

2.6.18 is definitly to old for the new and shiny distros (most of the problems is the switch to upstart, which is a godsend, but needs more recent kernelfeatures). 2.6.32 seems to be still a bit buggy, and what's going on with OpenVZ ...

Maybe someone can shed more lights on this or has someone a stable playground with LXC, comparable to OpenVZ??

best
Ray

 

[tom]

Hello @all,

I want to discuss some issues I have right now, first I have stability problems with 2.6.32 and lvm not removing backup-snapshots, hanging uninterrubtable and the KVM in question inaccassable, leaving as only solution a reboot of the system -> Inaccaptable (see other tread of mine regarding this issue).

Going back to 2.6.18 helped and I do not experience this severe problem, but one of the problems with 2.6.18 is the missing fd_signal stuff, which in the OpenVZ 2.6.18 kernel is backported but up until now proxmox was not able to incorporate this kernel due to some kvm incompatibilities ?!

pls give more details about this.

Point is, I am very worried about the future, especially since it seems that Ubuntu (rumors are debian as well) will not support OpenVZ kernels officially in the next upcoming release, which for Ubunut means the LTS Release ...

The OpenVZ team is working to get into Squeeze (2.6.32). For Ubuntu 10.04, its too late as already freezed.

There rumoring about dropping OpenVZ in favor of LXC.

not in the near future. but at the end, you will have a full featured container solution in the mainline kernel. more or less all container projects are (and should) working on this goal.

Which, for me, would be fine if they would do it after this LTS Release, leaving us with an upgrade path of around 2-3 years and, more importantly, gives some time that LXC could mature.

LXC is nice, and best of all, is integrated in the kernel, but misses critical features like stable checkpointing and therefore live-migration.

So, what does this mean for "Container"ization in 2010??

2.6.18 is definitly to old for the new and shiny distros (most of the problems is the switch to upstart, which is a godsend, but needs more recent kernelfeatures). 2.6.32 seems to be still a bit buggy, and what's going on with OpenVZ ...

Maybe someone can shed more lights on this or has someone a stable playground with LXC, comparable to OpenVZ??

best
Ray

I run 2.6.18 for all OpenVZ production systems as this is the only stable OpenVZ release. OpenVZ for 2.6.32 is still experimental (therefore not yet available in Proxmox VE), lets see how fast it would go to a stable and feature complete release.

I am also interested in feedback from people running LXC.

 

[Ray]

The OpenVZ team is working to get into Squeeze (2.6.32). For Ubuntu 10.04, its too late as already freezed.

Which leaves us with an Ubuntu LTS without supported kernel, i wonder how good the RedHat 2.6.32 kernel will fit in the ubuntu LTS ... While i had no problems until recently using other distros kernels i guess in the future might be dependencies on kernel-features, which can make it harder to deploy something in this way or another.

not in the near future. but at the end, you will have a full featured container solution in the mainline kernel. more or less all container projects are (and should) working on this goal.

pls give more details about this.

Not in the near future, well, it's defintly near for ubuntu:

--- snip of the blueprint of lucid (next ubuntu LTS) containerization ---
Summary
Containers offer a simple way to confine processes to a particular namespace. LXC is the container technology in the upstream kernel, and it provides an excellent upgrade path for people using the openvz kernel we had in Hardy. It does not require special hardware support, so in situations where kvm doesn't work it might offer people a saner alternative to Xen
--- snip ---

Here is the Lucid Containerization Blueprint: https://blueprints.launchpad.net/ubuntu/+spec/server-lucid-contextualization
Here is the more detailed specification: https://wiki.ubuntu.com/ContainersSpec

which states: OpenVZ is officially dead (in ubuntu), Long live LXC.


Problems with 2.6.18 and new distros:
https://bugs.launchpad.net/ubuntu/+source/mountall/+bug/436130
https://bugs.launchpad.net/ubuntu/+source/netbase/+bug/431065

The solutions therein don't work for everybode, I myself have a Lucid container which is not working 100% correctly even if I implement all the suggestions in this bugreports. I still have one open console with an errormessage (which of course I am not able to do anything about because it's inside OpenVZ and I do not have access to this console). But if I kill it it's shut down my vps, I did not investigate further because I had not the time to play around, but my aducated guess is that fd_signal patch, in the latest released OpenVZ 2.6.18, is one of the missing pieces to get this going.

Reference Information:
http://bugzilla.openvz.org/show_bug.cgi?id=533

I run 2.6.18 for all OpenVZ production systems as this is the only stable OpenVZ release.

I would be interested in you testing a recently lucid beta (2 i guess) inside a container and hear about your experiences.

 

[tom]

Ubuntu is not really container friendly (Hardy was the last trouble free), most people are running either Redhat (or Redhat clones like Centos) or Debian (on the hosts and in the guests). The OpenVZ support for Hardy was quite a bad story, officially announced by OpenVZ or Ubuntu (or both, can´t remember) but not supported or updated after-wards. All people moved to Ubuntu on the host have to cancel and move back to a more stable distro (Redhat or Debian). A good example of how to anger admins and wasting time.

Lucid in a container? Not tried yet. We have already a Debian Squeeze template, running well (just the reboot issue, not critical) - so why going for Lucid as a OpenVZ container as they "officially" announced OpenVZ is dead - without having a real usable replacement.

I really like Ubuntu on the Desktop and I am using it on my netbook and notebook, but for servers I take the original (Debian).

(I can´t find info about the "fd_signal patch" in the latest 2.6.18 OpenVZ release notes)

 

[Ray]

sorry, wasn't the latest, was this:
http://wiki.openvz.org/Download/kernel/rhel5/028stab068.3

is this patch incorporated now in proxmox 2.6.18 kernel?


Ubuntu: As a matter of fact, I have no real choice, we work a lot with government and bigger institutions and they require a support contract from the vendor, so for new projects it's either Redhat or Ubuntu, and I really love administration on an Debian/Ubuntu (well an old dog does not like to learn new tricks, having now more than 8 years expierence with debian ...).

And as a matter of fact, I like the LTS release cycle from ubuntu, you have something to plan for, you can tell your clients look, we take LTS, we are supported hasta 2015, so we plan a migration to the next LTS in 2013 (when the next LTS is one year old), switch in end 2013 and are free to go for another few years.

When i was jounger I did not understand this concept very well, i was under the impression that debian is the best for everyone and did not understand why businesses don't use it, but having to calculate costs for at least 1 upgrade-cycle to come, and having a timeline you can depend on makes your live easier in big-deployments where you only get the money when you have requested it before in a projected manner ....

update: not to imply you are young or you don't understand this, I wrote this mearly for other people reading the thread and wondering why sometimes the use of a structered release cycle is important.


best
Ray

 

[tom]

sorry, wasn't the latest, was this:
http://wiki.openvz.org/Download/kernel/rhel5/028stab068.3

is this patch incorporated now in proxmox 2.6.18 kernel?

See http://forum.proxmox.com/showthread.php?p=17119#poststop

Ubuntu: As a matter of fact, I have no real choice, we work a lot with government and bigger institutions and they require a support contract from the vendor, so for new projects it's either Redhat or Ubuntu, and I really love administration on an Debian/Ubuntu (well an old dog does not like to learn new tricks, having now more than 8 years expierence with debian ...).

And as a matter of fact, I like the LTS release cycle from ubuntu, you have something to plan for, you can tell your clients look, we take LTS, we are supported hasta 2015, so we plan a migration to the next LTS in 2013 (when the next LTS is one year old), switch in end 2013 and are free to go for another few years.

When i was jounger I did not understand this concept very well, i was under the impression that debian is the best for everyone and did not understand why businesses don't use it, but having to calculate costs for at least 1 upgrade-cycle to come, and having a timeline you can depend on makes your live easier in big-deployments where you only get the money when you have requested it before in a projected manner ....

update: not to imply you are young or you don't understand this, I wrote this mearly for other people reading the thread and wondering why sometimes the use of a structered release cycle is important.


best
Ray

Just to mention: Debian has a "structured" release cycle but a lot of people have problems handling this. It´s really bad that especially big institutions, mainly governmental, always think that this is an issue, in fact its a feature (release it only if it is ready and stable).

For a lot of people its hard to understand that a community project like Debian can survive and therefore they through their money mainly to North American companies (MS but also Linux) - Just do a small calculation about the whole money moving from Europe economy to USA just for OS and Office? You can see what is possible with just a few millions of dollars per year - Ubuntu.

Do you know the Limux project? A good example how Debian can be used in big governmental organizations (town of munich) - they realized that Debian is best suited for their needs.

 

[Ray]

Just to mention: Debian has a "structured" release cycle but a lot of people have problems handling this. It´s really bad that especially big institutions, mainly governmental, always think that this is an issue, in fact its a feature (release it only if it is ready and stable).

For a lot of people its hard to understand that a community project like Debian can survive and therefore they through their money mainly to North American companies (MS but also Linux) - Just do a small calculation about the whole money moving from Europe economy to USA just for OS and Office? You can see what is possible with just a few millions of dollars per year - Ubuntu.

Do you know the Limux project? A good example how Debian can be used in big governmental organizations (town of munich) - they realized that Debian is best suited for their needs.

I personally know all (what I consider for myself) pro's and con's about debian. I'm also interestingly following the Munich project since the beginning in 2005 (was it 2005?) I know why there are delays and that these delays have perfectly good reasons and that a lot what was buried deep was dug up and now they are able to fix not only the localized IT problems but their entire IT structure througout all IT departments. So, for munich it was a double win.

Point is not every city and not every government is like munich, and sometimes fighting like Don Quixote does not bring you a contract.

The only problem I have with the debian structured release its that you can not calculate it. I know i know, it's better, more stable, etc. but right now, today, a lot of businesses do not understand it and work in very restrained environments. That's why we have Ubuntu afterall.

One of my personal strategies is to deploy ubuntu and later on, if after a year the time comes where you have to through out lot's of money for the contracts again, try to have an open dialog with my customer about this and sometimes we can deploy debian afterwards, but this is not normal business withhin our normal clientel.

But in the end, it's more philosophical than anything else ... My best educated guess is our debates right now won't matter in 250 years anymore

best
Ray

 

[Ray]

Hi Tom,

I am not sure why you got me the link to the forumspage with the relaese of 1.5

in 1.5 you have Openvz 2.6.18 kernel 028stab067.4

whereas the fix for fd_signal is in 028stab068.3:

Changes Since 028stab067.4:
Rebase to 2.6.18-164.11.1.el5 kernel: a number of security and bug fixes (see links to RHSA and CVEs below)
Improvements:
Support for signalfd() syscall (OpenVZ Bug #1415)
Bug fixes for the following issues:
A kernel panic may occur while reading /proc/bc/0/ioprio_queues
A kernel panic may occur while unregistering a VLAN device inside a container if the device was registered on the Hardware Node
Network devices leakage may occur
OpenVPN fails to change tx queue length on tun0 inside a container
The number of running processes (procs_running in /proc/stat) may be reported incorrectly inside a container


best
Ray

 

[tom]

Thanks for sharing your experiences here, we could discuss the whole day about all these quite interesting topics.

 

[tom]

ups, you are right, my fault. we had some issues with the latest OpenVZ kernels and KVM, therefore we did not released them. but as soon as we got them we will update our 2.6.18 kernel.

 

[Ray]

Ok, I hope you'l sort out the problems with the KVM soon as I'm really eager to test the new 2.6.18 kernel.


I am not affiliated with the blog, it just landed right now in my inbox and fit's the topic and represents the thinking of lots of my client (regarding to having a date to depend on (and calculate for)):
http://blogs.techrepublic.com.com/hiner/?p=4028&tag=nl.e101

best
Ray

 

[tom]

This guy is looking on the headlines only - and yes, it looks like most customers are doing this also, sadly.

We know that writing "LTS" or "stable" beside a release does not help in real live. Example is OpenVZ in Ubuntu Hardy LTS, there are more.

 

[dietmar]

whereas the fix for fd_signal is in 028stab068.3:

Yes, but that kernel version have serious problems with timers (KVM) when the openvz patch is applied - so we decided to not release that.

 

[Ray]

Yes, but that kernel version have serious problems with timers (KVM) when the openvz patch is applied - so we decided to not release that.

Hi Dietmar,

so, this was 3 OpenVZ releases ago, do the current OpenVZ 2.6.18 kernels have the same problem?

best
Ray

 

[dietmar]

so, this was 3 OpenVZ releases ago, do the current OpenVZ 2.6.18 kernels have the same problem?

Yes, the bug is still there. You can find an updated kernel in the pvetest repository.

 

[Ray]

Ok, is there somewhere a description to the latency problem with the KVM in this kernel? does it effect all guest os or is it specific to some os's like windows?

thank you
best

 

[dietmar]

Ok, is there somewhere a description to the latency problem with the KVM in this kernel? does it effect all guest os or is it specific to some os's like windows?

Simply try a w2k8 install - mouse movement does not really work. It is also observable with other OS.

 

[dietmar]

To show the behavior I use the following command (you need X11 for that test):

kvm -drive file=ubuntu-9.04-desktop-amd64.iso,if=ide,index=2,media=cdrom -m 512 -net none -clock dynticks