Any news on lxc online migration?

grin

Renowned Member
Dec 8, 2008
177
24
83
Hungary
grin.hu
It's been more than 2 years now since I last asked about the status report of online migration of containers.
Oh, and got no answer, so it's been 5 years since I have heard anything about this from you guys.

Would be nice to know that at least you keep in mind that we're right here waiting.

Thanks!
 
I do not think that it is possible to live migrate a container.

A container are just processes running in a different namespace on the host's kernel.

You cannot freeze a process and transport it to another host and unfreeze it there like you can do with a virtual machine.
 
I do not think that it is possible to live migrate a container.

A container are just processes running in a different namespace on the host's kernel.

You cannot freeze a process and transport it to another host and unfreeze it there like you can do with a virtual machine.

I understand that you don't think that.
Have you ever seen OpenVZ? It also have done the "impossible", and had a working checkpointing and freeze/thaw infrastructure.

Like: https://www.kernel.org/doc/Documentation/cgroup-v1/freezer-subsystem.txt

I guess you never heard of criu either?
 
  • Like
Reactions: exp and guletz
I also remember, half a decade ago that someone have complained about various mount(ed) namespaces and that their freeze isn't well developed in the kernel, but since then a lot of years have came to pass, and it may work.

I probably shall play a little with criu and lxc and see what I get, but it's not the same as our renowned px developers. :cool:
 
  • Like
Reactions: waltar
It was THE most disappointing realization that proxmox cannot do what OpenVZ did more than 10 years ago. Understand, it's not easy but heck, everything needed is out there.

I tend to start processes in screen sessions and it's just not a solution to restart the container. It's frustrating that I'd need a VM just because of that.
 
  • Like
Reactions: Kingneutron
I also remember, half a decade ago that someone have complained about various mount(ed) namespaces and that their freeze isn't well developed in the kernel, but since then a lot of years have came to pass, and it may work.

I probably shall play a little with criu and lxc and see what I get, but it's not the same as our renowned px developers. :cool:
Hi grin,
were you able to use criu to live migrate lxc? I am just starting out with proxmox and testing with my first xlc and saw your post, which is exactly what I needed.

thank you
 
were you able to use criu to live migrate lxc? I am just starting out with proxmox and testing with my first xlc and saw your post, which is exactly what I needed.
Very unlikely.

Proxmox developers do not work neither on this nor with criu developers, they responded that "criu does not work because X, Y and Z", and when I asked whether they want to discuss this with criu developers they did not respons meaningfully; I consluded that they internalised that "criu doesn't work", and "criu cannot work because we think it won't" and they refuse to work with criu developers about this.

Or possibly they do think differently but I was extremely and persistently unlucky in communications with them.
They have this place to prove me wrong, showing that they do want to resolve this problem. I wish.
 
  • Like
Reactions: newuser2025

Yes, he keep implying that he would love to ignore CTs and love VMs, because they are so much easier to handle, okay, slower, okay, crappier, okay, have a lot of problems but man, it's so comfy, especially since PX don't have to develop anything since it's already've been done. :-(
I keep hearing that and I got tired responding.

Also, they usually conjure a very complex environment as an example of impossibility. Changing kernel between save and restore, nice! Even if live migration would usually happen on a homogenous cluster, same kernel, same OS versions. It would be possible to handle just the minority exceptions, so in the usual ("ideal") environment it could just work, but no, there could be a case when it wouldn't so they simply don't support it, only periodically mention how they really loved to support live migration, in a non specified far future point in time. Forgive me for being sour, it's been going on for a decade now.

As for technical points true, there are about a 100 freeze points and about 10 of those can't be reliably restored. They may be detected or not, they may be simply rejected or not, they may be even handled by OS level scripts for what we know, but since it's rejected, nobody looks at it with an eye of "we really have to resolve this, let's actually think about that". Or, like, pay someone to do that, whith the required skill set. Or let them say it's not possible. I've been in these discussions and criu people claimed that many of px statements were not, hm, properly described. But I ain't no postman between them.
https://forum.proxmox.com/threads/proxmox-ve-8-4-released.164821/page-2#post-762577
CRIU doesn't seem to be powerful / mature enough to be used as an option and IMHO seems that Proxmox would have to devel a tool for live migrating an LXC, something that no one has done yet and isn't really that useful in practice.

PX do not have either the resources nor the knowledge to create such a complex thing. Funny that openVZ had it 20 years ago, ain't it?
 
PX do not have either the resources nor the knowledge to create such a complex thing.
Interesting take. the PVE devs have made their position known on the subject, which you decided they "have neither the resources or the knowledge." Lets suppose thats true; why are you speaking as if that is some personal insult to you?

Funny that openVZ had it 20 years ago, ain't it?
Funny indeed. Why are you not using openVZ, but insisting PVE be turned into it?

Its always been POSSIBLE to migrate containers in a manner similar to virtual machines, but no way to plug the massive security holes involved. For anyone serious, Kubernetes represents the modern way to do container clustering- plus many more features not even possible using lxc.
 
  • Like
Reactions: Johannes S
Interesting take. the PVE devs have made their position known on the subject, which you decided they "have neither the resources or the knowledge." Lets suppose thats true; why are you speaking as if that is some personal insult to you?

Because there are at least half a dozen tries behind me to help this to be at least moved forward, and almost every case ended up in silence, or with an example of why was it impossible, instead of something tangible like actually talking with criu people and actually trying to examine the problems, and maybe, just maybe letting the people doing criu development responding the problem instead of answering those instead of them from the sideline. And, as I mentioned, it's been going on since forever, and in the beginning it was "we're working on it, soon, next version maybe" and then silence for years and when I asked about it it was mainly… silence, or why-nots. I do not remember any case when they would show me any real discussion about the specific problems they mentioned. I got tired and annoyed, and nothing happened since to make me feel better. And I don't bother them about it since otherwise they're nice people, doing useful stuff, that's one reason they get a helluva lot of money from us. [Minor for them, maybe. Who am I to judge?]

Also, I have seen kernel development, and I have seen proxmox development, like, they can't even fix a broken scrollbar for 5 years now, because they used a broken external lib 5 years ago and they can't move over to a nonbroken one. I do not believe that if this is an unsolvable problem for them then a full featured kernel based checkpointing (which has been in development for 10+ years by criu) would be in their league. But I'd love anyone to prove me wrong.

Funny indeed. Why are you not using openVZ, but insisting PVE be turned into it?

Its always been POSSIBLE to migrate containers in a manner similar to virtual machines, but no way to plug the massive security holes involved.

Not in proxmox, no. Well, not since moving over lxc (which wasn't a choice, since OpenVZ was pretty much discontinued.)

For anyone serious, Kubernetes represents the modern way to do container clustering- plus many more features not even possible using lxc.

I understand your opinion. K8s is good for some tasks and horrible for others. (Also it's a kludgy-buggy hell, but maybe I only say that because I looked at what's inside.) People choosing containers (lxc) do it for a reason and it's not the right approach to tell them to use something else you like instead. Same as with VMs, they're good for some tasks and very not good for some others, and they are definitely not lighweight.
I seriously dislike your categorisation, but I can live with that, and the world will survive (that at least) as well.
 
OpenVZ use(d) a heavily patched kernel to accomplish live migration ... criu is a try to do this in userspace .... which at least till now has not the same features & stability - if it's ever possible to do

but yes ... I would like to have it working also in proxmox/lxc .... real multi platform/cloud system container live migrations
 
Last edited:
  • Like
Reactions: Johannes S
Because there are at least half a dozen tries behind me to help this to be at least moved forward, and almost every case ended up in silence, or with an example of why was it impossible, instead of something tangible like actually talking with criu people and actually trying to examine the problems, and maybe, just maybe letting the people doing criu development responding the problem instead of answering those instead of them from the sideline
what of it? the developers of PVE have their priorities. you have yours. if you really have a business case, develop it and contribute to the code.

I understand your opinion.
Its my opinion, sure. it just so happens to be shared with other operators, which is why you dont see any effort to do what you ask.

People choosing containers (lxc) do it for a reason and it's not the right approach to tell them to use something else you like instead.
No one is telling "people" (you) anything. you are the one asking for development for something you want, not the other way around.
 
  • Like
Reactions: Johannes S
> When will live migration on ProxMox for lxc containers re-appear?

Not anytime soon, and it's not unlikely that this will never come for
LXC. Use VMs when you need live-migration anytime soon.

Background:
Containers are to intertwined with the kernel, and while CRIU tries to
provide a solution for serializing a set of processes (like those of
the CT) to a file and be able to load it again, it's quite limited and
that's not to say the CRIU devs don't try, it's an extremely hard
problem. And that gets even harder due to the kernel continuously
changing, gaining new features and changing how the internal state
looks like for existing ones; i.e., even if one would have answers for
every CRIU problem, it still would need a ton of maintenance work to tag
along.

Virtual machines OTOH have a clear state and memory boundary designed
with these requirement in mind, thus it's relatively (!) easy to do
there
-- https://lists.proxmox.com/pipermail/pve-user/2025-November/017697.html