OpenVZ Ploop support

[gkovacs]

And what's this about the RHEL kernel? I'm running a Debian system.

The Proxmox kernel is based on the official OpenVZ kernel, which in turn is built by patching the actual RHEL kernel. The current is based on RHEL6, while RHEL7 is expected sometime next year, so hopefully the OpenVZ team will build their next kernel in 2014. Don't hold your breath though...

For some unknown reason the Proxmox devs want to delay the activation of ploop, as mentioned above by tying it it to the release of the RHEL7 based OpenVZ kernel... I'm not sure why, but it can easily mean another year without ploop.

 

[mathx]

Im interested in ploop because of the journaling IO loads affecting other containers, as well as the snapshotting abilities. Not to mention the live migration iterative loop which uses FAM/inotify style logging (like constant data replicator) to track what was modified during the initial copy and needs to be copied over again. This is because I dont want to use centralized storage for all my nodes as it makes things more complex in my environment. rsync to move a container is great til mysql/postgres expect an inode to be in the right file for writing, but things have shifted. Unless I'm wrong about that - but Im sure some apps/daemons that do direct io will be messing with disk directly and rsync will not be enough of a guarantee.

ploop support is a killer feature that is definitely required, and is one of the major blocking issues for me diving in full bore into proxmox with subscriptions for all my servers. Hope marketing/sales is reading this thread.

please advise - getting a reply from design/sales instead of support might be nice to see what ProxMox's "future vision" is as opposed to the obvious technical answer right now which is "unsupported" (and so far, "likely not to be supported in future" - need more explanation of that statement! it's quite troubling!)


in the meantime how do people find live migrations between seperate boxes on individual local storage with things like postgres busy in the middle of a big write?

 

[gkovacs]

Im interested in ploop because of the journaling IO loads affecting other containers, as well as the snapshotting abilities. Not to mention the live migration iterative loop which uses FAM/inotify style logging (like constant data replicator) to track what was modified during the initial copy and needs to be copied over again. This is because I dont want to use centralized storage for all my nodes as it makes things more complex in my environment. rsync to move a container is great til mysql/postgres expect an inode to be in the right file for writing, but things have shifted. Unless I'm wrong about that - but Im sure some apps/daemons that do direct io will be messing with disk directly and rsync will not be enough of a guarantee.

ploop support is a killer feature that is definitely required, and is one of the major blocking issues for me diving in full bore into proxmox with subscriptions for all my servers. Hope marketing/sales is reading this thread.

Agree with all your points. Fast local storage (8 disk RAID10 arrays and SSDs) are much more preferable to us as well, not to mention there is no "central" storage possibility for OpenVZ AFAIK.

Ploop is badly needed for: snapshotting, live migration, backup speed and not to forget quota calculation speed at host boot.

please advise - getting a reply from design/sales instead of support might be nice to see what ProxMox's "future vision" is as opposed to the obvious technical answer right now which is "unsupported" (and so far, "likely not to be supported in future" - need more explanation of that statement! it's quite troubling!)

Proxmox is a small team, you basically speak with the same 3 guys all the time. They put a lot of energy into this, but unfortunately they often seem disconnected or oblivious to the real world needs of the community. I hope that ploop is important and stable enough for inclusion.

in the meantime how do people find live migrations between seperate boxes on individual local storage with things like postgres busy in the middle of a big write?

We only have experience with MySQL boxes, but it's practically impossible to live migrate them.

 

[mathx]

Agree with all your points. Fast local storage (8 disk RAID10 arrays and SSDs) are much more preferable to us as well, not to mention there is no "central" storage possibility for OpenVZ AFAIK.

Ploop is badly needed for: snapshotting, live migration, backup speed and not to forget quota calculation speed at host boot.

Proxmox is a small team, you basically speak with the same 3 guys all the time. They put a lot of energy into this, but unfortunately they often seem disconnected or oblivious to the real world needs of the community. [..]

We only have experience with MySQL boxes, but it's practically impossible to live migrate them.

Central storage: meant NFS or the like. Complexifies things needlessly in my environment.

Backup speed isnt a big deal to me right now (backing up with rsync to an old recycled target with --link-dest is pretty fast, even across 10M files (Im using linux-vserver right now) -but isnt quite a 'snapshot' of course (could use lvm snapshots to freeze it then do the backup...))

As for '3 guys' - that's fine - if they're experts and can handle the support load, that's great. And Im willing to pay for a year's subscription totalling into the 1000 of euros if this solution is at least targetting what I need (which is many CT's per machine with live migrations).

Mysql isnt a big deal in terms of migration - I can shut it down briefly - the problem is not knowing which services are going to be a problem. Fuser -mv /var/lib/vz/root/CTID# before migration shows all processes using the mount of course, so we cant tell which ones will become an issue til we try it - and if we add a new service to the image, we wont know if its migratable or not. I can already manually offline migrate linux-vserver, so there's no benefit there.

Can openvz do it? (apparently some bug...)

Anyway, getting off topic which is "ploop support is needed". If its in with the RHEL7 kernel that'd be great, if that kernel gets here sometime before summer...

 

[mo_]

unfortunately they often seem disconnected or oblivious to the real world needs of the community.

This is not fair. You make it sound like the community (aka EVERYBODY) would want ploop, which is just not true. ploop is a feature that could be nice to have (as in: gimicky), but its really not as essential as you make it sound.

also:

Im willing to pay for a year's subscription totalling into the 1000 of euros if this solution is at least targetting what I need

You might want to contact the proxmox team via email about that (so that it contains your companies address and such, for liability reasons) because I am guessing that it would be possible for you to sponsor the integration and testing of ploop with that kind of funding (its what other open source projects usually allow for anyway)

 

[mathx]

This is not fair. You make it sound like the community (aka EVERYBODY) would want ploop, which is just not true. ploop is a feature that could be nice to have (as in: gimicky), but its really not as essential as you make it sound.

Right now im under the belief that use of ploop for containers would solve the mysql migration issue by providing an actual filesystem image where inodes map to the same files on any host once live-migrated (vs living on the very different filesystems of the hosts). If im wrong, then we can seperate the issues. (Btw, some quick (ie not deep) testing of postgres shows it migrates ok (!! - suprised, thought it reached deep in to the fs)).

The rest of the features of ploop are great too, but this is a blocking issue that I think is related.

How do you do your CT live-migrations of mysql?

 

[mo_]

This is probably not what you'd want to hear but: I just don't. However if I did have to tackle that, I would probably go implement a workaround for mysql's weird io handling. Meaning: modify vzmigrate to suspend mysqld during live migrations (its just a shell/perl script after all). I have even read somebody doing just that before, but I was unable to find that back, sorry. Pretty sure I read that either on the openvz forums or wiki, but I just can't find it.

Also I should add, that (at least to my knowledge) Proxmox has not been designed with webhosters as the primary use case, but corporate environments - as a vmware replacement of sorts.

 

[mo_]

Or heres an alternative idea:

You really only ever need 1 mysql server per openVZ host since you can just share the socket across all the containers.

Knowing that you can have a mysql container on each node and just keep the databases synced up in some way (clustering, replication... whatever) and then you don't need to migrate mysql containers, EVER. Instead you only ever migrate the application containers, which will find a mysql socket on any hardware node they happen to run on.


edited to add: Id recommend replicating the mysql server anyhow since then you can just shutdown the secondary server and perform a proper backup on it, without touching the production system with the live database

 

[mathx]

This is probably not what you'd want to hear but: I just don't. However if I did have to tackle that, I would probably go implement a workaround for mysql's weird io handling. Meaning: modify vzmigrate to suspend mysqld during live migrations (its just a shell/perl script after all). I have even read somebody doing just that before, but I was unable to find that back, sorry. Pretty sure I read that either on the openvz forums or wiki, but I just can't find it.

Also I should add, that (at least to my knowledge) Proxmox has not been designed with webhosters as the primary use case, but corporate environments - as a vmware replacement of sorts.

I dont think that would work - cant just suspend mysql, you need to completely shut it down. The reason is because this is the actual bug:

https://bugzilla.openvz.org/show_bug.cgi?id=2242#c26

summary: mysql keeps filehandles open on unlinked (deleted) files. [facepalm]

There's a patch to the OpenVZ kernel to fix this and it has been intimated before that this could be injected into current proxmox kernel, but there's resistance til RHEL7 kernel by the devs. I might attempt it myself in the meantime (thus destroying my support options, and thereby also removing a reason to purchase support. Funny.)

Ploop is a work around because it keeps the inodes of the unlinked files in the exact same spot (its a filesystem image) and of course rsync doenst copy unlinked files.

As far as 'enterprise' goes, plenty of mysql in enterprise environs, and plenty of poor code too, and likely something that also relies on open FH's to unlinked files. proxmox shouldnt include containers if it doesnt want to support them and just be a KVM based replacement for Vmware.

Applying that patch to the current kernel would fix this issue up as I mentioned. Ploop is merely a workaround, so either would do.

moving technical portion of this discussion over to the MYSQL-live-migration-failure thread here:

http://forum.proxmox.com/threads/16480-container-live-migration-failure-w-mysqld

 

[mo_]

I just made another post (just above yours). also by "suspend" I actually meant /etc/init.d/mysql stop. You'd need to do that after the first rsync pass to minimize mysql downtime

 

[mathx]

Or heres an alternative idea:

You really only ever need 1 mysql server per openVZ host since you can just share the socket across all the containers.

in an ideal world of cooperative under-one-person/design's control yes, that'd be great, but I support multiple different customers with various versions of mysql and debian etc etc. It's a big mixed bag. Any proper virtualization/containerization environment can support these disparate servers (right now using linux-vserver with centos and debian images, could even be a couple lenny's around

 

[mo_]

this is another possible angle to tackle this: http://openvz.org/Vzmigrate_filesystem_aware

if you have shared storage this will make vzmigrate skip the rsync completely and only transfer the memory content of the container.

Does MariaDB cause this issue too by the way? thats a mysql fork and maybe they fixed this mysql quirk/weirdness (just a wild guess there)

 

[gkovacs]

This is not fair. You make it sound like the community (aka EVERYBODY) would want ploop, which is just not true. ploop is a feature that could be nice to have (as in: gimicky), but its really not as essential as you make it sound.

I wasn't necessarily talking about the ploop implementation issue, but the general stance of the Proxmox devs when suggestions come from the community. Storage definition with backups is still very much backwards working, not to mention there is still no storage option when migrating - these are just two UI/usability issues that the community has been repeatedly asking for, but the devs are not even considering (or if they were, they forgot about it).

 

[tom]

I wasn't necessarily talking about the ploop implementation issue, but the general stance of the Proxmox devs when suggestions come from the community. Storage definition with backups is still very much backwards working, not to mention there is still no storage option when migrating - these are just two UI/usability issues that the community has been repeatedly asking for, but the devs are not even considering (or if they were, they forgot about it).

Please stop writing wrong statements here, there is a clear roadmap to ploop. How do you think that feature went into a product? Just a wish from "the community" and magically the feature appears one week later?

All features are there because people developed it or paid someone for developing it. Its that easy - If you want to join dev discussions, follow this: http://pve.proxmox.com/wiki/Developer_Documentation

 

[mathx]

this is another possible angle to tackle this: http://openvz.org/Vzmigrate_filesystem_aware

if you have shared storage this will make vzmigrate skip the rsync completely and only transfer the memory content of the container.

Does MariaDB cause this issue too by the way? thats a mysql fork and maybe they fixed this mysql quirk/weirdness (just a wild guess there)

maria has even more links to deleted files in /tmp for some reason (not that it matters, any more than zero is an issue). it's still better than mysql

however, im not always aware and can't vette every piece of software my customers will run, so migratability could be an issue without this being solved.

Tom: thanks for the reply, will check that wiki page, and thanks for hearing us.

 

[dietmar]

I dont think that would work - cant just suspend mysql, you need to completely shut it down. The reason is because this is the actual bug:

https://bugzilla.openvz.org/show_bug.cgi?id=2242#c26

summary: mysql keeps filehandles open on unlinked (deleted) files. [facepalm]

There's a patch to the OpenVZ kernel to fix this and it has been intimated before that this could be injected into current proxmox kernel

What patch do you talk about exactly?

 

[mathx]

What patch do you talk about exactly?

few comments down in that thread I posted on the bug: https://bugzilla.openvz.org/show_bug.cgi?id=2242#c33

more specifically: https://bugzilla.openvz.org/attachment.cgi?id=1886&action=diff

 

[dietmar]

more specifically: https://bugzilla.openvz.org/attachment.cgi?id=1886&action=diff

That patch is already included in our kernel.

 

[mathx]

That patch is already included in our kernel.

aha. I suppose the ultimate issue is rsync cant copy deleted files, even if we can get at the content via /proc/*/fd - no way to drop it in the right place in the filesystem for the migrated container to find without some kinda hacky support in the OpenVZ kernel. Actually Im quite suprised that many other processes tolerate files suddenly appearing at different inodes in the filesystem - would have thought there'd be more deep IO relying on that.

 

[jinjer]

I posted this in another thread, but I think it belongs here:

ploop fits nicely in the new ceph and gluster direction the proxmox project is headed. Currently containers are not usable on top of ceph and gluster. ploop can fix that, even if it's accessed over gluster, it will still be much faster than single file accesses.

Add this to snapshotting, rollbacks, live migration and so on...

Maybe a little bounty would push things forward?