Nodes must be on the same network??

[FrankB]

I read this line in the wiki:

• All nodes must be in the same network as corosync uses IP Multicast to communicate between nodes (See also Corosync Cluster Engine). Note: Some switches do not support IP multicast by default and must be manually enabled first. See multicast notes for more information about multicast.

So having nodes across the country from each other is out of the the question?? I was hoping to have nodes on different providers. If this is true, this really blows my whole plan out of the water.

 

[1nerdyguy]

The nodes need to be able to talk to eachother for the clustering to work. I know in older versions it needed multicast, not sure if that's still true. I believe some people have had success setting up nodes to vpn to eachother, and using the vpn as the cluster network, but it was slow.

You shooting for redundancy for the different providers, or what?

 

[FrankB]

You shooting for redundancy for the different providers, or what?

Yeah. That was the plan. To get three servers in different datacenters from different providers and set them up in an HA Cluster or just even as regular cluster. That way no matter what natural disaster occurred, or where is occurred, their services would never go down.

But if the clustering daemon/agent can't use standard IP to communicate, then that plan is scr*wed.

 

[t.lamprecht]

But if the clustering daemon/agent can't use standard IP to communicate, then that plan is scr*wed.

It can, but only as a fallback, also multicast is in fact a standard "IP protocol" If the provider support routing these messages over the internet it could even work, AFAIK, but it wont work that good.

A normal setup of a PVE cluster needs to see and reach all nodes over multicast on the cluster network. For a small node count you may try falling back to unicast, but such a setup may be slow and you have to test it a lot to see that all works as expected else you even may get a little less reliability out of such a setup.

For unicast see: https://pve.proxmox.com/wiki/Troubl....29_instead_of_multicast.2C_if_all_else_fails

Also for a HA cluster with (live) migration you need to access all the VMs storage on each node at each data center, this means replicating all the storage on each location, which can need really a lot of traffic and bandwidth.
That would also be needed for a normal cluster to allow restarting a VM on another node if one datacenter "burns down" (for example).

 

[wosp]

And another issue you may forget is the IP-space. If you move a VM from one datacenter/network to another, will the VM still switch the IP he use? Offcourse you can setup VPN-tunnels to use the IP-space on all locations, but this don't give you the (network) redundancy you are looking for.

Only way you may let this work properly is to setup your own network (get your own ASN) in different datacenters and have fast, reliable and redundant traffic between the datacenters (your own dark fibers for example). But as I see your other posts, I think this is out of the question.

 

[FrankB]

Yeah, not exactly what I had in mind.

 

[FrankB]

And another issue you may forget is the IP-space. If you move a VM from one datacenter/network to another, will the VM still switch the IP he use? Offcourse you can setup VPN-tunnels to use the IP-space on all locations, but this don't give you the (network) redundancy you are looking for.

Only way you may let this work properly is to setup your own network (get your own ASN) in different datacenters and have fast, reliable and redundant traffic between the datacenters (your own dark fibers for example). But as I see your other posts, I think this is out of the question.

I know developers only have datacenters and the enterprise in mind, but I was hoping for more flexibility that would have functionality for the small enterprises as well. Like just being able to have clusters on separate networks be a normally supported option, so that you can easy migrate machines to mitigate loads, and things of that nature. Reading the wiki there seems to be so many limitations to doing so.

 

[FrankB]

So I guess, I will just have to run them as standalones. And if one fails, I will have to ask the datacenter to place the external storage device(freenas box connected to second nic) to a rented server so that I can rsync it across to the other server and recover that way.

 

[t.lamprecht]

Like just being able to have clusters on separate networks be a normally supported option, so that you can easy migrate machines to mitigate loads, and things of that nature. Reading the wiki there seems to be so many limitations to doing so.

Make the technology and/or talk with the datacenters to make that happen and we implement it (or "patches welcome"), i.e. talk to the corosync guys to allow reliable (!) realtime cluster communications happen over the whole internet, with real time storage replication and so on Not to be rude but we do not make artificial limits, if it was that easy we would have probably implemented it already. Also we have a lot of users which have small setups, so we also have them in mind.
These are more technical limitations than other ones, as also wosp said.

That said, with ZFS we have https://pve.proxmox.com/wiki/PVE-zsync which could be of use for you, also if on another storage rsync the VMs disk (and maybe also config) to the other node via cron or something, if one fails you are ready to restart it in minutes.

 

[tomstephens89]

Splitting a cluster out over a 'WAN' is typically not good practice. A cluster is supposed to describe a local entity, a group of hosts in the same rack/row/datahall for example. Cluster & storage networking is paramount to the performance of the cluster and therefore local networking for your singular cluster is always the best design.

For the purposes of DR for example, you would typically have a second cluster elsewhere that maintains a copy of the VM disks & configuration. Usually, and in my experiences this is achieved using storage replication at the SAN (or whatever storage you use). Your primary SAN provides fast access over a network to your primary cluster, but also has WAN connectivity to your second site over which it asynchronously replicates its LUNS to a partner SAN. You second cluster has fast network access to this partner SAN so that in the event of a disaster, it can take over operations.

Perhaps a feature recommendation for Proxmox would be the ability to add remote or additional clusters into the management GUI and provide a vMotion 'like' feature where we are able to move a VM to another cluster. VMWare vCenter is a prime example of a multiple cluster capable management interface, however works to a completely different 'single master' model which Proxmox doesn't.

Whilst I think the above feature might be cool, I also think that it might damage the principle and integrity of the de-centralised cluster management that Proxmox does so well.

 

[t.lamprecht]

Perhaps a feature recommendation for Proxmox would be the ability to add remote or additional clusters into the management GUI and provide a vMotion 'like' feature where we are able to move a VM to another cluster. VMWare vCenter is a prime example of a multiple cluster capable management interface, however works to a completely different 'single master' model which Proxmox doesn't.

That feature request is acknowledged since some time but nobody has stepped up to implement it yet, probably because it needs quite some effort/thought and other things/features where possibly more user profit are also to do. But it needs an other architecture, i.e. it wont be really possible to sync our cluster heart piece the pmxcfs with corosync over the WAN, and we really do not want any single master system, we see that as single point of failure.
Also datacenter overlapping storage migration needs a) storage replication support over WAN or b) quite some bandwidth so we can do storage migration.
So its on the wishlist, but nothing done regarding this yet, AFAIK

Whilst I think the above feature might be cool, I also think that it might damage the principle and integrity of the de-centralised cluster management that Proxmox does so well.

+1

 

[tomstephens89]

What I would like to see is the IP address of VM's pulled from the qemu agent and displayed in the GUI

 

[FrankB]

I wrote to the CoroSync guys about clustering over WAN. Hopefully they won't dismiss it.

Also we have a lot of users which have small setups, so we also have them in mind.

I'm not asking to have any functionality removed, just added.

I would probably be happy with a build in container/vm migration tool that was cluster independent. Maybe like a built in rsync frontend for vms and containers that gets enabled when you are not in cluster mode and gets disabled when you are in a cluster as that is already part of the features of a cluster. That IS with-in the power of PROXMOX to integrate into it's panel. As rsync already exists and works over wan and if they wanted to be nice about it, it would store the other servers transfer credentials so that you can reuse it to transfer more vms/cts if necessary. So when you click on the MIGRATE button, RSYNC Servers could appear in the NODE list. Maybe this option shouldn't be disabled in cluster mode as you may have cases where people want to transfer a vm/ct to another cluster. As long a the script makes sure that the ID is not in conflict before transfer, I don't see why it should be an issue.

Throw in a ping monitor that shows the status of the other server on this servers dash, and presto you have great info for the admin to go on when monitoring for problems. If you can throw in launching scripts when they go down, you've now created the ability to make dns changes and spinup a vm to pickup the slack for the unit(if the application already pulls from an existing established DB cluster). I know alot if this can be done with bash scripting and I currently do that, but I'm all excited about proxmox integration and having an ALL-ONE-TOOL.

It would be like a datacenter in a box. Sure, storage syncing would still be an issue, and could be done with rsync on a per app basis, but to have it almost be all-things-to-all-people is exciting.

 

[FrankB]

Corosync guys responded:

The problem is neither unicast or multicast. It´s the latency between
nodes that has to be < 2 ms in order for corosync to operate properly
under heavy load.

Fabio

So there is the answer. It's definitely never going to work in that fashion as any variation in MS will trigger a break.

 

[t.lamprecht]

I'm not asking to have any functionality removed, just added.

I did nowhere say that you implied that, we also wont do that

and if they wanted to be nice about it, it would store the other servers transfer credentials so that you can reuse it to transfer more vms/cts if necessary

See: [¹]

You can login to a remote Linux server without entering password in 3 simple steps using ssky-keygen and ssh-copy-id as explained in this article.

ssh-keygen creates the public and private keys. ssh-copy-id copies the local-host’s public key to the remote-host’s authorized_keys file. ssh-copy-id also assigns proper permission to the remote-host’s home, ~/.ssh, and ~/.ssh/authorized_keys.

Just generate an ssh key on you computer if not already done then add the public key to the other server, e.g.:

ssh-keygen
ssh-copy-id backupuser@remote-host-or-ip

now the authentication is done automatically through certificates, more secure and less to do, just keep your private key safe!

Throw in a ping monitor that shows the status of the other server on this servers dash, and presto you have great info for the admin to go on when monitoring for problems. If you can throw in launching scripts when they go down, you've now created the ability to make dns changes and spinup a vm to pickup the slack for the unit(if the application already pulls from an existing established DB cluster). I know alot if this can be done with bash scripting and I currently do that, but I'm all excited about proxmox integration and having an ALL-ONE-TOOL.

It would be like a datacenter in a box. Sure, storage syncing would still be an issue, and could be done with rsync on a per app basis, but to have it almost be all-things-to-all-people is exciting.

Did you read my post above, this - not exactly word by word, but the base idea - is on a todo list, but quite at the end
That said, we're open source so patches welcome.

So there is the answer. It's definitely never going to work in that fashion as any variation in MS will trigger a break

That indication to ask was a bit ironic, sorry probably not the best manner from my side. But the point is
a) there are technical limitation as you also got in the answer from the clusterlab guys
b) yeah some similar or a few feature can be made, but they also have to be made.
c) all here contributing to the PVE project have a big feature, bug, documentation, test, ... todo list they're working on (thanks to all of them!) wishing feature is the easiest part of this, while we naturally appreciate good ideas, and your basic idea is surely not bad and exist a little longer than this thread.

It may be implemented in the future (not in the near future, I guess though (at least from me)) but there is more to be done than the simple steps you listed, routing/switching has to be handled, collisions, a good way how different datacenter can work decentralized (!) together without breaking anything, what happens in error cases, how is fencing handled through WAN and different locations (fencing is needed for HA), even with frequent rsyncs you may loose data from the VM, this can be really critical for a VM which hosts a database or anything similar, not all that trivial.

 

[FrankB]

I know I can do what needs to be done manually with ssh key sharing and rsync. I've already successfully done it. I just thought it would be nice to see it built in.
Thank you to everyone for all their hard work! It's all greatly appreciated! I've made a donation to the paypal account and plan on making more.
Sorry for being such a pest, but I really think the product is great and would really like to see it mature further and flourish. Please don't take anything I've said as harsh criticism, I'm a fan of the product.

 

[t.lamprecht]

I just thought it would be nice to see it built in.

Understandable and acknowledged, but it will need some time.
Also thanks for the appreciation!

Sorry for being such a pest, but I really think the product is great and would really like to see it mature further and flourish. Please don't take anything I've said as harsh criticism, I'm a fan of the product.

No need for an excuse, really. I hope you also do not thank anything from me as to rude/harsh, that's surely not the indent behind my posts.
I only wanted to say that it will need a little more for a general and an error prone implementation.
If you plan to use ZFS you can get quite some wanted features already, as earlier posted see https://pve.proxmox.com/wiki/PVE-zsync