Multi Datacenter Management

[itNGO]

That's not an update as of November 2023, that's from ancient times as far as development is concerned.

Well, it is ready when it is ready.... as already written... they want it to be nice and stable..... anything about any "release-dates" will only give "pressure" to release something that is not finished.... a problem that nowadays is far to often happening.... release some bugged, garbage, just to release it....

I am pretty sure, when there is something to show, it will be posted here, as it will be the biggest thing since invention of Warp-Drive.... ;-)

 

[jsterr]

Its a "free" product, free to use - you can also make feature requests or even start coding and submit your own additions for a review to the proxmox team. Subscriptions are for tested updates, offline mirror and support. I cant understand those people that put pressure on developers that give so much to the people. Theres more important things then multi-datacenter management. The product needs to be stable, bug-free, uptodate regarding drivers and easy to use.

If you want to help start brainstorming about what features you would expect from a "Multi Datacenter"-Management.

 

[itNGO]

If you want to help start brainstorming about what features you would expect from a "Multi Datacenter"-Management.

If you ask me, coming from "SCVMM"... Start/Stop VMs... see all VMs in an overview... Migrate VMs between Clusters... not much more than that....

 

[jsterr]

My ideas so far (just brainstorming):

• Live Migration: between clusters (already possible on cli with qemu remote-migrate)
• Global Status Panel: with cluster-wide health (to see if all clusters are fine, or not)
• All Basic Operations: (the ones from host-management) you can do already should be in multi-datacenter management as well
• Easier Storage-Migration: for vms with multiple disks (not that important but would be cool)
• Full Monitoring: of the important features you have installed:
  • including ceph checks etc for datacenter-notification module introduced with 8.1
  • this could reduce the need for external looks like checkmk or zabbix
• Intelligent Dashboard: some intelligent dashboard that tells you things, based on logs, history-data etc.
  • including tips and optimizings for your infrastructure like:
  • You should think about adding a second corosync link
  • You should think about adding more disk-space, a disk failure in ceph would cause a complete downtime
  • You should think about removing installation-media from VM: 100, 110 they are in HA and cant be started on Host X,Y
  • You should think about installing qemu-guest agent on ressource X,Y because it is enabled but not installed in VM
  • You should think about upgrading your cluster, RAM is overcommited by XY % ...
  • ...
• No dependency to corosync, cluster management should be usable even when some nodes have problems
  • no one wants unwanted fencing because of corosync
• Ressource Prediction: an overview of your ressources (vcpu, mem, disk) that counts all nodes etc.
  • some predicting values like time to full
• Cluster wide Ressource Pools: that include ressources from multiple clusters
• ...

Im sure Proxmox will do their best to deliver the best product possible.

 

[LnxBil]

No dependency to corosync, cluster management should be usable even when some nodes have problems

It is, if you're quorate. If you're not quorate you have a much bigger problem and need to think about your cluster infrastructe and built it more error resilient.

 

[jsterr]

It is, if you're quorate. If you're not quorate you have a much bigger problem and need to think about your cluster infrastructe and built it more error resilient.

Yeah I meant quorum should be per cluster, at the current state if you put all your nodes in one ui (means one cluster) everything is dependend from each other. Means there must be something new. Imagine you have two 3node clusters managed in one multi-datacenter management. it would be bad, having problems login in and managing because one three-node-cluster is down.

 

[spirit]

Hi,
here a friend tool to manage multiple pve clusters. (It's an golang client , works on windows only currently, but works with wine on linux too)

https://cluster-manager.fr/

 

[Dunuin]

Well, it is ready when it is ready.... as already written... they want it to be nice and stable..... anything about any "release-dates" will only give "pressure" to release something that is not finished.... a problem that nowadays is far to often happening.... release some bugged, garbage, just to release it....

It's really hard to wait years for features that would be so great and could be so useful now...but better to wait until it is finished and stable than what the game industry is doing since the invention of pre-order and online patches...

I am pretty sure, when there is something to show, it will be posted here, as it will be the biggest thing since invention of Warp-Drive.... ;-)

Zefram Cochrane would probably agree ;-)

My ideas so far (just brainstorming):

I really would like to see some cross-cluster synced "cluster.fw" so all datacenters could share the same IP sets, aliases and security groups...so annoying to change these manually again and again for each datacenter...with the risk missing one datacenter where a migrated VM then wouldn`t work because an alias or security group isn't defined or even pointing to an old IP whitelisting some wrong VMs...

 

[LnxBil]

Yeah I meant quorum should be per cluster, at the current state if you put all your nodes in one ui (means one cluster) everything is dependend from each other. Means there must be something new. Imagine you have two 3node clusters managed in one multi-datacenter management. it would be bad, having problems login in and managing because one three-node-cluster is down.

Yes sure.

Zefram Cochrane would probably agree ;-)

here a friend tool to manage multiple pve clusters. (It's an golang client , works on windows only currently, but works with wine on linux too)

https://cluster-manager.fr/

Maybe it got released as open source, so that more people can work on it. The Wine remark is very good, I wanted to try to emulate it too. Looks promising (even that the author constantly mixes up Proxmox and Proxmox VE, as many people do).

I really would like to see some cross-cluster synced "datacenter.fw" so all datacenters could share the same IP sets, aliases and security groups...so annoying to change these manually again and again for each datacenter...with the risk missing one datacenter where a migrated VM then wouldn`t work because an alias or securty group isn't defined or even pointing to an old IP whitelisting some wrong VMs...

Maybe we should create a feature request about that. Would be great to have a folder for that so that you can just copy/sync files there and archieve what you described.

 

[Dunuin]

Maybe we should create a feature request about that. Would be great to have a folder for that so that you can just copy/sync files there and archieve what you described.

I already did that and if I remember correctly the response was that they might think about that in the far future and maybe I would want to script something myself using sshfs so all the nodes could use the same cluster.fw.

 

[Dunuin]

I thought it might not be that hard to quickly script something....

My idea was to only edit the ip sets, aliases and security groups on node A and then sync them from node A to node B, C, D and E. For that I would:
1.) install the "acl" package on the nodes B-E
2.) create a new sync user and private/public keys for it on nodes B-E
3.) copy private keys to node A
4.) set the ACL for /etc/pve/firewall/cluster.fw so the new "sync" user got write access too (as the default rights/owners are "640 root:www-data" so only root could change it and I don't want to sync stuff using root credentials...one advantage of not running is cluster is that when node A gets hacked it can'T easily screw up stuff on nodes B-E...wouldn't be great if node B-E root users private keys for SSH would be stored on node A for rsync...and not sure if the PMXCFS is supporting ACLs at all...)
5.) run a cronjob on node A that uses rsync to sync the cluster.fw from node A to B/C/D/E. This would be done using SSH using private keys via the "sync" users. But the "-o" flag of rsync looks problematic and I would need to use root again?:

• -o : Preserve user ownership (which is restricted to only superusers when dealing with other user’s files)

But the problem is that the cluster.fw isn't only containing the aliases, IP sets and security groups. It's also containing the datacenters firewall rules as well as firewall options. I only want to sync the aliases, security groups and IP sets but not those firewall rules and options...so I can't simply sync the whole file and would have to parse and compare and merge the actual content which makes this way more difficult and time consuming...

 

[spirit]

Personnaly, I think that the multimanagement tool , should be some kind of autonomous server app, and this app will manage sync of cluster.fw rules for example between cluster. (Rules could be managed by the app, and push to different clusters through the api)

The nodes between differents cluster don't need ssh currently. Remote migration is done through an http tunnel and token authentication.

What it could be great is to have a global authentification with sessions valid between clusters. (Personnaly I'm doing oidc, but a solution working without external auth could be great too).

As an example:
Look at some tools like xen-orchestra for example, it's a client-side app only, pooling vm config && vm state continuouly. Quorum && migration is still managed at xen-server/xcg-ng hypervisor side.

 

[yyyy]

Any updates on this? It's 2023 now and I'm using proxmox 8.11 but it seems to not support multiple datacenters (at different locations)

 

[Dunuin]

Any updates on this? It's 2023 now and I'm using proxmox 8.11 but it seems to not support multiple datacenters (at different locations)

Noting except cross-cluster migration preview.

 

[yyyy]

Noting except cross-cluster migration preview.

Damn... well that's a shame, I was thinking of just scrapping proxmox altogether as I've been encountering tonnes of bugs right from the start which need to be fixed by going into the shell all the time as the GUI is bugged doing this so many times and also realising cross-cluster is non-existent, really does seem like you are just better off running ubuntu server on bare metal anyways.

 

[AODix]

Firstly, I LOVE your work though this is my first time in forum here.
To the last post - I went from a bunch of Ubuntu servers and VMs to Proxmox and I have had the exact opposite experience - everything is SOOOOO simple on Proxmox and on those rare occasions where the command line is needed it's just Debian underneath anyway so no biggie.
I am REALLY looking forward to datacenter management as well +++111.
I have encountered hardly any bugs, like maybe MAYBE three or four TOTAL since I started on it, and it's ROCK SOLID stable (its Debian, of course it is).
Then again I try to treat it as an appliance - I added Cockpit, Samba, and a few tools, and that's probably more than I should have.
The VMs/containers are where the work is done so if you're spending all day in the hypervisor and aren't admin'ing hundreds of containers and VMs then perhaps you should rethink your end goals and the infrastructure you're using. If bare metal Ubuntu server is better than Proxmox for what you're doing then perhaps you don't need a hypervisor in the first place.
+++111 for management across different datacenters, that'd be a GAME-CHANGER. The ONE big issue I've had is with backups overwriting one another ifthe VMID is the same and I got around it by using a common templates directory with symlinks to it from individual directories for each datacenter with their own dump directory. It works really well. I have NOT used Proxmox Backup which is likely better but this has worked really well for me with four nodes NOT in a cluster and a few dozen VMs (not LXCs but full fat VMs mostly).

 

[yyyy]

Firstly, I LOVE your work though this is my first time in forum here.
To the last post - I went from a bunch of Ubuntu servers and VMs to Proxmox and I have had the exact opposite experience - everything is SOOOOO simple on Proxmox and on those rare occasions where the command line is needed it's just Debian underneath anyway so no biggie.
I am REALLY looking forward to datacenter management as well +++111.
I have encountered hardly any bugs, like maybe MAYBE three or four TOTAL since I started on it, and it's ROCK SOLID stable (its Debian, of course it is).
Then again I try to treat it as an appliance - I added Cockpit, Samba, and a few tools, and that's probably more than I should have.
The VMs/containers are where the work is done so if you're spending all day in the hypervisor and aren't admin'ing hundreds of containers and VMs then perhaps you should rethink your end goals and the infrastructure you're using. If bare metal Ubuntu server is better than Proxmox for what you're doing then perhaps you don't need a hypervisor in the first place.
+++111 for management across different datacenters, that'd be a GAME-CHANGER. The ONE big issue I've had is with backups overwriting one another ifthe VMID is the same and I got around it by using a common templates directory with symlinks to it from individual directories for each datacenter with their own dump directory. It works really well. I have NOT used Proxmox Backup which is likely better but this has worked really well for me with four nodes NOT in a cluster and a few dozen VMs (not LXCs but full fat VMs mostly).

Not being able to interlink two different data-centers in different locations kinda is a deal breaker for me really, I don't mind the CLI stuff, which is why I don't mind going back to ubuntu server, but absolutely I agree it is great and I love the UI, the connecting of different data-centers not being available is a deal-breaker unfortunately.

 

[yyyy]

Firstly, I LOVE your work though this is my first time in forum here.
To the last post - I went from a bunch of Ubuntu servers and VMs to Proxmox and I have had the exact opposite experience - everything is SOOOOO simple on Proxmox and on those rare occasions where the command line is needed it's just Debian underneath anyway so no biggie.
I am REALLY looking forward to datacenter management as well +++111.
I have encountered hardly any bugs, like maybe MAYBE three or four TOTAL since I started on it, and it's ROCK SOLID stable (its Debian, of course it is).
Then again I try to treat it as an appliance - I added Cockpit, Samba, and a few tools, and that's probably more than I should have.
The VMs/containers are where the work is done so if you're spending all day in the hypervisor and aren't admin'ing hundreds of containers and VMs then perhaps you should rethink your end goals and the infrastructure you're using. If bare metal Ubuntu server is better than Proxmox for what you're doing then perhaps you don't need a hypervisor in the first place.
+++111 for management across different datacenters, that'd be a GAME-CHANGER. The ONE big issue I've had is with backups overwriting one another ifthe VMID is the same and I got around it by using a common templates directory with symlinks to it from individual directories for each datacenter with their own dump directory. It works really well. I have NOT used Proxmox Backup which is likely better but this has worked really well for me with four nodes NOT in a cluster and a few dozen VMs (not LXCs but full fat VMs mostly).

Actually I did try to set up wiregaurd vpn for the purpose of getting two nodes which are on different networks (ie. management across different datacenters) to be able to connect together through vpn, but when wireguard interface is up and running the whole proxmox networking breaks in the GUI (i have internet but the GUI does not recognise the wireguard interface and no cluster can be created) unfortunately.

 

[AODix]

Actually I did try to set up wiregaurd vpn for the purpose of getting two nodes which are on different networks (ie. management across different datacenters) to be able to connect together through vpn, but when wireguard interface is up and running the whole proxmox networking breaks in the GUI (i have internet but the GUI does not recognise the wireguard interface and no cluster can be created) unfortunately.

I understand - I think the best thing I could say then if you do decide to go back to Ubuntu Server is to keep watching and if/when it is added then come on back! Luckily for me my machines are on the same network so that makes it easier to manage in that regard.
Having the common file share has made it simple enough for me with only 4 physical machines that its just been wonderful but if they were on separate networks then I think I would need to use a proxy connected through WireGuard then bridged with a PVE server in each location to have the same setup work and for me at least with my slow connection it'd likely be rough. It will definitely be easier for me with this feature if its added but in my case its not a deal breaker - but I get ya.
Have you considered trying that though - have an Ubuntu server instance or if you have the spare gear even have an Ubunut Server bare metal machine that connects through Wireguard which is bridged and then passes the connection on to the PVE node...? Then you could possibly get around the Wireguard problem by having Ubuntu basically be the gateway for each PVE instance - and hell you could possibly just have that Ubuntu instance be one of your VMs for that matter. It's a little late here so my brains hazy but I think you could get that to work without too much trouble and without Wireguard having and effect on Proxmoxs GUI since the Ubuntu instance would be handling that traffic in itself....for me anyways the question would be speeds and how much loss in speed/reilability or gain in latency there is by having another layer in there but I think it could work.
For that matter perhaps using a VM of pfSense or OpnSense or another router VM could work for you - have that handle the Wireguard connection and pass it back off to PVE.

 

[Dunuin]

Actually I did try to set up wiregaurd vpn for the purpose of getting two nodes which are on different networks (ie. management across different datacenters) to be able to connect together through vpn, but when wireguard interface is up and running the whole proxmox networking breaks in the GUI (i have internet but the GUI does not recognise the wireguard interface and no cluster can be created) unfortunately.

You shouldn't create a cluster anyway if the nodes are not on the same location. See the cluster requirements of <1ms latency between nodes (but I heard 10ms or a bit above (30ms?) might also work). So here cross-cluster management would be very handy as a single big cluster spanning over multiple countries just isn't possible.