Joining unlicensed PVE node to cluster that is licensed. . . what happens?

[BriBriBri]

Hi All,

I'm just curious. Certainly not going to try it.

But what happens (ie., what could/would go wrong) if we were to join a new PVE node to our PVE cluster (w/ ceph) when the new node is not yet licensed. The other eight nodes pre-existing on the cluster are licensed and on the enterprise repo.

Does anything bad happen by design?

Best Regards,

Brian

 

[sterzy]

Hi,

if you add a node that has no subscription to a cluster that otherwise does, the node will work just fine. Depending on how you configured the repositories on you subscribed nodes you might have to deal with some version mismatch issues, but nothing is currently known that should cause problems.

Be aware though that the subscription agreement clearly states that the subscription level defaults to the level of the node with the lowest level in the cluster. Hence, if a node does not have a subscription, the entire cluster will be handled at the no subscription level.

 

[privnote]

Be aware though that the subscription agreement clearly states that the subscription level defaults to the level of the node with the lowest level in the cluster. Hence, if a node does not have a subscription, the entire cluster will be handled at the no subscription level.

Can you describe this in more detail?

In this case, for example, you won't get any support?

 

[sterzy]

In this case, for example, you won't get any support?

Yes that is exactly my point. If you have four nodes that are properly subscribed, and you add one node that has no subscription, the entire cluster is considered as “no subscription”. So if you open a support ticket with such a cluster, your cluster will be treated as if it had no subscription at all.

There may be cases in which we show some leniency, such as if you're struggling to activate a subscription. Obviously we will still help you out with such issues. However, as per the subscription agreement section 3.7 [1]:

3.7. Subscriptions for a Proxmox VE cluster
In a Proxmox VE cluster all nodes need to have the same subscription level.

Example: Consider having a cluster with three nodes, each node has 1 CPU-socket. You want to get a “Standard Subscription” for your cluster. This means that you need to buy 3 x “Proxmox VE Standard Subscription 1 CPU/year”

[1]: https://proxmox.com/en/downloads/pr.../agreements/proxmox-ve-subscription-agreement

 

[privnote]

Yes that is exactly my point. If you have four nodes that are properly subscribed, and you add one node that has no subscription, the entire cluster is considered as “no subscription”. So if you open a support ticket with such a cluster, your cluster will be treated as if it had no subscription at all.

ok...
Especially for smaller teams with a limited budget, who may still have test servers in their cluster, this is very expensive.

Perhaps there is an alternative?
Why not define individual nodes as testing and therefore without support?

Of course you could set up your own cluster with the testing nodes, but this is then very cumbersome whenever you want to migrate VMs, etc.

 

[alexskysilk]

to amplify- a development cluster should run the same version of the software as the production server will (ultimately.) this is not really possible if the dev cluster runs nonfree repo and production runs enterprise.

 

[LnxBil]

Why not define individual nodes as testing and therefore without support?

What do you run on a "test node"? I don't get why you want to have dedicated nodes in a cluster for that. Why not just run the stuff on the production nodes?

 

[privnote]

What do you run on a "test node"? I don't get why you want to have dedicated nodes in a cluster for that. Why not just run the stuff on the production nodes?

e.g. to avoid using the production nodes more than necessary.
The testing nodes can then be made available to developers and others without them being able to jeopardize the production nodes with tests / benchmarks / etc

I also agree with @alexskysilk argument

 

[Dunuin]

to amplify- a development cluster should run the same version of the software as the production server will (ultimately.) this is not really possible if the dev cluster runs nonfree repo and production runs enterprise.

Evil companies would buy cheaper community subscriptions for a dedicated test cluster, that needs no support, to be one the same enterprise repo version...

 

[sterzy]

The testing nodes can then be made available to developers and others without them being able to jeopardize the production nodes with tests / benchmarks / etc

Mixing testing and production nodes is a really bad idea. Remember that a cluster's stability depends on it being able to find a stable quorum. If you do your testing with a subset of your nodes, they may interfere with the quorum of your cluster. Which is something you don't want in a production cluster for obvious reasons.

If by “test nodes” you just mean nodes that host guests that are used for testing purposes, that is another matter. But I would argue that you aren't running “test nodes” then so much as “test guests”.

Of course you could set up your own cluster with the testing nodes, but this is then very cumbersome whenever you want to migrate VMs, etc.

You can do migration between clusters these days with the qm remote-migrate command [1]. It's still marked as “experimental”, but works quite well in our testing, and we had some customers where it came in handy too.


[1]: https://git.proxmox.com/?p=qemu-server.git;a=commit;h=192bbfda
[2]: https://pve.proxmox.com/pve-docs/api-viewer/index.html#/nodes/{node}/qemu/{vmid}/remote_migrate

 

[alexskysilk]

Evil companies would buy cheaper community subscriptions for a dedicated test cluster, that needs no support, to be one the same enterprise repo version...

evil? explain...

The testing nodes can then be made available to developers and others without them being able to jeopardize the production nodes with tests / benchmarks / etc

then you really shouldnt mix that with your production.

My comment wasn't about rationalizing or justifying the ask, it was more as an observation that tying the software stack to a support contract has... drawbacks.

 

[Dunuin]

evil? explain...

Ok, not that evil as I later removed the apt mirror part, of my post, to keep subscription costs down. But still not that nice to get a subscription targeted at private/non-profit use to maximize company profits.

 

[alexskysilk]

If its any consolation, I dont think you're evil.

much.

 

[sterzy]

My comment wasn't about rationalizing or justifying the ask, it was more as an observation that tying the software stack to a support contract has... drawbacks.

Well, a) the software stack is and will remain open source, so it isn't tied to support contracts and b) if this is about making sure that your production and testing environments use the same packages, there are ways to ensure that (e.g., POM can help here [1,2]). For one, the packages in the no-subscription repo are just the next version of the production repo's packages. So you with a bit of extra effort you could make sure you never update to things your production cluster isn't running. And secondly, a test environment may be just the place to test out the next version of our software stack, before moving over the production environment.

It's not like we are hiding some extra special better software in the enterprise repo that we don't publish elsewhere. All it really is, is a convenient method to make sure you run the most tested and most stable version of our software stack.

[1]: https://forum.proxmox.com/threads/proxmox-offline-mirror-released.115219/
[2]: https://pom.proxmox.com/offline-media.html#example-local-http-server

 

[LionBurr]

I guess I'll add a question onto this thread.

I'd like to replace a node in a HA Cluster with another node - 1 CPU socket. Cluster is currently running Proxmox 7 latest. New node will be running Proxmox 8.

Is there anything wrong with my proposed process?

1) Add Proxmox 8 unlicenced node to the Proxmox 7 licensed cluster. (And perform backups of existing VMs and containers)
2) Disable node being removed from replication, and set HA priorities to run VM/Containers on new node and remove old node from priorities.
3) Should containers/VMs automatically move to the new node? Hopefully so.
4) Once VMs/Containers are moved off of old node, remove old node from cluster.
5) Sign into Proxmox license site and reissue license for new node.
6) Upgrade new node, then perform Proxmox 7->8 upgrades on existing nodes as per instructions.

Any thoughts are appreciated.

-Bear

 

[gfngfn256]

1) Add Proxmox 8 unlicenced node to the Proxmox 7 licensed cluster.

The major part(s) of your question, concern a correct upgrade procedure for your node replacement and cluster upgrade - you should open a new Topic post for this.

But in as far as this has been posted on the current Topic (which is subscription-centric), why not immediately license a subscription on the new node, and then proceed with the rest of the cluster-join & config/update.

 

[LionBurr]

But in as far as this has been posted on the current Topic (which is subscription-centric), why not immediately license a subscription on the new node, and then proceed with the rest of the cluster-join & config/update.

I don't want to buy a new license for a new node, when I'm retiring an existing node shortly after moving my containers/VMs off of it, and want to re-use that old license for the new node.

 

[leesteken]

I don't want to buy a new license for a new node, when I'm retiring an existing node shortly after moving my containers/VMs off of it, and want to re-use that old license for the new node.

If you do not ask for support until the subscription is transferred (and the old node retired), I can't imagine that there would be a problem. If you worry about it, why not do it the other way around in order to adhere to the letter of the subscription.

 

[sterzy]

I don't want to buy a new license for a new node, when I'm retiring an existing node shortly after moving my containers/VMs off of it, and want to re-use that old license for the new node.

As @leesteken and @gfngfn256 said: Simply remove the subscription from the old node then and transfer it to the new one. When you do that, is up to you, as long as you don't have both nodes in the cluster you should have enough subscriptions. The key point is that all nodes need to have the same subscription level when you ask for support. However, this is really a separate topic, please open a new thread.