[SOLVED] Cluster without vlan / vrack (ovh)

[MiniMiner]

Hey!

I want to create a proxmox cluster but don't have the possibility to create a vlan(vrack). I don't want to use the cluster for HA, just for migrating Containers between the nodes.
Are there any problems with this or can I just use the external IPs?

My second question: What if I have several servers ruinning Proxmox without a cluster. (But with Containers on it)
Is there any way to add the nodes to the new cluster or do I have to backup/import them manually?

Thanks in advance,

Arnim

 

[spirit]

My second question: What if I have several servers ruinning Proxmox without a cluster. (But with Containers on it)
Is there any way to add the nodes to the new cluster or do I have to backup/import them manually?

Arnim

you can't join a node in a cluster, if you have vm or ct created on this nodes.

if you don't have vmid conflict, you can,

- move vm and ct configuration (/etc/pve/qemu-server/*.conf , /etc/pve/lxc/*.conf) from the node you want to join in the cluster, to a temp directory. (vm and ct still running).

-add the node to the cluster. (the node will have now the /etc/pve from the cluster).

-move back the vm and ct config files to /etc/pve ...

done.


but you really need to not have vmid conflict.

if not, make backup/restore with a new vmid.

 

[MiniMiner]

Hello,

thanks four your reply and the vm/ct move solution.
Do you know anything regarding my main question?

I want to create a proxmox cluster but don't have the possibility to create a vlan(vrack). I don't want to use the cluster for HA, just for migrating Containers between the nodes.
Are there any problems with this or can I just use the external IPs?

Kind regards,

Arnim

 

[spirit]

Hey!

I want to create a proxmox cluster but don't have the possibility to create a vlan(vrack). I don't want to use the cluster for HA, just for migrating Containers between the nodes.
Are there any problems with this or can I just use the external IPs?
Arnim

you need multicast by default. don't known who is your hosting provider ? (maybe ovh ? (vrack)). generally you can't do it without a vrack or something similar.

but you can setup unicast
https://pve.proxmox.com/wiki/Multic....29_instead_of_multicast.2C_if_all_else_fails


for vm/ct migration, the traffic is encrypted between proxmox nodes, so no problem.

 

[MiniMiner]

Hello @spirit ,

thanks for your quick reply.

The wiki says that I shouldn't use unicast with more than 4 nodes. The problem is that my cluster would have more than 4 nodes. My hosting provider is indeed ovh but as I said, I don't have the possibility to use a vrack. I already talked to a supporter from there and he said that I can use a vpn server to solve the vrack problem.
But this would mean that every traffic would be routed through the vpn to the internet. The nodes work independently so this isn't an option I think.

Do you know/think if/that there is a way to use the vpn server for internal communication and let the nodes communicate externally? (Not through the internal vpn)

Kind regards,

Arnim

 

[spirit]

I'm currently working to add vxlan support to proxmox, it should help with your setup.
I think it'll be ready in 1 or 2 months.

about the unicast 4 nodes limit, it's really depend of the network latency.
I'm running a 20nodes cluster with unicast in my datacenter, with my good switches. (0,025ms) latency

 

[MiniMiner]

Hello,

this sounds great!

I tested the network latency between two existing nodes:

tcp_lat:
latency = 112 us
msg_rate = 8.89 K/sec
loc_send_bytes = 8.9 KB
loc_recv_bytes = 8.89 KB
loc_send_msgs = 8,896
loc_recv_msgs = 8,895
rem_send_bytes = 8.89 KB
rem_recv_bytes = 8.89 KB
rem_send_msgs = 8,895
rem_recv_msgs = 8,895

Would this be fast enough?
And just to confirm: Unicast would work with external IPs? (I'm not very experienced with multi- or unicast)

Thanks,

Arnim

 

[MiniMiner]

Hello,

I finally solved the problem. I set up an internal network using meshvpn (peervpn). (The installation and configuration is really easy)
After starting the vpn on both nodes, I created a cluster on the first node. Proxmox automatically used the internal IP adress for the cluster.
Adding the second cluster node via the GUI worked first-try.

Some notes for people who also try to create an ovh cluster without vrack:

- Edit the /etc/hosts file so the hostname resolves to the internal IP adress e.g.:

10.0.0.1 nsxxxxxxx.ip-xxx-xxx-xxx.eu nsxxxxxxx

10.0.0.2 nsxxxxxxx.ip-xxx-xxx-xxx.eu nsxxxxxxx

- start the vpn before creating the cluster
- use IPTables to only allow cluster nodes to connect to the vpn
- add meshvpn/peervpn to startup (within a screen session for example)

I want to note that I tested this with only two nodes. I don't know if this works with x nodes as the latency is really high in comparison to the one @spirit wrote.

Have a great day,

Arnim