Separate Ceph and Cluster. Please help

[cocobanana]

Hi All,

I have problem with our proxmox setup. I did try my best and did tried just to setup CEPH and Cluster and I FAILED. I really need many advice from you. I spent almost a week to fix this and i did reformat more than 10 times to resetup proxmox. I am using the latest proxmox 6.4

Ok, this are my servers setup

- 3 nodes R730 ( node-1 , node-2 , node-3 )
- Each node has 1 NIC ( 4 ports )
- Public IP ( we have our own AS number and in this case i name it 103.x.x.x/24 )

Our NIC setup for each servers
port 1 : connected to our internal switch and it is for VPN ( 103.16.x.x )
port 2 : connected to a single switch. For Cluster purpose ( and all 3 nodes are connected to this switch ) - 192.168.100.200/24
port 3 : connected to a single switch, For Ceph purpose ( and all 3 nodes are connected to this switch ) - 192.168.200.200/24
port 4 : we set this for our VM to be connected to the internet . This port is connected to the core switch and use IP 72.x.x.x

The questions
1) What is the best setup/config that you can advise?
2) Can we separate the CEPH and the CLUSTER which means Port 2 only for cluster and Port 3 only for CEPH?
3) What is the Best way for the I stuck on the CEPH setup ( Public IP and CLuster IP because it is super confuse )
4) Can I use IP 172.x.x.x as vmbr0 or should i use our public IP 103.x.x.x ?
5) Which Ceph package is better for my setup? nautilus or octopus?
6) I plan for the access to the Proxmox Management GUI, I plan to use IP 172.x.x.x , is it possible? and how to accomplish that?

** By default vmbr0, I am using our public IP ( 103.x.x.x )
**

All my setup above will bring the issue to ( Timeout Error once i setup the CEPH )

Node-1 : Network COnfig : http://prntscr.com/15sdg20
Node-2 : Network Config : http://prntscr.com/15sdlng
Node-3 : networn Config : http://prntscr.com/15sdquh

please advise.

Thanks

 

[oz1cw7yymn]

Hi!

I'm sorry, I'm not fully following what your question is and what you're having problems with, but maybe I can give some useful input.

First - avoid talking about cluster, that will just cause confusion (both proxmox and ceph are "clustered" with different technologies).

Second - it is very useful for the proxmox to have a separate "corosync" network (the proxmox cluster technology), so I suggest put that on a separate nic with a separate switch.

Third - it is very useful for ceph to have a separate backend network, so I suggest you put that on a separate nic with a separate switch.

Fourth - if no VMs need direct access, you could put the ceph frontend network on a separate nic. I'm not actually suggesting that, but it's possible and might be best practice depending on your work load. I don't have it configured like that myself, but it would protect ceph from being exposed to the internet.

Fifth - you need a network that the VMs use that can reach the internet and possibly be reached by the internet. This is your last essential nic, connected to the external network, usually vmbr0.

Sixth - extras. You can decide to use a second network as a backup for the corosync proxmox network. For ceph, the frontend network works as a fallback for the backend network if needed. Additionally you can decide in promos which of all of these networks should be used for moving and replicating VMs.

Not sure if I'm answering any of your questions..

 

[cocobanana]

Hi!

I'm sorry, I'm not fully following what your question is and what you're having problems with, but maybe I can give some useful input.

First - avoid talking about cluster, that will just cause confusion (both proxmox and ceph are "clustered" with different technologies).

Second - it is very useful for the proxmox to have a separate "corosync" network (the proxmox cluster technology), so I suggest put that on a separate nic with a separate switch.

Third - it is very useful for ceph to have a separate backend network, so I suggest you put that on a separate nic with a separate switch.

Fourth - if no VMs need direct access, you could put the ceph frontend network on a separate nic. I'm not actually suggesting that, but it's possible and might be best practice depending on your work load. I don't have it configured like that myself, but it would protect ceph from being exposed to the internet.

Fifth - you need a network that the VMs use that can reach the internet and possibly be reached by the internet. This is your last essential nic, connected to the external network, usually vmbr0.

Sixth - extras. You can decide to use a second network as a backup for the corosync proxmox network. For ceph, the frontend network works as a fallback for the backend network if needed. Additionally you can decide in promos which of all of these networks should be used for moving and replicating VMs.

Not sure if I'm answering any of your questions..

Thank your for your reply.

I managed to startup my proxmox and now i am a midst on setting up a cloud-init with windows VM + WHMCS Intergration.