CEPH Public Cluster Network Basic Setup Question

[apap]

Beginner on Proxmox, CEPH and Linux in general. Question on basic behavior.

Node 1:
Mgmt IP - 192.168.86.111/24
CEPH Public Net IP - 10.10.10.11/24 (no gateway)
CEPH Cluster Net IP - 10.10.20.11/24 (no gateway)

Node 2:
Mgmt IP - 192.168.86.112/24
CEPH Public Net IP - 10.10.10.12/24 (no gateway)
CEPH Cluster Net IP - 10.10.20.12/24 (no gateway)

I've only setup the above in network, I have not yet setup CEPH.

I tested pings in shell, Node 1 and Node 2 cannot ping each other on the CEPH Public or Cluster IP. They can only ping each other on the mgmt network.

Can someone confirm that this is not normal behavior. (I feel that since every node has 3 IPs, they should be able to ping each on each IP.)

Thanks

 

[aaron]

They should be able to ping each other on each network. So take a look at your network config and cabling if something is amiss there.

Also, Ceph needs at least 3 nodes to work and behave as you would expect and want it!

 

[apap]

They should be able to ping each other on each network. So take a look at your network config and cabling if something is amiss there.

Also, Ceph needs at least 3 nodes to work and behave as you would expect and want it!

It is a network issue. I had set the vlan on the switch wrong.

Which leads to the next question. I understand that it is best if CEPH Public and Cluster dataflow is on a separate network (or VLAN).

What impact would it have if they were on the same VLAN/Subnet. There is basically only 4 IPs (2 Public IP, 2 Cluster IP for 2 servers) on 2 subnets on the same network. I dont think it should have an impact, am I correct?

 

[aaron]

It really comes down to how much Ceph is utilizing the network. If the network is becoming a bottleneck, you can place the optional Ceph cluster network on another physical link to spread the load.