Ceph cluster/private network

[jop]

Hello,

I currently have this CEPH configuration on a three-node Proxmox 5.1 network:



[global]
auth client required = cephx
auth cluster required = cephx
auth service required = cephx
cluster network = 172.16.0.0/12
fsid = xxxxxxxxxxxxxxx
keyring = /etc/pve/priv/$cluster.$name.keyring
mon allow pool delete = true
osd journal size = 5120
osd pg bits = 14
osd pgp bits = 14
osd pool default min size = 2
osd pool default size = 3
public network = 172.16.0.0/12

[osd]
keyring = /var/lib/ceph/osd/ceph-$id/keyring

[mon.appcl02]
host = appcl02
mon addr = 172.17.0.2:6789

[mon.appcl01]
host = appcl01
mon addr = 172.17.0.1:6789

[mon.appcl03]
host = appcl03
mon addr = 172.17.0.3:6789


A bit often I see in the dmesg of the three nodes the following message:

[21587785.122771] libceph: mon2 172.17.0.3:6789 session lost, hunting for new mon
[21587786.730591] libceph: mon0 172.17.0.1:6789 session established
[21587815.847358] libceph: mon0 172.17.0.1:6789 session lost, hunting for new mon
[21587815.848485] libceph: mon2 172.17.0.3:6789 session established
[21587846.567845] libceph: mon2 172.17.0.3:6789 session lost, hunting for new mon
[21587846.569205] libceph: mon1 172.17.0.2:6789 session established
[21587877.284345] libceph: mon1 172.17.0.2:6789 session lost, hunting for new mon
[21587878.760801] libceph: mon0 172.17.0.1:6789 session established
[21587908.004866] libceph: mon0 172.17.0.1:6789 session lost, hunting for new mon


I've read somehwere here that it's a bad practice to put the ceph cluster network in the same network as the public network. But does this create performance/additional problems like the above?
And, if that is the case..how could I migrate to separate network without creating problems? The cluster is in production, with a hundred of VMs running.
Thanks

 

[sb-jw]

What CEPH Version is currently running on your Cluster? Are all Nodes on the same Versions?

What about your Network?

 

[jop]

Ceph version is:
ceph version 12.2.2 (215dd7151453fae88e6f968c975b6ce309d42dcf) luminous (stable)

And of course yes, all versions are the same.
What do you want to know about the network?

 

[sb-jw]

I cant understand why People who need help are not able to give some Basic Information.

What do you want to know about the network?

Everything.

I recommend you to upgrade, it seems there are other People with an similar Problem.
https://tracker.ceph.com/issues/23537

 

[jop]

I cant understand the attitude.
I simply asked what other details you'd need, because data regarding network addressing both for public and cluster network are already in the configuration I specified. Thanks anyway, I can use google search ;-)

 

[udo]

I cant understand the attitude.
I simply asked what other details you'd need, because data regarding network addressing both for public and cluster network are already in the configuration I specified. Thanks anyway, I can use google search ;-)

Hi,
I guess sb-jw mean something like:
What kind of network: 10GB SFP+, 1GB, or anything else
Type of connections: bonding (which mode)...
different switches?
Are vm+ceph-Network the same?
MTU...

BTW an /12 network looks, that there are a lot of other traffic (many broadcasts)?! Do you think it's an good idea to use such an network for ceph?

Udo

 

[jop]

Thanks, now it's more clear

Network is 10Gbps: all three main servers are in the same network, no switches in between. vm and ceph live in the same network, though VM has vlan tagging and run in their own vlan.

I know that /12 network is too large. In fact I'm asking how to migrate ceph to a different network without losing data/having downtime.
Thank you
J

 

[alexskysilk]

[edit- whoops, deleted part of the post to fix the numbers OP, please repost from your email notification, I forgot what I wrote.]