pveceph init --network x.x.x.x/24 needed on all nodes?

[Andrew Hart]

Hi,

I recently re-installed proxmox 5.1 and ceph and followed instructions: "After installation of packages, you need to create an initial Ceph configuration on just one node, based on your network"
I found that ceph-osd.?.log had references to the non-ceph network in it, so I ran "pveceph init --network x.x.x.x/24" on all the nodes and rebooted and the "problem" did not happen this time.

Which is the correct thing to do, one node or all nodes?

Andrew

P.S. This is what the refs look like, 172 is the ceph and 10 is web interface network.

2018-01-10 09:01:02.251764 7fb55f71a700  0 -- 172.16.0.114:6802/8991 >> 10.111.4.118:6801/11829 conn(0x5586f7a83000 :6802 s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=0).handle_connect_msg accept connect_seq 0 vs existing csq=0 existing_state=STATE_CONNECTING_WAIT_CONNECT_REPLY
2018-01-10 09:02:29.557267 7fb55ff1b700  0 -- 172.16.0.114:6802/8991 >> 10.111.4.126:6801/10947 conn(0x5586f7b40800 :6802 s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=0).handle_connect_msg accept connect_seq 0 vs existing csq=0 existing_state=STATE_CONNECTING_WAIT_CONNECT_REPLY
2018-01-10 09:03:48.463093 7fb55ef19700  0 -- 172.16.0.114:6802/8991 >> 10.111.4.122:6801/4704 conn(0x5586f7b5a800 :6802 s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=0).handle_connect_msg accept connect_seq 0 vs existing csq=0 existing_state=STATE_CONNECTING_WAIT_CONNECT_REPLY

 

[Alwin]

You only need to do the init on one node, this will set the public & cluster network in the ceph.conf. Then all the services need to be restarted to pick up the new networks. If you have different cluster and public network then the services listen on both. This way OSD recovery traffic is separated from the public network (client access).

 

[Andrew Hart]

OK, so you think that the reboot was the important part to fix the "problem"...I note that if you do not run init on all nodes then the useful links to ceph config are not created on all the nodes and "ceph -s" will not work on all the nodes.

 

[Alwin]

'ceph init' (over)writes the config. As soon as you add a ceph service to one of the nodes, the links will be created.