Proxmox 5.2 + Ceph 12 Upgrade to 10G/s

[Sergii]

Hello.
I have next ceph configuration: :

# ceph.conf
[global]
     auth client required = none
     auth cluster required = none
     auth service required = none
     cluster network = 172.16.6.0/24
     filestore xattr use omap = true
     keyring = /etc/pve/priv/$cluster.$name.keyring
     mon allow pool delete = true
     osd journal size = 5120
     osd pool default min size = 1
     public network = 172.16.6.0/24

[osd]
     keyring = /var/lib/ceph/osd/ceph-$id/keyring

[mon.pve06]
     host = pve06
     mon addr = 172.16.6.18:6789

[mon.pve04]
     host = pve04
     mon addr = 172.16.6.16:6789

[mon.pve01]
     host = pve01
     mon addr = 172.16.6.14:6789

Network 172.16.6.0/24 is ceph cluster/public network and proxmox management network.
We would to upgrade ceph cluster network to 10Gb/s and split proxmox and ceph to different connections. We have installed additional network adapters to each node.
How I can change ceph subnet without cluster reinitialization?
Is enouth to change this option: cluster network = 172.16.99.0/24? (For example)

Do I need to add options like this for each OSD?:
[osd.X]
public addr = {host-public-ip-address}
cluster addr = {host-cluster-ip-address}[/CODE]

Or will be better to move ceph cluster completelly to another subnet?:

cluster network = 172.16.99.0/24
public network = 172.16.99.0/24

Thanks all.

 

[AlexLup]

Hi,
Adding cluster network and reloading ceph should suffice.

Here are the commands to do a live reload..

systemctl stop ceph\*.service ceph\*.target
systemctl start ceph.target

Specifying host is only needed for mons, unless you wanna be super picky about it and define hosts in osds as well...

 

[Sergii]

Hi,
Adding cluster network and reloading ceph should suffice.

Here are the commands to do a live reload..

systemctl stop ceph\*.service ceph\*.target
systemctl start ceph.target

Specifying host is only needed for mons, unless you wanna be super picky about it and define hosts in osds as well...

Work fine.
I restarted ceph without stops VMs.
Thanks.

 

[udo]

Work fine.
I restarted ceph without stops VMs.
Thanks.

Hi,
but if you changed the cluster network only, only the replica-network-traffic is on 10GB!
Normaly you also want the public network on 10GB too - the mon-access and the kvm-processes to access the osds are using the public network!

10GB-cluster network is for speed up writes (and resyncs) only.

Udo

 

[Sergii]

Hi,
but if you changed the cluster network only, only the replica-network-traffic is on 10GB!
Normaly you also want the public network on 10GB too - the mon-access and the kvm-processes to access the osds are using the public network!

10GB-cluster network is for speed up writes (and resyncs) only.

Udo

Hi.
My goal by now was split network connections. Ceph doc recommends use different networks for cluster and public network.

 

[udo]

Hi.
My goal by now was split network connections. Ceph doc recommends use different networks for cluster and public network.

Yes it's recomend - but it's make absolut no sense if your private network is on 1GB!!
The goal to splitting networks is not to saturate the client network by resync data - in your case the 1GB network is the bottleneck and the 10Gb-network idle around...

Udo

 

[Sergii]

Yes it's recomend - but it's make absolut no sense if your private network is on 1GB!!
The goal to splitting networks is not to saturate the client network by resync data - in your case the 1GB network is the bottleneck and the 10Gb-network idle around...

Udo

Ok.
Is enough to change public network and restart ceph on all notes for copplete moving?
P.S. I have dual 10G adapters (Mellanox) but now can use only 1 port on each adapter.