[Solved]Remove/Create proxmox-ceph.osd-ceph.mon node

[Sakis]

Hi,

My proxmox-ceph cluster has 7 nodes and i want to remove a particular node and move ceph osds and mon to an existing one. I have 4 ceph nodes and 3 of them have a monitor. The node i want to remove is configured also as a ceph monitor. The correct procedure as described in ceph docs is that first we add a new monitor and then remove the old one (so that we dont lose quorum wile been with 2 mons). I will have 4 monitors for some minutes until removing the old one.

When i try to add a monitor (either with pveceph createmon or gui) i get this error

unable to find local address within network '10.0.0.0/24'  (500)

This new node allready has ceph installed and hosts 2 osds so far.

My nodes network confs

auto lo
iface lo inet loopback

auto eth0
iface eth0 inet static
        address public ips
        netmask xxxx
        network xxxx
        broadcast xxxx
        gateway xxxx

auto eth1
iface eth1 inet static
address 0.0.0.0
netmask 0.0.0.0

auto vmbr1
iface vmbr1 inet manual
        bridge_ports dummy0
        bridge_stp off
        bridge_fd 0
        post-up /etc/pve/kvm-networking.sh

auto vmbr0
iface vmbr0 inet static
address 10.0.0.X
netmask 255.255.255.0
bridge_ports eth1
bridge_stp off
bridge_fd 0

Nothings strange at ceph.conf

[global]
     auth client required = cephx
     auth cluster required = cephx
     auth service required = cephx
     auth supported = cephx
     cluster network = 10.0.0.0/24
     filestore xattr use omap = true
     fsid = xxxxxxxxxxxxxxxx
     keyring = /etc/pve/priv/$cluster.$name.keyring
     osd journal size = 5120
     osd mount options xfs = rw,noatime,inode64
     osd pool default min size = 1
     public network = 10.0.0.0/24

[osd]
     keyring = /var/lib/ceph/osd/ceph-$id/keyring
     osd crush update on start = false

[mon.0]
     host = node2
     mon addr = 10.0.0.2:6789

[mon.1]
     host = node3
     mon addr = 10.0.0.3:6789

[mon.2]
     host = node7
     mon addr = 10.0.0.7:6789

I would love to fix this issue and be able to use gui or else i will have to install it manualy
http://ceph.com/docs/master/rados/operations/add-or-rm-mons/#adding-a-monitor-manual
I hope these steps work in a pveceph deployed ceph cluster.

 

[spirit]

Re: Remove/Create proxmox-ceph.osd-ceph.mon node

Hi,

what is

"

auto eth1
iface eth1 inet static
address 0.0.0.0
netmask 0.0.0.0

"

?

it should be

"
auto eth1
iface eth1 inet manual
"

if you have ip on your vmbr0

 

[Sakis]

Re: Remove/Create proxmox-ceph.osd-ceph.mon node

I followed the ceph document and was able to remove and add new monitor service in another node manually.
I also changed the network conf as you suggested. I don't know if pveceph utility can retrieve the ip of the node. It doesn't matter now.
Thanks