Ceph completely broken - Error got timeout (500)

[spamsam]

I have a test cluster of 3 VMs in virtualbox on my pc.

Their IPs changed as follows:
A: 172.16.0.150 > 192.168.100.1 (NOT a gateway)
A: 172.16.0.151 > 192.168.100.2
A: 172.16.0.152 > 192.168.100.2

Now I get the following message when i run 'service ceph-mon@local status':

Processor -- bind unable to bindto v2:172.16.0.150:3300/0: (99) Cannot assign requested address.

And this is driving me mad as it downed ceph on all 3 nodes with the same error.

I updated the address in the /etc/ceph/ceph.conf (which is just a symlink to /etc/pve/ceph.conf). It is as follows:

[global]
     auth_client_required = cephx
     auth_cluster_required = cephx
     auth_service_required = cephx
     cluster_network = 192.168.100.1/24
     fsid = bbc8efc5-af69-4460-8e18-c5d5e76d0c9e
     mon_allow_pool_delete = true
     mon_host =  192.168.100.1
     ms_bind_ipv4 = true
     ms_bind_ipv6 = false
     osd_pool_default_min_size = 2
     osd_pool_default_size = 3
     public_network = 192.168.100.1/24

[client]
     keyring = /etc/pve/priv/$cluster.$name.keyring

[mon.localhost]
     public_addr = 192.168.100.1

I rebooted the node multiple times, same error. I scrubbed the whole system of every instance of 172.16.0.150, same error.

IT is still a fresh cluster that I am using to test ceph quincy on proxmox 7.3 prior to going into production. Tho at the moment i am weary.

Any advise here?

 

[spamsam]

Just as further info, commands like ''ceph -s" hang indefinitely. all pveceph commands say timeout

 

[Moayad]

Hi,

Is the above Ceph config is from /etc/pve/ceph.conf? if yes can you compare it with /etc/ceph/ceph.conf as well?

Did you see anything interesting in the ceph monitor logs?

 

[jaykavathe]

Did you find any resolution for this? I am in the same boat and getting help from folks here but wondering what fixed it for you.. if it did.

 

[Whatever]

Did you find any resolution for this? I am in the same boat and getting help from folks here but wondering what fixed it for you.. if it did.

check /etc/hosts and update accordantly

 

[Maximiliano]

Hello, could you please post you MON map?

ceph --admin-daemon /run/ceph/ceph-mon.$(hostname).asok mon_status

Should provide plenty info. In case the networks are wrong there I would advice to take a look at [1].

[1] https://docs.ceph.com/en/latest/cephadm/services/mon/#moving-monitors-to-a-different-network

 

[EagleBBS]

ceph fs error got timeout (500) during fist time installation... any idea?