[SOLVED] Ghost monitors, managers and meta data servers

[bitflow]

Hi. I am running Proxmox 7.0-11 with CEPH version 16.2.6. I have some trouble with the GUI Datacenter -> Ceph -> Services.

I see a few services marked as unknown for some reason that is related to CEPH.. See screenshot below.


I've tried systemctl restart pveproxy on all nodes and clearing my browser cookies but with no result. Running for example ceph mon dump shows only three monitors which is correct so I don't get where these extra ones comes from. Also checked /var/lib/ceph/[mon|mgr|mds] in the CLI and there was only one entry per service there.

Any clues on how to resolve this?

 

[bitflow]

bump

 

[dcsapak]

what does

pvesh get /cluster/ceph/status

show ?

 

[bitflow]

what does

pvesh get /cluster/ceph/status

show ?

I can't send this output to a file/clipboard it seems so I am taking a screenshot. Let me know if you need anything more or else.

 

[dcsapak]

I can't send this output to a file/clipboard

you can redirect the output into a file (also please add --output-format json):

pvesh get /cluster/ceph/status --output-format json > /tmp/status.txt

its then in the file /tmp/status.txt

 

[bitflow]

you can redirect the output into a file (also please add --output-format json):

pvesh get /cluster/ceph/status --output-format json > /tmp/status.txt

its then in the file /tmp/status.txt

Thanks. Here it is.

 

[fluxX04]

Hi,

i had the same problem: https://forum.proxmox.com/threads/duplicated-ceph-mon-mgr-and-mds-in-ceph-overview.89374/
My fix was that only the node name should be present in /etc/hostname and not the full fqdn - maybe that helps.

Greetz

 

[dcsapak]

thx, i forgot to also ask for

pvesh get /cluster/ceph/metadata --output-format json > /tmp/metadata.txt

(the gui queries both api endpoints and merges them)

 

[bitflow]

Hi,

i had the same problem: https://forum.proxmox.com/threads/duplicated-ceph-mon-mgr-and-mds-in-ceph-overview.89374/
My fix was that only the node name should be present in /etc/hostname and not the full fqdn - maybe that helps.

Greetz

Hi.

Thank you for the suggestion!

I tried updating the hostname on the three nodes without using FQDN and then I restarted mon, mgr and mds services. Unfortunately they were still present.

I rebooted a node at the time and after that the ghost services seemed to have vanished..

Is this a bug in Proxmox or Ceph? Would be nice if we could use FQDN in `/etc/hostname`.

thx, i forgot to also ask for

pvesh get /cluster/ceph/metadata --output-format json > /tmp/metadata.txt

(the gui queries both api endpoints and merges them)

See attached. I replaced the hostnames fyi.

Looks good now.

 

[dcsapak]

I tried updating the hostname on the three nodes without using FQDN and then I restarted mon, mgr and mds services. Unfortunately they were still present.

I rebooted a node at the time and after that the ghost services seemed to have vanished..

did you restart/reload the browser/ui in between?
if not that could explain it, since the ui tries to keep them.. but i'll look into it if it's a bug

Is this a bug in Proxmox or Ceph? Would be nice if we could use FQDN in `/etc/hostname`.

normally you would define both:

my.own.ip.address hostname hostname.domain.tld

EDIT: in general, the nodename must resolve to a valid ip (without the fqdn)

 

[bitflow]

did you restart/reload the browser/ui in between?
if not that could explain it, since the ui tries to keep them.. but i'll look into it if it's a bug


normally you would define both:

my.own.ip.address hostname hostname.domain.tld

I cleared the cache of my browser and tried to restart pveproxy between but it was after the reboot the ghost services disappeared.

Isn't that the format of the /etc/hosts? That file I haven't touch I only changed /etc/hostname for now. hostname command returns the short name and hostname --fqdn shows the full hostname as it should.

 

[dcsapak]

Isn't that the format of the /etc/hosts? That file I haven't touch I only changed /etc/hostname/ for now. hostname command returns the short name and hostname --fqdn shows the full hostname as it should.

ah yes sorry

/etc/hostname should only contain the nodename and the fqdn should be in /etc/resolv.conf

domain xyz.com
search xyz.com
nameserver some.dns.ip.address

EDIT again: for more details about how to set what, check 'man hostname'

 

[bitflow]

ah yes sorry

/etc/hostname should only contain the nodename and the fqdn should be in /etc/resolv.conf

domain xyz.com
search xyz.com
nameserver some.dns.ip.address

EDIT again: for more details about how to set what, check 'man hostname'

Thank you. I don't know if I should flag this as solved right now.

 

[dcsapak]

try it out. if it it does not occur after some time, please mark it as solved