Unable to create new Ceph monitor - "Invalid IP string"

[alext]

Hi,

I've recently added a couple of new nodes to my system and now want to move the Ceph monitors to the new nodes, but am struggling.
When I go to Ceph - Monitor in the Proxmox GUI, click Create and select the new node, it almost instantly give me the error message "Invalid IP string", followed by the IP of one of the currently active monitors - not always the same one.

I've tried to create the Ceph monitor on other nodes as well (both new and old), but always the same problem, and not always the same IP that shows up, but always an IP of a current Ceph monitor node. I've even removed a current monitor and tried to recreate it a few minutes later, but the same problem occurred (was running on 3 monitors, now on 2 - that test may not have been my smartest decision ever...

Having a good search online, I found this article https://www.mail-archive.com/pve-devel@lists.proxmox.com/msg03848.html that lists the same exact error message, but I'm not quite sure how this would relate to my problem.

I'm running Proxmox 7.1-10 with the latest patches.

Anyone got any ideas? Thanks in advance.

 

[fiona]

Hi,
could you please share your /etc/pve/ceph.conf and output of pveversion -v?

 

[alext]

Sure!

/etc/pve/ceph.conf :

[global] auth_client_required = cephx auth_cluster_required = cephx auth_service_required = cephx cluster_network = 192.168.1.0/24 fsid = 1166d897-f578-49fc-a943-008a01b67dc1 mon_allow_pool_delete = true mon_host = prox3,prox6,prox7 osd_journal_size = 5120 osd_pool_default_min_size = 2 osd_pool_default_size = 3 public_network = 192.168.1.0/24 [client] keyring = /etc/pve/priv/$cluster.$name.keyring [mds] keyring = /var/lib/ceph/mds/ceph-$id/keyring [mds.prox3] host = prox3 mds_standby_for_name = pve [mds.prox8] host = prox8 mds_standby_for_name = pve [mds.prox9] host = prox9 mds_standby_for_name = pve [mon.prox3] host = prox3 mon_addr = 192.168.1.83:6789 [mon.prox6] host = prox6 mon_addr = 192.168.1.86:6789

pveversion -v

proxmox-ve: 7.1-1 (running kernel: 5.13.19-4-pve) pve-manager: 7.1-10 (running version: 7.1-10/6ddebafe) pve-kernel-helper: 7.1-12 pve-kernel-5.13: 7.1-7 pve-kernel-5.11: 7.0-10 pve-kernel-5.13.19-4-pve: 5.13.19-9 pve-kernel-5.13.19-3-pve: 5.13.19-7 pve-kernel-5.13.19-2-pve: 5.13.19-4 pve-kernel-5.13.19-1-pve: 5.13.19-3 pve-kernel-5.11.22-7-pve: 5.11.22-12 pve-kernel-5.11.22-2-pve: 5.11.22-4 ceph: 16.2.7 ceph-fuse: 16.2.7 corosync: 3.1.5-pve2 criu: 3.15-1+pve-1 glusterfs-client: 9.2-1 ifupdown: 0.8.36+pve1 ksm-control-daemon: 1.4-1 libjs-extjs: 7.0.0-1 libknet1: 1.22-pve2 libproxmox-acme-perl: 1.4.1 libproxmox-backup-qemu0: 1.2.0-1 libpve-access-control: 7.1-6 libpve-apiclient-perl: 3.2-1 libpve-common-perl: 7.1-3 libpve-guest-common-perl: 4.1-1 libpve-http-server-perl: 4.1-1 libpve-storage-perl: 7.1-1 libqb0: 1.0.5-1 libspice-server1: 0.14.3-2.1 lvm2: 2.03.11-2.1 lxc-pve: 4.0.11-1 lxcfs: 4.0.11-pve1 novnc-pve: 1.3.0-2 proxmox-backup-client: 2.1.5-1 proxmox-backup-file-restore: 2.1.5-1 proxmox-mini-journalreader: 1.3-1 proxmox-widget-toolkit: 3.4-6 pve-cluster: 7.1-3 pve-container: 4.1-4 pve-docs: 7.1-2 pve-edk2-firmware: 3.20210831-2 pve-firewall: 4.2-5 pve-firmware: 3.3-5 pve-ha-manager: 3.3-3 pve-i18n: 2.6-2 pve-qemu-kvm: 6.1.1-2 pve-xtermjs: 4.16.0-1 qemu-server: 7.1-4 smartmontools: 7.2-pve2 spiceterm: 3.2-2 swtpm: 0.7.0~rc1+2 vncterm: 1.7-1 zfsutils-linux: 2.1.2-pve1

 

[fiona]

Sure!

/etc/pve/ceph.conf :

[global] auth_client_required = cephx auth_cluster_required = cephx auth_service_required = cephx cluster_network = 192.168.1.0/24 fsid = 1166d897-f578-49fc-a943-008a01b67dc1 mon_allow_pool_delete = true mon_host = prox3,prox6,prox7 osd_journal_size = 5120 osd_pool_default_min_size = 2 osd_pool_default_size = 3 public_network = 192.168.1.0/24 [client] keyring = /etc/pve/priv/$cluster.$name.keyring [mds] keyring = /var/lib/ceph/mds/ceph-$id/keyring [mds.prox3] host = prox3 mds_standby_for_name = pve [mds.prox8] host = prox8 mds_standby_for_name = pve [mds.prox9] host = prox9 mds_standby_for_name = pve [mon.prox3] host = prox3 mon_addr = 192.168.1.83:6789 [mon.prox6] host = prox6 mon_addr = 192.168.1.86:6789

Was this configuration created via Proxmox VE? If yes, approximately what version?

It usually looks like

mon_host = 10.10.22.11 10.10.22.12
...
[mon.node1]
         public_addr = 10.10.22.11

and I guess in your case, the port number might confuse our handling.

Is /etc/ceph/ceph.conf a symbolic link to /etc/pve/ceph.conf on each node? If not, could you share those as well?

 

[alext]

Hi Fabian,

Thanks for your help!

>> Was this configuration created via Proxmox VE? If yes, approximately what version?

Yes, I think it was originally created on version 5.4-ish.
I do seem to remember an upgrade of corosync from version 5 to 6 or so that was changing port numbers / protocol versions? Maybe that's where the port number got introduced? ...or am I speaking nonsense now...

>> and I guess in your case, the port number might confuse our handling.

Makes sense... It should be OK if I manually remove the port numbers from the file /etc/pve/ceph.conf?

>> Is /etc/ceph/ceph.conf a symbolic link to /etc/pve/ceph.conf on each node?

Yes it is on all nodes.

 

[fiona]

Hi Fabian,

Thanks for your help!

>> Was this configuration created via Proxmox VE? If yes, approximately what version?

Yes, I think it was originally created on version 5.4-ish.
I do seem to remember an upgrade of corosync from version 5 to 6 or so that was changing port numbers / protocol versions? Maybe that's where the port number got introduced? ...or am I speaking nonsense now...

It's true, but corosync and Ceph are not really related
Thanks, I was now able to reproduce the issue! I'll see about fixing it, as old configurations should of course continue to work.

>> and I guess in your case, the port number might confuse our handling.

Makes sense... It should be OK if I manually remove the port numbers from the file /etc/pve/ceph.conf?

Yes, that should work. EDIT: Not sure if Ceph will be happy about it after all.

>> Is /etc/ceph/ceph.conf a symbolic link to /etc/pve/ceph.conf on each node?

Yes it is on all nodes.

 

[alext]

>> Thanks, I was now able to reproduce the issue! I'll see about fixing it, as old configurations should of course continue to work.

Great! Having a quick look on Ceph's pages, I keep seeing the mon_addr in the config files WITH the port number, so I'll await the outcome of your assessment before I try it on my cluster.

Thanks so far!

 

[fiona]

Yes, it might not be a good idea to change it after all (even if it's just the port number).

If you need the new monitor quickly, try adding it manually via Ceph instead of Proxmox VE. Otherwise, let's hope the patch doesn't take too long.

 

[fiona]

I sent a preliminary patch. You might want to switch your mon_host config line to use spaces as separators instead for consistency (if you create a new monitor it'll be separated by a space). Also, the prox7 that's listed there doesn't seem to have a mon.prox7 section further down (unless it got cut off when posting).

 

[alext]

>> I sent a preliminary patch.

Brilliant! Will keep an eye out for it. As long as nothing else goes wrong at the moment, should be fine running off 2 monitors for now.
Will update this thread if that's gone to plan.

>> You might want to switch your mon_host config line to use spaces as separators instead for consistency (if you create a new monitor it'll be separated by a space).

Will have a look at that when it's back to normal again, thanks.

>> Also, the prox7 that's listed there doesn't seem to have a mon.prox7 section further down (unless it got cut off when posting).

Correct, this was the node that I removed the monitor from in order to reinstall it and see if that would work (which it didn't). It's surprising though that it didn't clear out of the entry from the mon_host entry - though maybe that's because of the forementioned comma-separated issue?

 

[fiona]

>> Also, the prox7 that's listed there doesn't seem to have a mon.prox7 section further down (unless it got cut off when posting).

Correct, this was the node that I removed the monitor from in order to reinstall it and see if that would work (which it didn't). It's surprising though that it didn't clear out of the entry from the mon_host entry - though maybe that's because of the forementioned comma-separated issue?

I can't see where Proxmox VE ever used comma-separated hostnames for mon_host. AFAICT it always used space-separated IPs for mon_host, see here. We did change how the mon sections are generated.

 

[alext]

Hi Fabian,

Do you know if the patch will be rolled out soon? I did go for the manual route and that did the trick for now, but it seems to cause some trouble when rebooting the box (monitor doesn't start automatically) - putting it back the way ProxMox likes it seems to be the most suitable option so would like to recreate the monitor using ProxMox' GUI.

Thanks,

 

[fiona]

Unfortunately, the patch hasn't been reviewed and tested by others yet. I guess it was missed, because of all the changes for the recent release. I bumped it now, so let's hope somebody has time to check it out.

 

[alext]

Thanks for the update. Will keep an eye out for it and at least good to know I haven't missed it somewhere

 

[fiona]

It's been applied to git now and will be available in pve-manager 7.2-5, but might still take a while until that is packaged/uploaded to repos.

 

[juanyque]

F.Y.I
I had the same problem as alext (error on create new monitors).
I have deleted the port on /etc/pve/ceph.conf and all works again. The result has been good, both by PROXMOX GUI and shell.

[global]
...
         mon_host = 1.2.3.4 1.2.3.5
...

[mon.node5]
         public_addr = 1.2.3.5

[mon.node4]
         host = node4
         mon_addr = 1.2.3.4

Note differences on old (node4) and new (node5) config params.

 

[alext]

After upgrading to 7.2-5, the error indeed disappeared!
Have removed my manually created monitors and recreated the monitors via the GUI and can see in the Ceph config that the port numbers are no longer showing for the newly created monitors. (for good measure, have replaced all monitors to clean up the config if a bit Thanks Fabian!