Ceph add monitor any valid prefix is expected rather than "192.168.0.0/24, 192.168.1.0/24".

[TheMrg]

pveceph mon create
Error: any valid prefix is expected rather than "192.168.0.0/24, 192.168.1.0/24".
command '/sbin/ip address show to '192.168.0.0/24, 192.168.1.0/24' up' failed: exit code 1

what is the reason?
proxmox 6.1-8
ceph 14.2.8

ceph.conf:
[global]
<------> auth_client_required = cephx
<------> auth_cluster_required = cephx
<------> auth_service_required = cephx
<------> cluster_network = 192.168.0.0/24
<------> fsid = 00cf7097-2cb5-47f6-b342-e55094bf839e
<------> mon_host = 192.168.0.20 192.168.0.2 192.168.0.1 192.168.0.3
<------> mon_initial_members = storage3 storage2
<------> public_network = 192.168.0.0/24, 192.168.1.0/24

[client]
<------> keyring = /etc/pve/priv/$cluster.$name.keyring

PS:
Create new OSDs on storage4 works without any issue.
We create 4 new OSDs successfully.

Thanks

 

[fabian]

that part of the code does not handle multiple networks correctly. could you file a bug at https://bugzilla.proxmox.com please?

 

[Alwin]

There are patches available but they are not yet packaged.
https://bugzilla.proxmox.com/show_bug.cgi?id=2422

EDIT: As a workaround, use only one network in the ceph.conf on service creation.

 

[TheMrg]

thanks.
Ok, is there an other workaround? f.e. create mon with ceph not with proxmox?
or how are the steps? should i edit the ceph.conf ?
we need the two networks.

Thanks so much. system is productive, so our ceph cluster have to stay online.

 

[TheMrg]

ok we edit manually the ceph config. add monitor and add the other network again to ceph.conf.
But with monitor we get:

Apr 14 12:43:37 storage4 ceph-mon[125140]: 2020-04-14 12:43:37.908 7f68e236e700 -1 mon.storage4@-1(probing) e0 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied

manually edit the keyring to the key ring of other mons:
/var/lib/ceph/mon/ceph-storage4
mon is up, but in log:
Apr 14 13:08:03 storage4 ceph-mon[168545]: 2020-04-14 13:08:03.531 7f4f6446e700 -1 mon.storage4@4(electing) e9 failed to get devid for : fallback method has serial ''but no model

and is it a good idea to manually change the /var/lib/ceph/mon/ceph-storage4/keyring file?

Thanks so much

 

[Alwin]

Both networks need to be routed between each other.

and is it a good idea to manually change the /var/lib/ceph/mon/ceph-storage4/keyring file?

If the keyring or the MON exists on the cluster, then yes.

 

[TheMrg]

Both networks need to be routed between each other.


If the keyring or the MON exists on the cluster, then yes.

Thanks for answer. What do you mean with: "Both networks need to be routed between each other." ?

ok now we have 4 mon nodes with all the same /var/lib/ceph/mon/ceph-..../keyring file. hope this is ok. storage1-3 hase the same file (key). add new monitor on storage4, /var/lib/ceph/mon/ceph-storage4/keyring get an other key. we delete the key and added the key from
/var/lib/ceph/mon/ceph-storage1/keyring.

now all 4 /var/lib/ceph/mon/ceph-storageN/keyring files are the same.

 

[Alwin]

Thanks for answer. What do you mean with: "Both networks need to be routed between each other." ?

Traffic from one network needs to be able to pass into the other network. Form all Ceph nodes.

ok now we have 4 mon nodes with all the same /var/lib/ceph/mon/ceph-..../keyring file. hope this is ok. storage1-3 hase the same file (key). add new monitor on storage4, /var/lib/ceph/mon/ceph-storage4/keyring get an other key. we delete the key and added the key from
/var/lib/ceph/mon/ceph-storage1/keyring.

Everything should be done automatically by the pveceph mon create. Once the multiple networks work with our tooling, the manual edit of the ceph.conf shouldn't be necessary anymore.

 

[TheMrg]

Thanks, but with pveceph mon create we get:
mon.storage4@-1(probing) e0 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied

we edit the /var/lib/ceph/mon/ceph-storage4/keyring to the key all other mon nodes uses,
now all is fine
hope this is ok. sometimes we get:
(electing) e9 failed to get devid for : fallback method has serial ''but no model

 

[Alwin]

(electing) e9 failed to get devid for : fallback method has serial ''but no model

This message is not related to that issue. It doesn't hinder running Ceph.

 

[TheMrg]

thanks very much.
ok, it's from ceph monitor:

Apr 14 13:19:13 storage4 ceph-mon[168545]: 2020-04-14 13:19:13.760 7f4f6446e700 -1 mon.storage4@4(electing) e9 failed to get devid for : fallback method has serial ''but no model
Apr 14 13:19:13 storage4 ceph-mon[168545]: 2020-04-14 13:19:13.760 7f4f6446e700 -1 mon.storage4@4(electing) e9 failed to get devid for : fallback method has serial ''but no model
Apr 14 13:19:13 storage4 ceph-mon[168545]: 2020-04-14 13:19:13.760 7f4f6446e700 -1 mon.storage4@4(electing) e9 failed to get devid for : fallback method has serial ''but no model
Apr 14 13:20:12 storage4 ceph-mon[168545]: 2020-04-14 13:20:12.707 7f4f6446e700 -1 mon.storage4@3(electing) e10 failed to get devid for : fallback method has serial ''but no model
Apr 14 13:20:12 storage4 ceph-mon[168545]: 2020-04-14 13:20:12.707 7f4f6446e700 -1 mon.storage4@3(electing) e10 failed to get devid for : fallback method has serial ''but no model
Apr 14 13:20:17 storage4 ceph-mon[168545]: 2020-04-14 13:20:17.507 7f4f6446e700 -1 mon.storage4@3(electing) e10 failed to get devid for : fallback method has serial ''but no model
Apr 15 07:59:23 storage4 ceph-mon[168545]: 2020-04-15 07:59:23.343 7f4f6446e700 -1 mon.storage4@3(electing) e10 failed to get devid for : fallback method has serial ''but no model
Apr 15 07:59:23 storage4 ceph-mon[168545]: 2020-04-15 07:59:23.343 7f4f6446e700 -1 mon.storage4@3(electing) e10 failed to get devid for : fallback method has serial ''but no model
Apr 15 07:59:23 storage4 ceph-mon[168545]: 2020-04-15 07:59:23.343 7f4f6446e700 -1 mon.storage4@3(electing) e10 failed to get devid for : fallback method has serial ''but no model
Apr 15 07:59:50 storage4 ceph-mon[168545]: 2020-04-15 07:59:50.832 7f4f6446e700 -1 mon.storage4@2(electing) e11 failed to get devid for : fallback method has serial ''but no model
Apr 15 07:59:50 storage4 ceph-mon[168545]: 2020-04-15 07:59:50.832 7f4f6446e700 -1 mon.storage4@2(electing) e11 failed to get devid for : fallback method has serial ''but no model
Apr 15 07:59:52 storage4 ceph-mon[168545]: 2020-04-15 07:59:52.840 7f4f6446e700 -1 mon.storage4@2(electing) e11 failed to get devid for : fallback method has serial ''but no model