Creating a Ceph monitor failed with No active IP for requested public network

[webthingz]

Hi All, I'm new to the Proxmox community and just bought 3x community licenses for my new cluster Yeahh! But, I'm having a problem with my installed cluster with 3 nodes and each of them has 2 disks where the seconds disk will be used for ceph.
I have a public ip with 51.15.x.x and configured a private IP with 172.17.x.x on all 3 nodes. The cluster works find each node can ping eachother (ip + hostname).
Now I was trying to use Ceph and installation is done. With the first installation on the configuration tab I selected the Public IP for Public Network and the private IP for the cluster network. The first monitor is installed and seems to be fine. I think I messed up my config file also..

[global]
auth_client_required = cephx
auth_cluster_required = cephx
auth_service_required = cephx
cluster_network = 172.17.33.0/24 (I think i changed this from the IP 172.17.33.11/24 to the network address)
fsid = 0a82c9e9-0d02-4c5f-b7c6-b779d5cc9b3b
mon_allow_pool_delete = true
mon_host = xxx.xxx.xxx.xxx (I added here all public IP's from all nodes...)
ms_bind_ipv4 = true
ms_bind_ipv6 = false
osd_pool_default_min_size = 2
osd_pool_default_size = 3
public_network = xxx.xxx.xxx..0/24 (same thing change the IP xxx.xxx.xxx.xxx/24 to network address)

[client]
keyring = /etc/pve/priv/$cluster.$name.keyring

[mon.lotus]
public_addr = xxx.xxx.xxx.xxx

Is there someone here who could help me with a correct config file? And what could be the problem.

 

[webthingz]

I'm stuck... I'm now wondering if my network config is working ok at all... I'm at Scaleway where I have 3 dedicated servers. They all have 2 nics where one is with a Public IP and the other is with a Private IP. Now I've been looking trough the network manual (https://pve.proxmox.com/wiki/Network_Configuration) and see 3 options. At the moment i'm reinstalling the servers with Debian and installing it like that because the default install is configure with raid. Can someone tell me what is best practice if you have 2 nics? They also have a RPNv2 available and with this I can create a vlan for these servers to communicate. What network config should I choose?

 

[Falk R.]

I'm stuck... I'm now wondering if my network config is working ok at all... I'm at Scaleway where I have 3 dedicated servers. They all have 2 nics where one is with a Public IP and the other is with a Private IP. Now I've been looking trough the network manual (https://pve.proxmox.com/wiki/Network_Configuration) and see 3 options. At the moment i'm reinstalling the servers with Debian and installing it like that because the default install is configure with raid. Can someone tell me what is best practice if you have 2 nics? They also have a RPNv2 available and with this I can create a vlan for these servers to communicate. What network config should I choose?

Hi,

please redo the Ceph configuration again.
The public network should not be in the internet public network.
This option is only for large clusters to separate the Ceph client traffic from the backend traffic. For simple setups, always use only one network, which is definitely private, because there your VM data will be transferred in clear text.

Here is my configuration:

[global]
     auth_client_required = cephx
     auth_cluster_required = cephx
     auth_service_required = cephx
     cluster_network = 192.168.250.1/24
     fsid = 81eb2e57-0d05-4098-bb99-1c6f42aed98b
     mon_allow_pool_delete = true
     mon_host = 192.168.250.1 192.168.250.2 192.168.250.3
     ms_bind_ipv4 = true
     ms_bind_ipv6 = false
     osd_pool_default_min_size = 2
     osd_pool_default_size = 2
     public_network = 192.168.250.1/24

[client]
     keyring = /etc/pve/priv/$cluster.$name.keyring

[mon.srv1]
     public_addr = 192.168.250.1

[mon.srv2]
     public_addr = 192.168.250.2

[mon.srv3]
     public_addr = 192.168.250.3

P.S.
Actually, you should read up on a topic like Ceph before you just click on it. The storage is the foundation for all VMs and containers, so you should approach it with some care.

 

[webthingz]

Hi! Thanks for your reply! Ok, i think i have to go some steps back and find out how I can configure my cluster in a private network. Like I said Scaleway is offering a RPN service and i'm still finding out how to use this. So when i'm done with that I will return on this issue. But I hope this problem will be no issue then. I find that the proxmox manual is sometimes hard to read for beginners where i'm missing some basics or in more dept explanations. But anyway , now looking for a good explanation how to setup my servers with a private network.

Here https://pve.proxmox.com/wiki/Network_Configuration I found this:

Default Configuration using a Bridge​

auto lo
iface lo inet loopback

iface eno1 inet manual

auto vmbr0
iface vmbr0 inet static
address 192.168.10.2/24
gateway 192.168.10.1
bridge-ports eno1
bridge-stp off
bridge-fd 0


here is my config:
Where eno1 is my nic with the public address. Is this the right way to config the network?

auto lo
iface lo inet loopback

iface eno1 inet manual

iface eno2 inet manual

auto vmbr0
iface vmbr0 inet static
address xxx.xxx.xxx.xxx/24
gateway xxx.xxx.xxx.xxx
bridge-ports eno1
bridge-stp off
bridge-fd 0

auto vmbr1
iface vmbr1 inet static
address 192.168.1.10/24
bridge-ports eno2.100
bridge-stp off
bridge-fd 0

- eno2.100 is the port Scaleway tell's me to config for the private network.
- Do I need to configure vmbr0 if I don't want to use public network?

 

[Falk R.]

auto lo
iface lo inet loopback

iface eno1 inet manual

iface eno2 inet manual

auto vmbr0
iface vmbr0 inet static
address xxx.xxx.xxx.xxx/24
gateway xxx.xxx.xxx.xxx
bridge-ports eno1
bridge-stp off
bridge-fd 0

auto vmbr1
iface vmbr1 inet static
address 192.168.1.10/24
bridge-ports eno2.100
bridge-stp off
bridge-fd 0

- eno2.100 is the port Scaleway tell's me to config for the private network.
- Do I need to configure vmbr0 if I don't want to use public network?

The configuration looks OK so far.
What surprises me is that you should set VLAN 100 on port eno2. Do you also get other VLANs for other services on the network card?
You definitely need vmbr0 for access to the Internet. But you don't have to give the bridge an IP if you access your PVE host differently.
You could let a virtual firewall talk about vmbr0 on the public network and give the host an IP on another internal bridge, which you can then use to manage the host behind the firewall.

 

[Maximiliano]

Hello, first of all can all nodes in the cluster talk with each other over the plublic_network? Note that the public network is also where clients interact with the Ceph cluster so they also need to reach this network. Whats the output of `ceph status`?

Do note that monitors should be on the public network, not the cluster network (in case it is a different network) for them to work properly [1]. For a small cluster or a simple setup I would advice to use the same network for both the public and cluster network.

Also note that when changing the IP of a monitor its easier to do it by destroying a monitor with `pveceph mon destroy` and creating it again with `pveceph mon create`. Also make sure that you always have at least 2 monitors working before modifying any other, otherwise they lose quorum.

172.16.0.0/20 (172.16.0.0–172.31.255.255) is a private network so there is no harm in sharing the IPs being used, which would be helpful to understand the setup.

Finally, you also need to restart the Ceph service for network changes in ceph.conf to take effect (`systemctl restart ceph.target`).

[1] https://docs.ceph.com/en/latest/rados/configuration/network-config-ref/

 

[webthingz]

Thanks for your answer Maximiliano. I must say that you are confusing me even more haha. From a lot of corners I asked my question, I received the comment that I should not want to use my proxmox in a Public Network (IP is connected to internet). So in this case I have to install the server on the Public address and need to configure the private IP later. The servers are in a vlan @ my provider so and on my servers I add the Bridge to eno2 with vlan 100 (eno.100)
Again this is my net config at the moment.

server 1:

auto lo
iface lo inet loopback

auto eno1
iface eno1 inet manual

iface eno2 inet manual

auto vmbr0
iface vmbr0 inet static
address xxx.xxx.xxx.xxx
gateway xxx.xxx.xxx.xxx
bridge-ports eno1
bridge-stp off
bridge-fd 0

auto vmbr1
iface vmbr1 inet static
address 192.168.10.10/24
bridge-ports eno2.100
bridge-stp off
bridge-fd 0

server 2:

auto lo
iface lo inet loopback

auto eno1
iface eno1 inet manual

iface eno2 inet manual

auto vmbr0
iface vmbr0 inet static
address xxx.xxx.xxx.xxx
gateway xxx.xxx.xxx.xxx
bridge-ports eno1
bridge-stp off
bridge-fd 0

auto vmbr1
iface vmbr1 inet static
address 192.168.10.20/24
bridge-ports eno2.100
bridge-stp off
bridge-fd 0

server 3:

auto lo
iface lo inet loopback

auto eno1
iface eno1 inet manual

iface eno2 inet manual

auto vmbr0
iface vmbr0 inet static
address xxx.xxx.xxx.xxx
gateway xxx.xxx.xxx.xxx
bridge-ports eno1
bridge-stp off
bridge-fd 0

auto vmbr1
iface vmbr1 inet static
address 192.168.10.30/24
bridge-ports eno2.100
bridge-stp off
bridge-fd 0

--

They can all communicate on public vmbr0 and private vmbr1. So that is no problem. I reinstalled everyting because I broke Ceph again :-/ and did not know how to solve. So now I'm at the point of creating my cluster.

Should I choose the public IP what is connected to the internet or go for the private ? If I do pick one Can I change it later on?

After that what should I choose for my public and cluster network in nde Ceph config?

I'm sorry if this is noobie haha but I don't feel that comfort on this because reinstalling these servers took me some time. The default installation with my provider is with Raid with no other option to choose from. So I have first had to install Debian and after that Proxmox.

 

[Maximiliano]

Ceph's `public_network` is the network on which the OSDs, MONs, and clients communicate, basically the primary network on which Ceph traffic happens. It can be a private network, even if the name does not suggest it.

> Should I choose the public IP what is connected to the internet or go for the private ? If I do pick one Can I change it later on?

If you need to access the Cluster from the outside, use a public IP, if not a private one. But yes, you can switch both Ceph's networks in the future, just note that it is not as simple as changing the IPs in the config, you need to restart services, recreate MONs, etc, in such a way that the cluster remains operational throughout the process.

> After that what should I choose for my public and cluster network in nde Ceph config?

Since this is a small cluster/simple setup, I would recommend to use the same network for both the `public_network` and the `cluster_network`. In principle it can be any network, but I strongly recommend to use a dedicated network for Ceph, the only important part is that Clients (e.g. VMs using this storage for their images) need to be able to access this network.

 

[webthingz]

Ceph's `public_network` is the network on which the OSDs, MONs, and clients communicate, basically the primary network on which Ceph traffic happens. It can be a private network, even if the name does not suggest it.

> Should I choose the public IP what is connected to the internet or go for the private ? If I do pick one Can I change it later on?

If you need to access the Cluster from the outside, use a public IP, if not a private one. But yes, you can switch both Ceph's networks in the future, just note that it is not as simple as changing the IPs in the config, you need to restart services, recreate MONs, etc, in such a way that the cluster remains operational throughout the process.

> After that what should I choose for my public and cluster network in nde Ceph config?

Since this is a small cluster/simple setup, I would recommend to use the same network for both the `public_network` and the `cluster_network`. In principle it can be any network, but I strongly recommend to use a dedicated network for Ceph, the only important part is that Clients (e.g. VMs using this storage for their images) need to be able to access this network.

Thanks! You made it clear for me. Now I had to move my project to a new provider because I only had 2 disks and I installing the server without raid was not supported. Because the new provider costs are higher I need to switch to antoher way to create some redundancy. So I will have to come back on this later. The plan is to use ZFS for now with 2 servers and add a disk later when I have my 3th server. I have 2 servers now with each of them having 2 disks. With this provider I'm having the option to create a complete custom setup

Is it best practice for not using ZFS on a boot/os partition? Maybe I should have used 3 disks so I could use 2 disks with hardware raid :-/ and the 3th with ZFS... darn...