[#681] Proxmox+Ceph: pveceph createmon: unable to find local address within network

C

cscracker

New Member
Jun 13, 2015
9
1
1
I'm following the guide to set up a four-node Proxmox Ceph cluster on my Dell c6100.
https://pve.proxmox.com/wiki/Ceph_Server

Each node has dual gigabit ethernet and dual infiniband adapters. Everything has gone smoothly until I get to pveceph createmon, where it apparently can't find an address even though it is clearly configured, available, and working.

Code: 
root@pve1:~# cat /etc/network/interfaces
# This file describes the network interfaces available on your system
# and how to activate them. For more information, see interfaces(5).


# The loopback network interface
auto lo
iface lo inet loopback


# The primary network interface
allow-hotplug eth0
iface eth0 inet manual
bond-master bond0


allow-hotplug eth1
iface eth1 inet manual
bond-master bond0


auto bond0
iface bond0 inet static
address 192.168.1.21
netmask 255.255.255.0
gateway 192.168.1.1
dns-nameservers 192.168.1.5 192.168.1.1
bond-mode 6


# infiniband interfaces
allow-hotplug ib0
iface ib0 inet static
address 10.13.37.21
netmask 255.255.255.0


allow-hotplug ib1
iface ib1 inet static
address 10.13.38.21
netmask 255.255.255.0
root@pve1:~# ping 10.13.38.22
PING 10.13.38.22 (10.13.38.22) 56(84) bytes of data.
64 bytes from 10.13.38.22: icmp_req=1 ttl=64 time=0.068 ms
64 bytes from 10.13.38.22: icmp_req=2 ttl=64 time=0.051 ms
64 bytes from 10.13.38.22: icmp_req=3 ttl=64 time=0.045 ms
^C
--- 10.13.38.22 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 1999ms
rtt min/avg/max/mdev = 0.045/0.054/0.068/0.012 ms
root@pve1:~# pveceph init --network 10.13.38.0/24
root@pve1:~# pveceph createmon
unable to find local address within network '10.13.38.0/24'

The Proxmox cluster is configured on ib0 and working properly as far as I can tell. There are no errors, and all the nodes show fine in the web interface. The ceph cluster is intended to be on ib1, but I can't create the monitor for some reason. Any ideas?

Update: For testing purposes, I also attempted to add ib1 to a bond interface, and it still failed. I additionally tried to just use the ethernet bond, and that fails too.
Code: 
root@pve1:~# pveceph init --network 192.168.1.0/24
root@pve1:~# pveceph createmon
unable to find local address within network '192.168.1.0/24'
 
Last edited:
Re: Proxmox+Ceph: pveceph createmon: unable to find local address within network

After a lot of time spent troubleshooting, running strace, and perl debugging, I figured out that it wasn't reading the "address" lines from my interfaces file.

Code: 
  DB<102> x $config
0  HASH(0x439b5e0)
   'bond0' => HASH(0x43a09a0)
      'active' => 1
      'autostart' => 1
      'method' => 'static'
      'priority' => 5
      'type' => 'bond'
   'eth0' => HASH(0x43a0850)
      'exists' => 1
      'method' => 'manual'
      'priority' => 3
      'type' => 'eth'
   'eth1' => HASH(0x439c0a8)
      'exists' => 1
      'method' => 'manual'
      'priority' => 4
      'type' => 'eth'
   'ib0' => HASH(0x42f8e00)
      'active' => 1
      'method' => 'static'
      'priority' => 6
      'type' => 'unknown'
   'ib1' => HASH(0x43a0eb0)
      'active' => 1
      'method' => 'static'
      'priority' => 7
      'type' => 'unknown'
   'lo' => HASH(0x43a0a48)
      'active' => 1
      'autostart' => 1
      'method' => 'loopback'
      'priority' => 2
      'type' => 'loopback'

I discovered that if I added tabs to the beginning of those lines, it worked.

Works:
Code: 
allow-hotplug ib1
iface ib1 inet static
        address 10.13.38.21
        netmask 255.255.255.0

Does not work:
Code: 
allow-hotplug ib1
iface ib1 inet static
address 10.13.38.21
netmask 255.255.255.0

Since the latter works with everything else, this should either be patched or at least noted in the wiki as an issue. Hopefully, at the very least, this post will show up in the Google ranking for this problem.
 
  • Like
Reactions: fibo_fr
Re: Proxmox+Ceph: pveceph createmon: unable to find local address within network

Hi,

can you try to edit
/usr/share/perl5/PVE/INotify.pm

and change the lines

if ($line =~ m/^\s*(eth\d+):.*/) {

->

if ($line =~ m/^\s*((eth|ib)\d+):.*/) {

?
 
Re: Proxmox+Ceph: pveceph createmon: unable to find local address within network

It only reads addresses "eth*" devices. Which is a somewhat limited heuristic. The above change should almost do it, except the address you're trying to use is on bond0 not ib0, so make that (eth|bond) (or include all three of them).
Ah and it only reads lines which start with whitespace as part of the previously declared interface.
So yeah you're better off using tabs than not using them.
 
Last edited:
Re: Proxmox+Ceph: pveceph createmon: unable to find local address within network

Yeah, I tried duplicating the whole if statement with ib instead of eth previously (I wasn't sure how to get the syntax right for a single statement), but it didn't solve the problem due to the whitespace issue, so I undid that change. Once I added the whitespace to the interfaces file, it did work without making that change, so I'm not sure whether it should or shouldn't be made in the future.
 
Re: Proxmox+Ceph: pveceph createmon: unable to find local address within network

You shouldn't change that file really (the PVE file).

For one thing - the spacing for the interfaces file is pretty standard how it should be laid out, which includes indenting the settings that refer to a specific interface.
for another thing - that file (the PVE file) could be changed at any time from an update, and if you revert your interfaces file to the non-spaced format you could break your system at a very bad time.
 
Re: Proxmox+Ceph: pveceph createmon: unable to find local address within network

Seems the parsing code is a bit strict there. But then again there are a bunch of options for interfaces, there are mappings, and there's no good general parser available that I can find. (All the 3rd party libraries lack a lot more than the one in PVE).
After reading the code, it should be pretty safe to change the file manually if you just stick to the following rules:
*) Indent every line part of an `iface` block by at least one whitespace character. (Improves readability anyhow, so if you don't do that you're just being silly :P)
*) Have at least one empty line after an iface block.
*) Have NO empty lines _during_ an iface block (you may have comments, though).

I'd say those requirements are pretty "sane" and only make the file more readable and maintainable.
 
Re: Proxmox+Ceph: pveceph createmon: unable to find local address within network

I'm having this same issue under the current PVE 4.0 beta. I'm using debian Jessie backports for ceph, so running at 0.80.9-2~bpo8+1.

Code: 
root@pve1:~# cat /etc/network/interfaces
# network interface settings; autogenerated
# Please do NOT modify this file directly, unless you know what
# you're doing.
#
# If you want to manage part of the network configuration manually,
# please utilize the 'source' or 'source-directory' directives to do
# so.
# PVE will preserve these directives, but will NOT its network
# configuration from sourced files, so do not attempt to move any of
# the PVE managed interfaces into external files!

auto lo
iface lo inet loopback

iface eth0 inet manual

iface eth1 inet manual

auto eth2
iface eth2 inet static
        address  192.168.2.11
        netmask  255.255.255.0

auto vmbr0
iface vmbr0 inet static
        address  10.11.111.14
        netmask  255.255.254.0
        gateway  10.11.110.1
        bridge_ports eth0
        bridge_stp off
        bridge_fd 0

auto vmbr1
iface vmbr1 inet static
        address  192.168.1.11
        netmask  255.255.255.0
        bridge_ports eth1
        bridge_stp off
        bridge_fd 0

root@pve1:~# ifconfig eth2
eth2      Link encap:Ethernet  HWaddr 52:54:00:00:24:c9
          inet addr:192.168.2.11  Bcast:192.168.2.255  Mask:255.255.255.0
          inet6 addr: fe80::5054:ff:fe00:24c9/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:2922 errors:0 dropped:0 overruns:0 frame:0
          TX packets:648 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:207454 (202.5 KiB)  TX bytes:92612 (90.4 KiB)

root@pve1:~#  pveceph createmon
unable to find local address within network '192.168.2.0/24'
Dan
 
Last edited:
Re: Proxmox+Ceph: pveceph createmon: unable to find local address within network

spirit said:
Maybe because of double spaces between "address 10.11.111.14" ?
Click to expand...

I attempted removing the double space, but it didn't make a difference. Also, the PVE web UI put the double-spaces back the next time I looked at the interfaces ;)

Dan
 
Re: Proxmox+Ceph: pveceph createmon: unable to find local address within network

I brushed off my perl brain-folds and figured out the following seems to fix it. Looks like the references to the hash values of the network settings were at the wrong level:

Code: 
root@pve1:/usr/share/perl5/PVE/API2# pwd && diff Ceph.pm Ceph.pm.orig 
/usr/share/perl5/PVE/API2
775c775
< 	my $d = $ifaces->{$iface};
---
> 	my $d = $config->{$iface};

I'll submit the same to the bugzilla report once my account there is active.

Dan
 
Re: [#681] Proxmox+Ceph: pveceph createmon: unable to find local address within netwo

i also run into this problem. running the latest beta 4 and added the gitbuilder.ceph.com repo to install ceph on it. works fine until i try to create my monitor nodes:
Code: 
# pveceph createmonunable to find local address within network '192.168.159.0/24'
Code: 
auto vmbr0iface vmbr0 inet static	address 77.xxx.xxx.xx	netmask 255.255.255.224	gateway 77.xxx.xxx.xx	bridge_ports eth0	bridge_stp off	bridge_fd 0auto vmbr1iface vmbr1 inet static        address  192.168.159.14        netmask  255.255.255.0        bridge_ports eth1        bridge_stp off        bridge_fd 0        up route add -net 224.0.0.0/4 dev vmbr1
i use the local network also for clustering proxmox.having 5 nodes HP Gen9 with 12 Disk (each 6TB) - can't wait to get ceph running3 of them still running on 3.4 without ceph. my plan was to start with ceph on 4.0 because im scared to run this already with 3.4 because i do not know if it will be easy to upgrade to 4.0 once it become stable.
 
Does this thera implies that ceph reads addresses for the interfaces file?
Rather unfortunate for me since the hi-speed private network of my hosting company is dhcp based, and so the IP is NOT in the interfaces file (even though it does not change!)
 
Just ran into this issue as well. It isn't ONLY the tabs. It is also the fact the first 2 lines of the stanza DO NOT have tabs.....
 
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom