Ceph OSDs not being able to be configured in fully routed Environment

F

flax8334

New Member
May 27, 2026
5
0
1
I run a PVE Cluster of 3 Nodes which have a fully seperated and fully routed network intended for cephs cluster network. Due to the Loopback addresses which are getting announced via OSPF not being part of a consecutive subnet (as they are individual /32s) I have to specify more than one cluster Network in the ceph config (which is supported by ceph itself).
But apparently theres something not quite working in the config parsing part of the Ceph OSD Creation as i get the following Error:

Code: 
pveceph osd create /dev/sdc --crush-device-class ssd --encrypted 1   
Error: any valid prefix is expected rather than "192.168.99.1/32, 192.168.99.2/32, 192.168.99.3/32".
command '/sbin/ip address show to '192.168.91.1/32, 192.168.92.2/32, 192.168.93.3/32' up' failed: exit code 1

Apparently this issue has been mentioned several times dating back to 2019 Is there a know work around to this limitation?
Here is a diagram of my current topology:
1790253818374.png
The /30 P2P links run OSPF to announce the loopback addresses to each node. So pinging each loopback from each other results in success.

The Public Network is an LACP Bundle to a Set of MLAGed Switches doing the regular L2+L3 Suff.
 
tschmidt said:
Tried making em part of a single range?
Iirc thats what the SDN ospf fabric option does.
Click to expand...
That would certainly work I suppose, but this is just a test for a bigger cluster that gets deployed later in which I do not necessarily have that liberty
 
The proxmox helper scripts don't handle multiple ceph networks. They only expect to find one in the configuration file and will throw that error when doing certain operations.

When creating OSDs, we work around it by removing all the networks from the config except for the one where the OSD is being added. Once the OSDs are created, add all the networks back into the config.

When creating monitors, you can avoid the error by running
Code: 
pveceph mon create --mon-address (ip of host)
 
You must log in or register to reply here.
Top Bottom
Back