Ceph support - Not Proxmox

HireRyanToday

New Member
Mar 18, 2025
2
0
1
There’s not a whole lot of support forms on the Internet for ceph, so if this is misplaced, I apologize.

This is my little one page platform I'm putting together
https://github.com/rlewkowicz/micro-platform

I use ceph nano to launch a standalone ceph cluster:
https://github.com/rlewkowicz/micro-platform/blob/main/Docker/docker-compose.yml

It works great. Afaik, I don't think there's anything else out there this simple to have a quick functional cluster. I don't recall why I do the nginx proxy. This started on windows theres a reason. Anywho, all that works. I can telnet:
```
$ telnet ryan-Raider-GE78HX-13VI 3300
Trying 127.0.1.1...
Connected to ryan-Raider-GE78HX-13VI.
Escape character is '^]'.
ceph v2
```
From the pod I can curl:
```
# curl ryan-Raider-GE78HX-13VI:8082
<!doctype html>
<html lang="en-US">
<head>
...
...
```
Over all, pretty darn quick and pretty darn slick.

However, when I goto test rados etc it just doesn't respond:
```
# rados --debug-client=30 -n client.admin --key 'AQA8YdlnFtNzOxAAxHeGfTVsWfAsQBNw+kazcA==' -m ryan-Raider-GE78HX-13VI:3300 -p ceph-csi-pool ls
2025-03-18T17:52:46.482+0000 77326a519fc0 -1 monclient: get_monmap_and_config failed to get config
```

Now, that telnet above is from the host, as the pod doesn't have telnet but if the curl works from the pod and the telnet works from the host it all should be working and we should be cooking.

But Idk what that rados packet looks like. The mons on that nano hook to 127.0.0.1:
```
docker exec -ti ceph-nano-images ceph mon dump
epoch 1
fsid 88f15bc4-1a97-48d2-9882-bde5d8e04742
last_changed 2025-03-18T12:04:13.053266+0000
created 2025-03-18T12:04:13.053266+0000
min_mon_release 16 (pacific)
election_strategy: 1
0: [v2:127.0.0.1:3300/0,v1:127.0.0.1:6789/0] mon.ceph-nano-images-faa32aebf00b
dumped monmap epoch 1
```
I run that traefik instance to route from 127.0.0.1 etc and thought I was clever but I don't know what headers are getting passed etc.

TLDR: If a mon is bound to 127.0.0.1 even if I can route to it, will it respond correctly?
 
I don't think there's anything else out there this simple to have a quick functional cluster
For me easiest and simplest way to setup CEPH cluster - setup Proxmox cluster then activate CEPH on each node in two clicks.
CephADM - another way to setup standalone CEPH cluster, little bit more complicated but still simple enough.
 
If a MON has 127.0.0.1 as its IP there is something fundamentally wrong in the setup.
The MONs need an IP from Ceph's public_network so that they are reachable from all the other Ceph daemons and the clients.

It’s ceph nano. It’s a docker container, and originally I used it for S3 development, so that’s the only interface that needed to be functional

More so, my question is does anybody know what that packet looks like? I’m gonna go see if I can pull that apart today. I got to see what’s getting sent. I mean I linked an entire functional environment. If anybody cares to try it only takes a moment.

You can route to those just fine (due to the proxy set up) so the question is, when I connect via rados using the hostname is something funky happening with headers?

Or is my connection string not right? Because from the pod, it might be trying to use the local configuration, although that should be being overwritten with my flags.