[TUTORIAL] Request for community help - S3 on Ceph cluster

[SpeakEZ.ai]

We are soo... close! I have an NFS share set up with turnkey file server and a cephFS instance for the "backplane". I've made some headway with getting cephadm set up and sharing some of the managers and monitors between them (though that still has some rough edges) but I *still* don't *quite* have Object Gateway/S3 APIs up and running.







I have followed a few different resources - primarily the Ceph documentation and this page. But as you can see I'm not quite there, yet. If anyone wants to help nail this down with Proxmox 8.x and Ceph Reef 18.x I'd be glad to take notes and confirm the "garden path" as it emerges. I've already found a few wrinkles in the extant resources and how they diverge from current resource paths. So I imagine there will be a varity of sharp edges to discover and smooth out along the way. If anyone's interested in helping out, give a shout!

 

[fakebizprez]

Figure it out?

We are soo... close! I have an NFS share set up with turnkey file server and a cephFS instance for the "backplane". I've made some headway with getting cephadm set up and sharing some of the managers and monitors between them (though that still has some rough edges) but I *still* don't *quite* have Object Gateway/S3 APIs up and running.

View attachment 67941

View attachment 67942

View attachment 67943

I have followed a few different resources - primarily the Ceph documentation and this page. But as you can see I'm not quite there, yet. If anyone wants to help nail this down with Proxmox 8.x and Ceph Reef 18.x I'd be glad to take notes and confirm the "garden path" as it emerges. I've already found a few wrinkles in the extant resources and how they diverge from current resource paths. So I imagine there will be a varity of sharp edges to discover and smooth out along the way. If anyone's interested in helping out, give a shout!

 

[SpeakEZ.ai]

Actually - not yet. I lost my quorum with Hurricane Helene (power outage took out one of my nodes - through the UPS, no less) so I've put it on pause for now. I'm still very interested in getting this up and running but have other projects on-deck so I'll circle back to it sometime in 2025. For now I'm using Cloudflare R2.

 

[fakebizprez]

Actually - not yet. I lost my quorum with Hurricane Helene (power outage took out one of my nodes - through the UPS, no less) so I've put it on pause for now. I'm still very interested in getting this up and running but have other projects on-deck so I'll circle back to it sometime in 2025. For now I'm using Cloudflare R2.

I'll be tackling this soon..let you know what I come away with.

 

[SpeakEZ.ai]

I'll be tackling this soon..let you know what I come away with.

That's great - thanks for being the latest guinea pig in getting this to work. I "broke down" and put a NFS/Samba server on a node that is running 20TB of NVMe storage so I can shuttle around media files between Windows systems. Two of them attach the network via ZeroTier and there were some wrinkles getting the SAN to let the traffic in from that sub-net. AMA!

But back on topic: I'm sure you may know this but spelunking through threads here - there are a few resources in wiki pages and blog posts, but I honestly had a hard time getting them to harmonize into a cohesive "garden path" that got things fully working. There are also some YouTube videos from

Daniel Persson - https://www.youtube.com/@DanielPersson

...who seems to be pretty straightforward about getting it to work. Maybe I was just trying to reconcile too many resources and got lost/confused...

 

[fakebizprez]

@SpeakEZ.ai mission accomplished: https://pve.proxmox.com/wiki/User:Grin/Ceph_Object_Gateway

I followed those instructions.

Additionally, I could be wrong, but I believe the Ceph Native Dashboard that you posted screenshots of no longer functions with Proxmox. There's an issue open on GitHub with maintainers working on it, from the Ceph side, but from what I've read, it looks like a bear of a task.

This isn't the greatest solution, but I used https://s3browser.com/ in lieu of the Ceph dashboard. It works OK.

This actually inspired me to create a TUI for the RADOS Gateway, using Go. My plate is full so I doubt I'll be able to finish that until Summer.

 

[SpeakEZ.ai]

Thanks for keeping this on a side burner. I'm not hung up on the Ceph Native Dashboard so much as I was using it as a means to verify that I had set things up properly. I'd be curious to know if the Persson guidance was otherwise correct. Can you confirm that?

As a sidebar on my own three-node cluster - through part-swapping I finally figured out I lost one of the three - even on the "other side" of a proper UPS - due to the PCIe 10G NIC failing - to the point it would prevent the system from posting. The truly odd thing is that it's connected via fiber so I'm completely clueless as to how/why that happened - other than happenstance of being at the same time as the storm wailed on us. Such as it is I've got to pick up a new 10G NIC and go through the motions of "knocking out the cobwebs" in getting Ceph/S3 operating over that network/switch. (FWIW I'm using a Mikrotik CRS309-1G-8S+) so I have some head-scratching to do once I get back into the lab to install the new card and set up that machine.