MinIO is dead... What's next?

A

adhoore

New Member
Nov 18, 2024
5
0
1
Hey all,

I'm the administrator for a bunch of servers here at the university (both students and researchers).
We have quite a lot of data science / machine learning projects, and we store the data in locally hosted S3 buckets.

For the last few years, I have just been spinning up a separate MinIO server for each project, or for a group of related projects.
These MinIO servers are just VMs on top of Proxmox, with disks on Proxmox Ceph, and backed up to PBS (Proxmox Backup Server).
So I have done this for years: MinIO service (S3 API) > VM disks > Proxmox Ceph (block storage).

Now that MinIO is officially dead: https://www.reddit.com/r/selfhosted/comments/1pd97nq/minio_is_in_maintenance_mode_and_is_no_longer/

I'm wondering what to do next. I know that Reddit /r/selfhosted is a fan of Garage, so I'm wondering if I should go that route.

The thing is: we are already heavily using Ceph for all our Proxmox storage. All the disks in our system are in Ceph pools. All VM/LXC disks are on Ceph.
I'm also not scared to learn and invest time into configuring Ceph Object Gateway if that will give me a nice, stable, locally hosted S3 system.

My fear is just that I don't like messing with the Proxmox host OS. I basically NEVER install any extra services on the Proxmox host itself.
This is a real production environment. I have around 100 students and researchers using these clusters.
I want to stay on the Proxmox "officially supported" path.

So my question to you all is: how far "off the happy path" do I stray when I do this?

Thank you for your advice,
Alexander
 
Similar boat here looking at similar options, only played with rustfs since it was pretty easy to stand up. I dont think I would "extend" the Ceph that proxmox provides. Treat that and keep it as a separate thing.

Rustfs, lots of promise, still a bit beta though... releases weekly. No SSE-KMS or Kafka file event notification. Installation/configuration is very much like minio'ish. I'm already a bit fan but probably wouldn't use in production yet. Nice UI too. Might be enough for your needs perhaps? I have a small cluster for testing and am quite excited by what it offers already.

Other ones I've looked at but haven't tested yet.

Seaweed, Garage, NVIDIA AiStore and Apache Ozone

We do have HW coming for a dedicated ceph cluster to run the Object Gateway as I'm fine learning to drive Ceph. Lots of potential options!
 
  • Like
Reactions: adhoore
You say RustFS is "still a bit beta", and that just isn't good enough for me.
I need mature software that I can deploy now and keep running for years.
On that front I fully trust Ceph to be stable and supported.

As a counterexample, the Garage S3 object store looks like an interesting project.
However, I don't see any commercial support or company backing.
Most Reddit users report it works "quite well". Yeah... that's not good enough.

I trust Ceph. I trust Proxmox.
But Ceph Object Store in Proxmox... is that supported?
If no, should that worry me, or should I just do it anyway?
 
adhoore said:
You say RustFS is "still a bit beta", and that just isn't good enough for me.
I need mature software that I can deploy now and keep running for years.
On that front I fully trust Ceph to be stable and supported.

As a counterexample, the Garage S3 object store looks like an interesting project.
However, I don't see any commercial support or company backing.
Most Reddit users report it works "quite well". Yeah... that's not good enough.

I trust Ceph. I trust Proxmox.
But Ceph Object Store in Proxmox... is that supported?
If no, should that worry me, or should I just do it anyway?
Click to expand...

I dont know if there is a specific "This is not supported" if you wanted to run the Object Gateway. I suspect due to the packaged nature of Ceph, there could be troubles. While PVE provides upstream ceph, they do their own packaging.. Not sure if they have all the deps that would be required for object gateway?

https://forum.proxmox.com/threads/ceph-object-rgw.160958/

The most recent example I could find.

I personally wouldn't co-mingle S3 Gateway with the Proxmox provided ceph. There's got to be other threads I'm sure but I haven't looked too hard. Especially since the current setup you have is production.

I do agree with both of your points on rustfs and even garage. However I watch both with great interest! Probably rustfs more so.
 
  • Like
Reactions: adhoore
adhoore said:
But Ceph Object Store in Proxmox... is that supported?
Click to expand...
By whom and for what purpose? its actively developed, if that's what you mean, but Proxmox has no tooling or integrated support. I've ran RGW in the past on Proxmox and the experience has been... mixed. tooling is a problem and there are conflicts. I would counsel NOT running RGW on PVE.

Minio the open source project may be dead, but depending on what your use case is, Minio users have three options:
1. shift to a fork. see https://github.com/OpenMaxIO/openmaxio-object-browser
2. shift to Minio's commercial product (AI Stor)
3. replace the whole thing with another solution. Yes, you can make a go at it with ceph, seeweed, rustfs, etc. ALL available options should be considered "beta" and have significant gotchas, but as long as you are able to read, not panic, and have understanding customers it should be possible.
 
  • Like
Reactions: Johannes S and cfgmgr
alexskysilk said:
Minio the open source project may be dead, but depending on what your use case is, Minio users have three options:
1. shift to a fork. see https://github.com/OpenMaxIO/openmaxio-object-browser
2. shift to Minio's commercial product (AI Stor)
3. replace the whole thing with another solution. Yes, you can make a go at it with ceph, seeweed, rustfs, etc. ALL available options should be considered "beta" and have significant gotchas, but as long as you are able to read, not panic, and have understanding customers it should be possible.
Click to expand...
Up to now we used to have a mature open-source option for hosting S3 on-prem (minio).

Now you say: "ALL available options should be considered beta"

That's a huge regression and I feel like it leaves a void, which Ceph Object Storage on Proxmox could fill.

My personal conclusion here is that I will wait a while, before migrating to something else.

(However, thank you for your advice. I appreciate it.)
 
Last edited:
You must log in or register to reply here.
Top Bottom