Expose Ceph RBD to VMs for Kubernetes

[thomas_l]

Hello,

I'd like to use Ceph embedded inside Proxmox for a Kubernetes cluster running inside VMs. This means exposing Ceph RBD endpoint to VMs.

Two questions : is it possible ? is it supported ?

I would not use Ceph for anything other than persistent storage for Kubernetes. I have a Proxmox 5.4 cluster (ZFS storage).

Thank you !

 

[spirit]

Hi.

Yes, it should work. (you just need network access from your vm ip to ceph monitor/osd.)

Don't know how you'll implement support of rbd inside kubernetes, but the new rook implementation don't support yet external ceph cluster (not in kubernetes). But I known that rancher for example, have a rbd plugin to do it.

 

[thomas_l]

Thank you for your answer !

I don't plan on using Rook since Ceph will be managed by Proxmox. Kubernetes can access it directly : https://kubernetes.io/docs/concepts/storage/storage-classes/#ceph-rbd

One remaining question : is it a supported way to use Ceph in Proxmox ? =)
Unsupported setup is a good way to have a problem with an upgrade.

 

[spirit]

One remaining question : is it a supported way to use Ceph in Proxmox ? =)
Unsupported setup is a good way to have a problem with an upgrade.

yes, no problem.
Just be carefull when you upgrade from major proxmox (5->6 for example), as generally, the ceph version supported increase too.
(should be nautilus for the next major upgrade).
Maybe create a dedicated pool if one day you want to manage vms on ceph too.
(avoid to give permissions at kubernetes on the same pool than vm, for security purpose)

 

[Alwin]

Two questions : is it possible ? is it supported ?

One remaining question : is it a supported way to use Ceph in Proxmox ? =)

If you ask with these questions, if Proxmox gives paid support for a Kubernetes + Ceph RBD setup? Then the answer is no. But from a technical standpoint, we build all packages from upstream, so there should be nothing hindering.

 

[thomas_l]

Thank you both for your answers !

If you ask with these questions, if Proxmox gives paid support for a Kubernetes + Ceph RBD setup? Then the answer is no. But from a technical standpoint, we build all packages from upstream, so there should be nothing hindering.

The question is not only about Kubernetes (even if it's what I plan to use), but more global; If I consume Proxmox Ceph RBD from inside VMs to create disks and mount them automatically, will it be a problem for support ?

If I understand correctly what you said, it's possible and supported, but of course you won't support what I do inside VMs (no support for Kubernetes obviously).

For now it's a homelab, but at some point, if it works correctly, I could decide to use it professionally, then : subscription.

 

[Alwin]

If I understand correctly what you said, it's possible and supported, but of course you won't support what I do inside VMs (no support for Kubernetes obviously).

Proxmox VE as client or Ceph cluster is under paid support.