Expose Ceph RBD to VMs for Kubernetes

T

thomas_l

Member
Nov 5, 2018
5
1
8
36
Hello,

I'd like to use Ceph embedded inside Proxmox for a Kubernetes cluster running inside VMs. This means exposing Ceph RBD endpoint to VMs.

Two questions : is it possible ? is it supported ?

I would not use Ceph for anything other than persistent storage for Kubernetes. I have a Proxmox 5.4 cluster (ZFS storage).

Thank you !
 
Hi.

Yes, it should work. (you just need network access from your vm ip to ceph monitor/osd.)

Don't know how you'll implement support of rbd inside kubernetes, but the new rook implementation don't support yet external ceph cluster (not in kubernetes). But I known that rancher for example, have a rbd plugin to do it.
 
thomas_l said:
One remaining question : is it a supported way to use Ceph in Proxmox ? =)
Unsupported setup is a good way to have a problem with an upgrade.
Click to expand...

yes, no problem.
Just be carefull when you upgrade from major proxmox (5->6 for example), as generally, the ceph version supported increase too.
(should be nautilus for the next major upgrade).
Maybe create a dedicated pool if one day you want to manage vms on ceph too.
(avoid to give permissions at kubernetes on the same pool than vm, for security purpose)
 
thomas_l said:
Two questions : is it possible ? is it supported ?
Click to expand...
thomas_l said:
One remaining question : is it a supported way to use Ceph in Proxmox ? =)
Click to expand...
If you ask with these questions, if Proxmox gives paid support for a Kubernetes + Ceph RBD setup? Then the answer is no. But from a technical standpoint, we build all packages from upstream, so there should be nothing hindering.
 
Thank you both for your answers !

Alwin said:
If you ask with these questions, if Proxmox gives paid support for a Kubernetes + Ceph RBD setup? Then the answer is no. But from a technical standpoint, we build all packages from upstream, so there should be nothing hindering.
Click to expand...
The question is not only about Kubernetes (even if it's what I plan to use), but more global; If I consume Proxmox Ceph RBD from inside VMs to create disks and mount them automatically, will it be a problem for support ?

If I understand correctly what you said, it's possible and supported, but of course you won't support what I do inside VMs (no support for Kubernetes obviously).

For now it's a homelab, but at some point, if it works correctly, I could decide to use it professionally, then : subscription.
 
  • Like
Reactions: Alwin
thomas_l said:
If I understand correctly what you said, it's possible and supported, but of course you won't support what I do inside VMs (no support for Kubernetes obviously).
Click to expand...
Proxmox VE as client or Ceph cluster is under paid support.
 
any chance anyone know of a how to do this.
have a proxmox cluster with cephcluster on top, thats providing storage pool to proxmox firstly.
have created a cephfs.
want to install/configure/get my k8s cluster running on top to use the ceph storage via csi driver as pv/pvc's.
G
 
georgelza said:
any chance anyone know of a how to do this.
have a proxmox cluster with cephcluster on top, thats providing storage pool to proxmox firstly.
have created a cephfs.
want to install/configure/get my k8s cluster running on top to use the ceph storage via csi driver as pv/pvc's.
G
Click to expand...
you can install rook operator and use it to only create persitant volume (rbd volumes) through ceph api.

https://rook.io/docs/rook/v1.10/CRDs/Cluster/external-cluster/


(by default rook can deploy ceph mon,osd,mds...but you only need to api part, as you'll use your ceph on proxmox cluster)
 
... ai ai... all of this implies allot of knowledge of the various components... sadly no idiots guide...
have Ceph deployed as cephpool on Proxmox cluster and a cephfs on the the pmox cluster.
have a basic k8s cluster deployed...
having to scan far to many url's to try and get something to work...

I realise the power of k8s... once younger it working, but it's a nightmare for ops to configured...

G
 
You must log in or register to reply here.
Top Bottom