question on kubernetes + proxmox + persistent volume

[Mani]

We are setting up Kubernetes cluster using proxmox, I want to know what is the best method to provide persistent volume.
i.e what is the best way to provide persistent volume in terms of stability & easy to handle to customers and admins.

If there is any link or any experience kindly share it.

 

[bbgeek17]

The best way to integrate persistent volumes with K8s is to use CSI compliant storage https://kubernetes.io/blog/2019/01/15/container-storage-interface-ga/.

I dont believe there is an out-of-the-box supported implementation provided by PVE.
There are drivers which align with PVE supported backends, like:
https://github.com/ceph/ceph-csi
https://github.com/metal-stack/csi-lvm
and many others. They all would be running in parallel to PVE and must be carefully isolated so they dont step on each other.

Also see discussion here:
https://forum.proxmox.com/threads/proxmox-cloud-provider-for-kubernetes.35864/

We allow our users to easily run multiple platforms in parallel via native multi-tenancy:
https://kb.blockbridge.com/guide/proxmox/
https://kb.blockbridge.com/guide/kubernetes/
https://kb.blockbridge.com/guide/vmware/
https://kb.blockbridge.com/guide/openstack/

---

Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[Mani]

The best way to integrate persistent volumes with K8s is to use CSI compliant storage https://kubernetes.io/blog/2019/01/15/container-storage-interface-ga/.

I dont believe there is an out-of-the-box supported implementation provided by PVE.
There are drivers which align with PVE supported backends, like:
https://github.com/ceph/ceph-csi
https://github.com/metal-stack/csi-lvm
and many others. They all would be running in parallel to PVE and must be carefully isolated so they dont step on each other.

Also see discussion here:
https://forum.proxmox.com/threads/proxmox-cloud-provider-for-kubernetes.35864/

We allow our users to easily run multiple platforms in parallel via native multi-tenancy:
https://kb.blockbridge.com/guide/proxmox/
https://kb.blockbridge.com/guide/kubernetes/
https://kb.blockbridge.com/guide/vmware/
https://kb.blockbridge.com/guide/openstack/

---

Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

Thanks, Will go through it.

Anyone tried openebs (https://openebs.io/) ?

 

[kamzata]

We are setting up Kubernetes cluster using proxmox, I want to know what is the best method to provide persistent volume.
i.e what is the best way to provide persistent volume in terms of stability & easy to handle to customers and admins.

If there is any link or any experience kindly share it.

I have the same question. What's the best way to integrate a storage solution for a K8s cluster of Virtual Machine in Proxmox? Maybe a NFS share?

 

[bbgeek17]

What's the best way to integrate a storage solution for a K8s cluster

There is very rarely "best way" to do anything in computers. It depends on your requirements, familiarity with solution and, often, budget.
Is NFS better than Ceph, LVM or Blockbridge? All 4 are completely different approaches with different outcomes.
You have a hypervisor at your disposal, spin up a quick "NFS server" in an extra VM and see if it works for you. Certainly it seems like one of the simplest approaches, with least amount of resources required. That is unless you start adding HA and performance on top of it.

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[kamzata]

There is very rarely "best way" to do anything in computers. It depends on your requirements, familiarity with solution and, often, budget.
Is NFS better than Ceph, LVM or Blockbridge? All 4 are completely different approaches with different outcomes.
You have a hypervisor at your disposal, spin up a quick "NFS server" in an extra VM and see if it works for you. Certainly it seems like one of the simplest approaches, with least amount of resources required. That is unless you start adding HA and performance on top of it.

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

What about adding HA and performance? CAS solutions like OpenEBS Mayastor seems a great choice to me but it isn't ready for production yet and cStor I've heard it's very slow.

 

[bbgeek17]

What about adding HA and performance?

not sure what you are asking. Adding HA to a homebrew NFS server and trying to achieve 500,000 IOPS, while running in a VM - may be possible, however, imho, is not a simple task. Investing in HA NFS NAS solution is a different conversation.

CAS solutions like OpenEBS Mayastor seems a great choice to me but it isn't ready for production yet and cStor I've heard it's very slow.

If you have budget to spend on this project - there are probably commercial offerings that will likely satisfy your needs and be ready for production while being fast.
Look for storage with CSI driver and evaluate their features and functionality as they relate to your requirements.

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[wolfspyre]

I think it's safe to say there's myriad paths to 'functional'. but many are silly, or expensive, or just overly complicated
I think it's ALSO reasonable to ask, and hope to have answered, the distilled question of

> what's the most-likely-to-be-pleasant-path for a moderately performant multinode solution

my "homelab" is a 6-way cluster of dell r720xd's each host populated with about 20 400G ssds (each) running ceph
I'd like to:
- utilize as much existing tooling as reasonable
- stay within arms reach of a `supported` configuration
- avoid any glaring anti-patterns
- try not to make the PVE engineers silently weep.
- try to encourage a path that "generally doesn't suck" for a large percentage of peeps wanting to use pve to facilitate a k8s cluster
- achieve reasonable performance for a small production environment.
- still be manageable/affordable for the average homelab


ceph-csi **SEEMS** to fit the bill...
Is there a moderately well trodden path here?
Does anyone know of hidden gotchas here that future-me would appreciate being informed of before wandering thru the minefield barefoot?

Is there anything else worth evaluating?

 

[kamzata]

I think it's safe to say there's myriad paths to 'functional'. but many are silly, or expensive, or just overly complicated
I think it's ALSO reasonable to ask, and hope to have answered, the distilled question of

> what's the most-likely-to-be-pleasant-path for a moderately performant multinode solution

my "homelab" is a 6-way cluster of dell r720xd's each host populated with about 20 400G ssds (each) running ceph
I'd like to:
- utilize as much existing tooling as reasonable
- stay within arms reach of a `supported` configuration
- avoid any glaring anti-patterns
- try not to make the PVE engineers silently weep.
- try to encourage a path that "generally doesn't suck" for a large percentage of peeps wanting to use pve to facilitate a k8s cluster
- achieve reasonable performance for a small production environment.
- still be manageable/affordable for the average homelab


ceph-csi **SEEMS** to fit the bill...
Is there a moderately well trodden path here?
Does anyone know of hidden gotchas here that future-me would appreciate being informed of before wandering thru the minefield barefoot?

Is there anything else worth evaluating?

To me, OpenEBS and CAS approach seem the future of K8s' storage. It seems really well integrated to K8s (Mayastor seems really promising!). However, I didn't try yet so... I could be wrong of course.

 

[Urbaman]

Hi,

I'm trying to achieve a stable cluster of k8s VMs. It's a 3xPVE node, the VMs are on a Ceph pool of fast nvme.
Actually using Longhorn on virtual disks on the same Ceph pool, but it seems to always fail when the vm backup jobs run or when I do restart the pve nodes for maintenance: I always have to restart pods cause storage attaching fails.
I also lost some postgres dbs in the last iteration...

I also implemented a Truenas VM with some NFS shares (hdd on ceph and ssd on ceph), those are probably more stable, using them for cooler data (docs, media). Do not know if it would be better to add another NFS share on faster storage (ceph on nvme).

Is there some insight on the more stable way to reach this (OpenESB, Rook, ...)?

 

[Urbaman]

I did try some different storage solutions for K8s running on Proxmox VMs, and probably I have found out some useful things to think about before deploying your cluster for choosing the best storage solution for your persistent volumes.

• If you have Ceph running on Proxmox, the best way is Ceph CSI or Rook with external Cluster based on Proxmox's. You can have both RBD (block devices) and CephSF (File access) possibilities.
• All other Storage solutions (GlusterFS, Longhorn, NFS, OpenEBS) are better on baremetal and not on virtual disks, you should only try with disk passthrough so that you avoid adding Ceph or ZFS lags to Gluster, NFS, Longhorn or OpenEBS lags. They need other resources to run (other VMs, k8s resources, ...) and only add to overall storage lag if not on their own environment.

Proxmox Clusters with enough nodes and resources (proper 10G+ networks for Ceph, disks) can set up Ceph and/or CephFS pools and access them via Rook or Ceph CSI, and that's the better quite native solution, so I think you should go there if you can, both for stability, reliability and performances.

After many tries, I decided to stick with Ceph, as it's native with Proxmox and because of my disk distribution (mostly 2 disks of same tipe and dimension per node, so can't passthrough them to all of the k8s workers, and it could be a real pain to differentiate them by tipe/dimension on all of those other storage solutions).

This way I can easily get all that I need on a stable and performant distributed storage.

 

[lknite]

@Urbaman did you happen to write up how to connect to the proxmox ceph via k8s or have a reference? I know I'll have to somehow export the ceph resources and import those in each of my clusters with ceph csi; looking for an article to help me save some time if possible

 

[lknite]

I know kubernetes but am less familiar with ceph. I can see proxmox is working with ceph and there are a couple pools, but in my setup cephfs is not yet enabled, so do I need to enable a metadata server and then enable cephfs? I mean for proxmox to be working it seems like ceph should have rbd or cephfs running no?

Here's my current config so far:

$ cat values.yaml
ceph-csi-rbd:

  csiConfig:
  - clusterID: "1c3f0c6f-696c-4cc0-b29c-e68f73ba9e4b"
    monitors:
    - "10.0.0.21:6789"
    - "10.0.0.23:6789"
    - "10.0.0.22:6789"
    rbd:
      netNamespaceFilePath: ""
      mirrorDaemonCount: 1
    readAffinity:
      enabled: true
      crushLocationLabels:
      - topology.kubernetes.io/region
      - topology.kubernetes.io/zone

  storageClass:
    # Specifies whether the storageclass should be created
    create: true
    name: ceph-rbd
    annotations:
      storageclass.kubernetes.io/is-default-class: "true"

    clusterID: "1c3f0c6f-696c-4cc0-b29c-e68f73ba9e4b"
    pool: "ceph-rbd"

The above is working for rbd. After I get rbd and cephfs working I'll write something up. Don't quite understand why "netNamespaceFilePath" set to "" works, or what the bug is that the github page is talking having to do with that variable.