Hi all,
I have a Proxmox cluster with 3 nodes and Ceph storage, I installed version 1.28/stable of microk8s on 3 VMs with the command:
I would like to use Proxmox Ceph cluster as shared storage for microk8s cluster and so I enabled rook-ceph and this is the status of the cluster:
I then took the ceph.conf and ceph.client.admin.keyring files from the Proxmox cluster and used them for the connection command:
which however returns an error:
I am aware that the problem is not related to Proxmox, which works perfectly, but if anyone has already done a similar configuration and can help me I would be very grateful.
I have a Proxmox cluster with 3 nodes and Ceph storage, I installed version 1.28/stable of microk8s on 3 VMs with the command:
sudo snap install microk8s --classic --channel=1.28/stableI would like to use Proxmox Ceph cluster as shared storage for microk8s cluster and so I enabled rook-ceph and this is the status of the cluster:
Code:
$ microk8s status
microk8s is running
high-availability: yes
datastore master nodes: 10.15.10.121:19001 10.15.10.122:19001 10.15.10.123:19001
datastore standby nodes: none
addons:
enabled:
dns # (core) CoreDNS
ha-cluster # (core) Configure high availability on the current node
helm # (core) Helm - the package manager for Kubernetes
helm3 # (core) Helm 3 - the package manager for Kubernetes
rook-ceph # (core) Distributed Ceph storage using Rook
disabled:
cert-manager # (core) Cloud native certificate management
cis-hardening # (core) Apply CIS K8s hardening
community # (core) The community addons repository
dashboard # (core) The Kubernetes dashboard
gpu # (core) Automatic enablement of Nvidia CUDA
host-access # (core) Allow Pods connecting to Host services smoothly
hostpath-storage # (core) Storage class; allocates storage from host directory
ingress # (core) Ingress controller for external access
kube-ovn # (core) An advanced network fabric for Kubernetes
mayastor # (core) OpenEBS MayaStor
metallb # (core) Loadbalancer for your Kubernetes cluster
metrics-server # (core) K8s Metrics Server for API access to service metrics
minio # (core) MinIO object storage
observability # (core) A lightweight observability stack for logs, traces and metrics
prometheus # (core) Prometheus operator for monitoring and logging
rbac # (core) Role-Based Access Control for authorisation
registry # (core) Private image registry exposed on localhost:32000
storage # (core) Alias to hostpath-storage add-on, deprecatedI then took the ceph.conf and ceph.client.admin.keyring files from the Proxmox cluster and used them for the connection command:
$ sudo microk8s connect-external-ceph --ceph-conf ceph.conf --keyring ceph.client.admin.keyring --rbd-pool microk8s-rbdwhich however returns an error:
Code:
Attempting to connect to Ceph cluster
Successfully connected to cd8ad1c4-fde4-4a3a-b169-3f4ba4762d4b (10.15.15.121:0/3422868158)
WARNING: Pool microk8s-rbd already exists
Configuring pool microk8s-rbd for RBD
Successfully configured pool microk8s-rbd for RBD
Creating namespace rook-ceph-external
Error from server (AlreadyExists): namespaces "rook-ceph-external" already exists
Configuring Ceph CSI secrets
Traceback (most recent call last):
File "/var/snap/microk8s/common/plugins/connect-external-ceph", line 184, in <module>
main()
File "/snap/microk8s/6089/usr/lib/python3/dist-packages/click/core.py", line 764, in __call__
return self.main(*args, **kwargs)
File "/snap/microk8s/6089/usr/lib/python3/dist-packages/click/core.py", line 717, in main
rv = self.invoke(ctx)
File "/snap/microk8s/6089/usr/lib/python3/dist-packages/click/core.py", line 956, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/snap/microk8s/6089/usr/lib/python3/dist-packages/click/core.py", line 555, in invoke
return callback(*args, **kwargs)
File "/var/snap/microk8s/common/plugins/connect-external-ceph", line 169, in main
import_external_ceph_cluster(ceph_conf, keyring, namespace, rbd_pool)
File "/var/snap/microk8s/common/plugins/connect-external-ceph", line 109, in import_external_ceph_cluster
p = subprocess.run(
File "/snap/microk8s/6089/usr/lib/python3.8/subprocess.py", line 516, in run
raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['/snap/microk8s/6089/usr/bin/python3', PosixPath('/var/snap/microk8s/common/plugins/.rook-create-external-cluster-resources.py'), '--format=bash', '--rbd-data-pool-name=microk8s-rbd', '--ceph-conf=ceph.conf', '--keyring=ceph.client.admin.keyring']' returned non-zero exit status 1.I am aware that the problem is not related to Proxmox, which works perfectly, but if anyone has already done a similar configuration and can help me I would be very grateful.
