[SOLVED] Kubernetes - Ceph storage not mounting

vikas027

New Member
May 27, 2019
7
4
3
41
Hello guys,

I am trying to use a persistent volume claim dynamically after defining a storage class to use Ceph Storage on a Proxmox VE 6.0-4 one node cluster.

The persistent volume gets created successfully on ceph storage, but pods are unable to mount it. It throws below error. I am not sure what is missing. Has anyone used it successfully?

Code:
Events:
  Type     Reason                  Age                  From                     Message
  ----     ------                  ----                 ----                     -------
  Warning  FailedScheduling        9m47s (x2 over 10m)  default-scheduler        pod has unbound immediate PersistentVolumeClaims
  Normal   Scheduled               9m45s                default-scheduler        Successfully assigned default/test-pod to debian-9
  Normal   SuccessfulAttachVolume  9m45s                attachdetach-controller  AttachVolume.Attach succeeded for volume "pvc-85075192-29ef-48f3-abeb-b4cab95efd19"
  Warning  FailedMount             9m38s                kubelet, debian-9        MountVolume.WaitForAttach failed for volume "pvc-85075192-29ef-48f3-abeb-b4cab95efd19" : fail to check rbd image status with: (exit status 1), rbd output: (2019-08-30 19:51:29.860874 7fc92f7fe700 -1 librbd::image::OpenRequest: failed to stat v2 image header: (1) Operation not permitted
rbd: error opening image 2019-08-30 19:51:29.860923 7fc92effd700 -1 librbd::ImageState: 0x55d0986ed4a0 failed to open image: (1) Operation not permitted
kubernetes-dynamic-pvc-b7c1b736-cb0b-11e9-afdd-5608b360c3fd: (1) Operation not permitted
)
  Warning  FailedMount  9m37s  kubelet, debian-9  MountVolume.WaitForAttach failed for volume "pvc-85075192-29ef-48f3-abeb-b4cab95efd19" : fail to check rbd image status with: (exit status 1), rbd output: (2019-08-30 19:51:30.459743 7ff58f04e700 -1 librbd::image::OpenRequest: failed to stat v2 image header: (1) Operation not permitted
2019-08-30 19:51:30.459804 7ff58e84d700 -1 librbd::ImageState: 0x558985da94a0 failed to open image: (1) Operation not permitted
rbd: error opening image kubernetes-dynamic-pvc-b7c1b736-cb0b-11e9-afdd-5608b360c3fd: (1) Operation not permitted
)

Steps to reproduce
  • Install a ceph cluster on Proxmox VE (GUI)
  • Add an OSD and make sure monitors are healthy and reachable
  • Create a Pool. The name of my pool is ceph-vm
  • Create a user as shown in GitHub gist
  • Use the gist to create RBAC, ServiceAccounts, StorageClass, Deployments, PersistentVolumeClaim and a test pod
    Code:
    ➜ ~ kubectl apply -f ceph.yaml
    serviceaccount/rbd-provisioner created
    clusterrole.rbac.authorization.k8s.io/rbd-provisioner created
    clusterrolebinding.rbac.authorization.k8s.io/rbd-provisioner created
    role.rbac.authorization.k8s.io/rbd-provisioner created
    rolebinding.rbac.authorization.k8s.io/rbd-provisioner created
    deployment.extensions/rbd-provisioner created
    secret/ceph-admin-secret created
    secret/ceph-kube-secret created
    storageclass.storage.k8s.io/rbd created
    persistentvolumeclaim/claim1 created
    pod/test-pod created
    ➜  ~ #
  • Check the rbd-provisioner-* pod log to confirm that the RBD image and PVC has been created
    Code:
    ➜ ~ # kubectl logs rbd-provisioner-98b88f5d6-2tn2q -f
    I0830 09:51:02.443771 1 main.go:85] Creating RBD provisioner ceph.com/rbd with identity: ceph.com/rbd
    I0830 09:51:02.444361 1 leaderelection.go:185] attempting to acquire leader lease default/ceph.com-rbd...
    I0830 09:51:19.837469 1 leaderelection.go:194] successfully acquired lease default/ceph.com-rbd
    I0830 09:51:19.838058 1 controller.go:631] Starting provisioner controller ceph.com/rbd_rbd-provisioner-98b88f5d6-2tn2q_ad5243f5-cb0b-11e9-afdd-5608b360c3fd!
    I0830 09:51:19.838415 1 event.go:221] Event(v1.ObjectReference{Kind:"Endpoints", Namespace:"default", Name:"ceph.com-rbd", UID:"48d8701f-bd54-43f5-a94e-03a6ccbf1e83", APIVersion:"v1", ResourceVersion:"9820", FieldPath:""}): type: 'Normal' reason: 'LeaderElection' rbd-provisioner-98b88f5d6-2tn2q_ad5243f5-cb0b-11e9-afdd-5608b360c3fd became leader
    I0830 09:51:19.938618 1 controller.go:680] Started provisioner controller ceph.com/rbd_rbd-provisioner-98b88f5d6-2tn2q_ad5243f5-cb0b-11e9-afdd-5608b360c3fd!
    I0830 09:51:19.938672 1 controller.go:987] provision "default/claim1" class "rbd": started
    I0830 09:51:19.941476 1 event.go:221] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"claim1", UID:"85075192-29ef-48f3-abeb-b4cab95efd19", APIVersion:"v1", ResourceVersion:"9781", FieldPath:""}): type: 'Normal' reason: 'Provisioning' External provisioner is provisioning volume for claim "default/claim1"
    W0830 09:51:19.944759 1 provision.go:203] error getting coredns service: services "coredns" not found. Falling back to kube-dns
    I0830 09:51:20.067213 1 provision.go:132] successfully created rbd image "kubernetes-dynamic-pvc-b7c1b736-cb0b-11e9-afdd-5608b360c3fd"
    I0830 09:51:20.067283 1 controller.go:1087] provision "default/claim1" class "rbd": volume "pvc-85075192-29ef-48f3-abeb-b4cab95efd19" provisioned
    I0830 09:51:20.067304 1 controller.go:1101] provision "default/claim1" class "rbd": trying to save persistentvvolume "pvc-85075192-29ef-48f3-abeb-b4cab95efd19"
    I0830 09:51:20.077129 1 controller.go:1108] provision "default/claim1" class "rbd": persistentvolume "pvc-85075192-29ef-48f3-abeb-b4cab95efd19" saved
    I0830 09:51:20.080161 1 controller.go:1149] provision "default/claim1" class "rbd": succeeded
    I0830 09:51:20.080627       1 event.go:221] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"claim1", UID:"85075192-29ef-48f3-abeb-b4cab95efd19", APIVersion:"v1", ResourceVersion:"9781", FieldPath:""}): type: 'Normal' reason: 'ProvisioningSucceeded' Successfully provisioned volume pvc-85075192-29ef-48f3-abeb-b4cab95efd
  • RBD Logs
    Code:
    ➜  ~ # rbd --pool ceph-vm ls -l    
    NAME SIZE PARENT FMT PROT LOCK 
    kubernetes-dynamic-pvc-b7c1b736-cb0b-11e9-afdd-5608b360c3fd 1024M 2 
    ➜ ~ #
    ➜ ~ # rbd status --pool ceph-vm kubernetes-dynamic-pvc-b7c1b736-cb0b-11e9-afdd-5608b360c3fd 
    Watchers: none
    ➜ ~ #
    ➜ ~ # rbd --pool ceph-vm info kubernetes-dynamic-pvc-b7c1b736-cb0b-11e9-afdd-5608b360c3fd
    rbd image 'kubernetes-dynamic-pvc-b7c1b736-cb0b-11e9-afdd-5608b360c3fd':
    size 1024 MB in 256 objects
    order 22 (4096 kB objects)
    block_name_prefix: rbd_data.19db76b8b4567
    format: 2
    features: layering
    flags: 
    ➜ ~ #
    ➜ ~ # kubectl get pvc
    NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
    claim1 Bound pvc-85075192-29ef-48f3-abeb-b4cab95efd19 1Gi RWO rbd 2m5s
    ➜ ~ #
    ➜ ~ # kubectl get po test-pod
    NAME READY STATUS RESTARTS AGE
    test-pod 0/1 ContainerCreating 0 2m31s
    ➜ ~ #

NOTE: I have opened this issue on Kubernetes GitHub as well

Thanks,
Vikas
 
  • Like
Reactions: Romsch and gowger
I was missing these two commands, the complete process has been described very well here.

Code:
ceph osd pool application enable kube rbd
ceph osd crush tunables hammer
 
Last edited:
  • Like
Reactions: Romsch and gowger