Hello guys,
I am trying to use a persistent volume claim dynamically after defining a storage class to use Ceph Storage on a Proxmox VE 6.0-4 one node cluster.
The persistent volume gets created successfully on ceph storage, but pods are unable to mount it. It throws below error. I am not sure what is missing. Has anyone used it successfully?
Steps to reproduce
NOTE: I have opened this issue on Kubernetes GitHub as well
Thanks,
Vikas
I am trying to use a persistent volume claim dynamically after defining a storage class to use Ceph Storage on a Proxmox VE 6.0-4 one node cluster.
The persistent volume gets created successfully on ceph storage, but pods are unable to mount it. It throws below error. I am not sure what is missing. Has anyone used it successfully?
Code:
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedScheduling 9m47s (x2 over 10m) default-scheduler pod has unbound immediate PersistentVolumeClaims
Normal Scheduled 9m45s default-scheduler Successfully assigned default/test-pod to debian-9
Normal SuccessfulAttachVolume 9m45s attachdetach-controller AttachVolume.Attach succeeded for volume "pvc-85075192-29ef-48f3-abeb-b4cab95efd19"
Warning FailedMount 9m38s kubelet, debian-9 MountVolume.WaitForAttach failed for volume "pvc-85075192-29ef-48f3-abeb-b4cab95efd19" : fail to check rbd image status with: (exit status 1), rbd output: (2019-08-30 19:51:29.860874 7fc92f7fe700 -1 librbd::image::OpenRequest: failed to stat v2 image header: (1) Operation not permitted
rbd: error opening image 2019-08-30 19:51:29.860923 7fc92effd700 -1 librbd::ImageState: 0x55d0986ed4a0 failed to open image: (1) Operation not permitted
kubernetes-dynamic-pvc-b7c1b736-cb0b-11e9-afdd-5608b360c3fd: (1) Operation not permitted
)
Warning FailedMount 9m37s kubelet, debian-9 MountVolume.WaitForAttach failed for volume "pvc-85075192-29ef-48f3-abeb-b4cab95efd19" : fail to check rbd image status with: (exit status 1), rbd output: (2019-08-30 19:51:30.459743 7ff58f04e700 -1 librbd::image::OpenRequest: failed to stat v2 image header: (1) Operation not permitted
2019-08-30 19:51:30.459804 7ff58e84d700 -1 librbd::ImageState: 0x558985da94a0 failed to open image: (1) Operation not permitted
rbd: error opening image kubernetes-dynamic-pvc-b7c1b736-cb0b-11e9-afdd-5608b360c3fd: (1) Operation not permitted
)Steps to reproduce
- Install a ceph cluster on Proxmox VE (GUI)
- Add an OSD and make sure monitors are healthy and reachable
- Create a Pool. The name of my pool is ceph-vm
- Create a user as shown in GitHub gist
- Use the gist to create RBAC, ServiceAccounts, StorageClass, Deployments, PersistentVolumeClaim and a test pod
Code:
➜ ~ kubectl apply -f ceph.yaml serviceaccount/rbd-provisioner created clusterrole.rbac.authorization.k8s.io/rbd-provisioner created clusterrolebinding.rbac.authorization.k8s.io/rbd-provisioner created role.rbac.authorization.k8s.io/rbd-provisioner created rolebinding.rbac.authorization.k8s.io/rbd-provisioner created deployment.extensions/rbd-provisioner created secret/ceph-admin-secret created secret/ceph-kube-secret created storageclass.storage.k8s.io/rbd created persistentvolumeclaim/claim1 created pod/test-pod created ➜ ~ # - Check the rbd-provisioner-* pod log to confirm that the RBD image and PVC has been created
Code:
➜ ~ # kubectl logs rbd-provisioner-98b88f5d6-2tn2q -f I0830 09:51:02.443771 1 main.go:85] Creating RBD provisioner ceph.com/rbd with identity: ceph.com/rbd I0830 09:51:02.444361 1 leaderelection.go:185] attempting to acquire leader lease default/ceph.com-rbd... I0830 09:51:19.837469 1 leaderelection.go:194] successfully acquired lease default/ceph.com-rbd I0830 09:51:19.838058 1 controller.go:631] Starting provisioner controller ceph.com/rbd_rbd-provisioner-98b88f5d6-2tn2q_ad5243f5-cb0b-11e9-afdd-5608b360c3fd! I0830 09:51:19.838415 1 event.go:221] Event(v1.ObjectReference{Kind:"Endpoints", Namespace:"default", Name:"ceph.com-rbd", UID:"48d8701f-bd54-43f5-a94e-03a6ccbf1e83", APIVersion:"v1", ResourceVersion:"9820", FieldPath:""}): type: 'Normal' reason: 'LeaderElection' rbd-provisioner-98b88f5d6-2tn2q_ad5243f5-cb0b-11e9-afdd-5608b360c3fd became leader I0830 09:51:19.938618 1 controller.go:680] Started provisioner controller ceph.com/rbd_rbd-provisioner-98b88f5d6-2tn2q_ad5243f5-cb0b-11e9-afdd-5608b360c3fd! I0830 09:51:19.938672 1 controller.go:987] provision "default/claim1" class "rbd": started I0830 09:51:19.941476 1 event.go:221] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"claim1", UID:"85075192-29ef-48f3-abeb-b4cab95efd19", APIVersion:"v1", ResourceVersion:"9781", FieldPath:""}): type: 'Normal' reason: 'Provisioning' External provisioner is provisioning volume for claim "default/claim1" W0830 09:51:19.944759 1 provision.go:203] error getting coredns service: services "coredns" not found. Falling back to kube-dns I0830 09:51:20.067213 1 provision.go:132] successfully created rbd image "kubernetes-dynamic-pvc-b7c1b736-cb0b-11e9-afdd-5608b360c3fd" I0830 09:51:20.067283 1 controller.go:1087] provision "default/claim1" class "rbd": volume "pvc-85075192-29ef-48f3-abeb-b4cab95efd19" provisioned I0830 09:51:20.067304 1 controller.go:1101] provision "default/claim1" class "rbd": trying to save persistentvvolume "pvc-85075192-29ef-48f3-abeb-b4cab95efd19" I0830 09:51:20.077129 1 controller.go:1108] provision "default/claim1" class "rbd": persistentvolume "pvc-85075192-29ef-48f3-abeb-b4cab95efd19" saved I0830 09:51:20.080161 1 controller.go:1149] provision "default/claim1" class "rbd": succeeded I0830 09:51:20.080627 1 event.go:221] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"claim1", UID:"85075192-29ef-48f3-abeb-b4cab95efd19", APIVersion:"v1", ResourceVersion:"9781", FieldPath:""}): type: 'Normal' reason: 'ProvisioningSucceeded' Successfully provisioned volume pvc-85075192-29ef-48f3-abeb-b4cab95efd - RBD Logs
Code:
➜ ~ # rbd --pool ceph-vm ls -l NAME SIZE PARENT FMT PROT LOCK kubernetes-dynamic-pvc-b7c1b736-cb0b-11e9-afdd-5608b360c3fd 1024M 2 ➜ ~ # ➜ ~ # rbd status --pool ceph-vm kubernetes-dynamic-pvc-b7c1b736-cb0b-11e9-afdd-5608b360c3fd Watchers: none ➜ ~ # ➜ ~ # rbd --pool ceph-vm info kubernetes-dynamic-pvc-b7c1b736-cb0b-11e9-afdd-5608b360c3fd rbd image 'kubernetes-dynamic-pvc-b7c1b736-cb0b-11e9-afdd-5608b360c3fd': size 1024 MB in 256 objects order 22 (4096 kB objects) block_name_prefix: rbd_data.19db76b8b4567 format: 2 features: layering flags: ➜ ~ # ➜ ~ # kubectl get pvc NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE claim1 Bound pvc-85075192-29ef-48f3-abeb-b4cab95efd19 1Gi RWO rbd 2m5s ➜ ~ # ➜ ~ # kubectl get po test-pod NAME READY STATUS RESTARTS AGE test-pod 0/1 ContainerCreating 0 2m31s ➜ ~ #
NOTE: I have opened this issue on Kubernetes GitHub as well
Thanks,
Vikas
