rookie NFS share ceph?
- Thread starter flexyz
- Start date
You don't...
Ceph does not provide a filesystem the same way NFS does - at least not directly. Ceph provides a storage model based on "objects". An "object" is basically a blob of bits that you can access using a unique id or handle.
Ceph provides a mechanism called RBD to simulate a block device using objects. Think of a block device like a raw disk. In Ceph, a "Rados Block Device" (RBD) is made up of a collection of objects that all happen to be the same size - 4MB. But this still does not give you a filesystem that you can share over NFS. RBD is a bit like sharing from a SAN using iSCSI.
Ceph also provides a meachanism to build a POSIX filesystem out of "objects". This is called CepfFS and has just reached supportable maturity for the first time with the Ceph Jewel release. Note that Proxmox does not support or expose CephFS, at least in part because Proxmox Ceph integration is still based on the Ceph Hammer release and cephfs is considered experimental in Hammer.
The primary use case for Ceph in Proxmox - and the only method directly supported by Proxmox's integration of Ceph - is RBD for the purpose of providing a shared storage model for VM Images.
IF cephfs were supported on Proxmox you could mount a cephfs filesystem onto a server and re-share it with NFS. But that would actually be a bit silly since most current Linux distros could just be set up to "mount" the cephfs directly. I suppose it might make sense if you were sharing with non-Linux devices that could not mount the cephfs directly. But doing the "mount/re-share" does create a single point of failure and a bottleneck that is a bit at odds with overall objectives of Ceph's distributed storage model.
Ceph does not provide a filesystem the same way NFS does - at least not directly. Ceph provides a storage model based on "objects". An "object" is basically a blob of bits that you can access using a unique id or handle.
Ceph provides a mechanism called RBD to simulate a block device using objects. Think of a block device like a raw disk. In Ceph, a "Rados Block Device" (RBD) is made up of a collection of objects that all happen to be the same size - 4MB. But this still does not give you a filesystem that you can share over NFS. RBD is a bit like sharing from a SAN using iSCSI.
Ceph also provides a meachanism to build a POSIX filesystem out of "objects". This is called CepfFS and has just reached supportable maturity for the first time with the Ceph Jewel release. Note that Proxmox does not support or expose CephFS, at least in part because Proxmox Ceph integration is still based on the Ceph Hammer release and cephfs is considered experimental in Hammer.
The primary use case for Ceph in Proxmox - and the only method directly supported by Proxmox's integration of Ceph - is RBD for the purpose of providing a shared storage model for VM Images.
IF cephfs were supported on Proxmox you could mount a cephfs filesystem onto a server and re-share it with NFS. But that would actually be a bit silly since most current Linux distros could just be set up to "mount" the cephfs directly. I suppose it might make sense if you were sharing with non-Linux devices that could not mount the cephfs directly. But doing the "mount/re-share" does create a single point of failure and a bottleneck that is a bit at odds with overall objectives of Ceph's distributed storage model.
Last edited:
for rados block devices accessed using librbd or mapped using the rbd kernel module, this is no problem - the ceph cluster nodes and the clients need not be the same machines, they just need to be connected.
for PVE this means that Qemu (which has librbd support) can just access the disk images stored on a ceph cluster, for containers we map the images as local block devices using the rbd kernel module and mount them for container usage.
you only need an explicit export mechanism if you need to access the same image from multiple clients at the same time or if you cannot use either librbd or rbd mapping to access your data.
in proxmox, add an rbd storage, and add your ceph monitors ip
https://pve.proxmox.com/wiki/Ceph_Server#Ceph_Client
Last thing any hints on how to map from normal linux like Ubuntu? - I would need to map a filesystem on it of course
Thanks
Felix
Thanks
Felix
Hi,
sorry - but the object-size in ceph is 4MB (default) and has nothing to do with physical block sizes.
Code:
ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok config show | grep rbd_default_stripe_unit
"rbd_default_stripe_unit": "4194304",