Multiple cephfs ?!
- Thread starter fxandrei
- Start date
So it seems its actually an experimental feature.
https://docs.ceph.com/docs/master/cephfs/experimental-features/
So i was asking this because i wanted to mount a cepfs from one cluster to another proxmox cluster, just do backups on them.
And i was thinking of doing that on other clusters as well.
So i would have 2 clusters that would have the same external cephfs mounted.
Im thinking that there would be a problem because each cluster will have a vm with id 100.
So my question now would be, how could i sepparate the vm backups if i have a cephfs mounted on 2 clusters ?
https://docs.ceph.com/docs/master/cephfs/experimental-features/
So i was asking this because i wanted to mount a cepfs from one cluster to another proxmox cluster, just do backups on them.
And i was thinking of doing that on other clusters as well.
So i would have 2 clusters that would have the same external cephfs mounted.
Im thinking that there would be a problem because each cluster will have a vm with id 100.
So my question now would be, how could i sepparate the vm backups if i have a cephfs mounted on 2 clusters ?
Well yeah, but i want to be able to use as a storage in the cluster. Im guessing that would mean mounting the cephfs folder in each of the destination nodes, and maybe add the folder as storage right ?!
Anyway, im using now a vm on the storage cluster, and that has the cephfs mounted inside of it, and that vm exposes different subfolders from that cephfs to the outside (nfs, cifs, etc).
Anyway, im using now a vm on the storage cluster, and that has the cephfs mounted inside of it, and that vm exposes different subfolders from that cephfs to the outside (nfs, cifs, etc).
You can specify the subfolder directly in the storage.cfg. And as @DerDanilo said, you should create a separate ceph user.
https://pve.proxmox.com/pve-docs/chapter-pvesm.html#storage_cephfs
https://pve.proxmox.com/pve-docs/chapter-pvesm.html#storage_cephfs
So i cant manage to use that subdir.
So i add the external cephfs, and then edit storage.cfg, but the storage stays monted on / of the cephfs.
Tried enable/disable, but its losing the "subdir" keyword.
So the thing is that i add "subdir xxxx" and the cephfs is still mounted, and accesible , but its on / of the cephfs.
So i add the external cephfs, and then edit storage.cfg, but the storage stays monted on / of the cephfs.
Tried enable/disable, but its losing the "subdir" keyword.
So the thing is that i add "subdir xxxx" and the cephfs is still mounted, and accesible , but its on / of the cephfs.
You need to create a subdir in cephfs, create a separate Ceph client key and restrict it with namespaces to only that subfolder. Then you need to put this Ceph key on all nodes (the pve config share can be of help here), configure the storage.cfg and mount the cephfs on all nodes. I am not aware of pve currently supports mounting an external cephfs but I am positiv that if it doesn't it will be implemented. Anyways you can just mount cephfs locally (using kernel client/fstab or fuse) and tell pve about that new mount locally.
Don't get me wrong but either you need to learn a bit more about Ceph or you can e.g. checkout croit.io which provides a good gui implementation when it comes to handling Ceph settings and configuration of services, permission etc via UI. But even this requires some time to learn. One should understand Ceph at least to some level before throwing production data on it, don't you think?
Well, proxmox supports mounting external cephfs. I just did that. The only problem is that is not mounting a subdir after editing the storage.cfg file.
I know i could go the way you said, that is mounting cephfs manually in a directory on all nodes, and use that as storage.
So if nothing works, that will probably be the way to go, as i have done that before.
And you are right, i need to learn more about ceph
.So my problem now is just that it mounts the / of the external cephfs.
This would be the config :
But the mount doesnt chage.
I can go the the mounted dir in /mnt/pve/external-cephfs and its the / of the cephfs.
If i run pvecm status i get this :
Well thats interesting.
So i changed "subdir backups" to "subdir /backups".
I then ran pvecm status and i got : "A filesystem is already mounted on /mnt/pve/external-cephfs".
I then ran "umount /mnt/pve/external-cephfs", and than saw that it was remounted, correctly with the subdir of the cephfs.
So would im guessing that i would need to do this everytime i add or re-add the storage right ?
So i changed "subdir backups" to "subdir /backups".
I then ran pvecm status and i got : "A filesystem is already mounted on /mnt/pve/external-cephfs".
I then ran "umount /mnt/pve/external-cephfs", and than saw that it was remounted, correctly with the subdir of the cephfs.
So would im guessing that i would need to do this everytime i add or re-add the storage right ?
I think that this only happened becauase it was previously mounted with the root folder before. If monted correctly you shouldn't receive such error.
Cool that you could solve this. Would be interessting though if subfolders would be possible to configure via ui, with the ui correctly configuring the required slashed etc.
Yeah, im not sure. I mean i tried to umount it manually.
Anyway, glad it works now.
And i really think this will surely make its way in the gui.
Subfolder mounts are not often needed, that's why they aren't on the GUI. The subdir needs a slash in front to be an absolute path, eg.
/backups.
I don't know your use cases but with the project we work with they most certainly always need to be able to use cephfs with more than just one level of access. Since it seems to be integrated to some level it seems okay. Would be nice if the UI also supports this. Thanks
So this feature appears to be functional, (or mostly so) in 7.x - you can create a secondary cephfs, it creates the data/meta pools, finds open mds servers, and starts.. only it's not mounted any where?
I would have assume it was created/mounted under /mnt/pve - but no dice. I'm guessing doing something with the default cephfs mount would be the next step (as mostly indicated here : https://forum.proxmox.com/threads/i-cant-mount-proxmox-cephfs.113765/ )
What's the right way to get the secondary cephfs mounted/survivable/etc?
I would have assume it was created/mounted under /mnt/pve - but no dice. I'm guessing doing something with the default cephfs mount would be the next step (as mostly indicated here : https://forum.proxmox.com/threads/i-cant-mount-proxmox-cephfs.113765/ )
What's the right way to get the secondary cephfs mounted/survivable/etc?