proxmxo managed cephfs mount on external computer (non porxmox) cant access file content

ilia987

Active Member
Sep 9, 2019
273
13
38
36
i have usefully integrated ceph(proxmox based) in all the lxc containers,
now i want to integrate it outside of proxmox for some user for read only access , to replace the current nfs share,
what do i need to do ? what params to put in /etc/fstab
 
i followed the gide above,

and when i mount i get
Code:
mount error 13 = Permission denied

i executed the command:
Code:
mount -t ceph ceph-mon1.storage:6789:/ /mnt/mycephfs -o name=cephx,secretfile=/etc/ceph/ceph.client.cephx.keyring
working only when setting only / in path when creating authorization


i could not mount sub dir, only root



works:
Code:
mount -t ceph ceph-mon1.storage:6789,ceph-mon2.storage:6789,ceph-mon3.storage:6789:/ /mnt/mycephfs -o name=cephx,secretfile=/etc/ceph/ceph.client.cephx.keyring

does not work fstab
Code:
ceph ceph-mon1.storage:6789,ceph-mon2.storage:6789,ceph-mon3.storage:6789:/ /mnt/ftd ceph name=cephx,secretfile=/etc/ceph/ceph.client.cephx.keyring,noatime 0 0
 
Last edited:
update :have some progress i can mount it and see the files but i dont have permission to read the files :(
Code:
ceph-mon1.storage:6789:/data /mnt/ftd ceph name=cephx,secretfile=/etc/ceph/ceph.client.cephx.keyring,ro,_netdev,noatime    0   0
 
Did you create a new user for accessing the cephfs?
or do you use the default root what is created by Proxmox VE?
 
Did you create a new user for accessing the cephfs?
or do you use the default root what is created by Proxmox VE?
what i did:
Code:
#on client, outside of proxmox (desktop)
sudo su
apt-get install ceph-common

mkdir /etc/ceph

ssh  ceph-mon1.storage
ssh root@ceph-mon1.storage "ceph config generate-minimal-conf" | tee /etc/ceph/ceph.conf
chmod 644 /etc/ceph/ceph.conf

ssh root@ceph-mon1.storage " ceph fs authorize cephfs-data client.cephx / rw" | tee /etc/ceph/ceph.client.cephx.keyring

chmod 600 /etc/ceph/ceph.client.cephx.keyring
#edit ceph.client.cephx.keyring to contain only key



mkdir /mnt/mycephfs
#edit fstab
ceph-mon1.storage:6789,ceph-mon2.storage:6789,ceph-mon3.storage:6789:/data /mnt/ftd ceph name=cephx,secretfile=/etc/ceph/ceph.client.cephx.keyring,ro,_netdev,noatime    0   0

mounts created using proxmox (gui) works inside lxc conainers,

but when i try to create cephfs mount outside the proxmox ( local desktop) i can only see the files , i cannot access theirs contents.
 
Do you use the same ceph version at the server and client?
 
Can you please post the ceph auth list (w/o key ;)) of the user?
 
Can you please post the ceph auth list (w/o key ;)) of the user?
all keys are removed
Code:
mds.pve-srv2                           
    key:                       
    caps:    [mds]    allow    *           
    caps:    [mon]    allow    profile    mds       
    caps:    [osd]    allow    rwx           
mds.pve-srv3                           
    key:                       
    caps:    [mds]    allow    *           
    caps:    [mon]    allow    profile    mds       
    caps:    [osd]    allow    rwx           
mds.pve-srv4                           
    key:                       
    caps:    [mds]    allow    *           
    caps:    [mon]    allow    profile    mds       
    caps:    [osd]    allow    rwx           
osd.0                           
    key:                       
    caps:    [mgr]    allow    profile    osd       
    caps:    [mon]    allow    profile    osd       
    caps:    [osd]    allow    *           
osd.1                           
    key:                       
    caps:    [mgr]    allow    profile    osd       
    caps:    [mon]    allow    profile    osd       
    caps:    [osd]    allow    *           
osd.10                           
    key:                       
    caps:    [mgr]    allow    profile    osd       
    caps:    [mon]    allow    profile    osd       
    caps:    [osd]    allow    *           
osd.11                           
    key:                       
    caps:    [mgr]    allow    profile    osd       
    caps:    [mon]    allow    profile    osd       
    caps:    [osd]    allow    *           
osd.2                           
    key:                       
    caps:    [mgr]    allow    profile    osd       
    caps:    [mon]    allow    profile    osd       
    caps:    [osd]    allow    *           
osd.3                           
    key:                       
    caps:    [mgr]    allow    profile    osd       
    caps:    [mon]    allow    profile    osd       
    caps:    [osd]    allow    *           
osd.4                           
    key:                       
    caps:    [mgr]    allow    profile    osd       
    caps:    [mon]    allow    profile    osd       
    caps:    [osd]    allow    *           
osd.5                           
    key:                       
    caps:    [mgr]    allow    profile    osd       
    caps:    [mon]    allow    profile    osd       
    caps:    [osd]    allow    *           
osd.6                           
    key:                       
    caps:    [mgr]    allow    profile    osd       
    caps:    [mon]    allow    profile    osd       
    caps:    [osd]    allow    *           
osd.7                           
    key:                       
    caps:    [mgr]    allow    profile    osd       
    caps:    [mon]    allow    profile    osd       
    caps:    [osd]    allow    *           
osd.8                           
    key:                       
    caps:    [mgr]    allow    profile    osd       
    caps:    [mon]    allow    profile    osd       
    caps:    [osd]    allow    *           
osd.9                           
    key:                       
    caps:    [mgr]    allow    profile    osd       
    caps:    [mon]    allow    profile    osd       
    caps:    [osd]    allow    *           
client.admin                           
    key:                       
    caps:    [mds]    allow    *           
    caps:    [mgr]    allow    *           
    caps:    [mon]    allow    *           
    caps:    [osd]    allow    *           
client.bootstrap-mds                           
    key:                       
    caps:    [mon]    allow    profile    bootstrap-mds       
client.bootstrap-mgr                           
    key:                       
    caps:    [mon]    allow    profile    bootstrap-mgr       
client.bootstrap-osd                           
    key:                       
    caps:    [mon]    allow    profile    bootstrap-osd       
client.bootstrap-rbd                           
    key:                       
    caps:    [mon]    allow    profile    bootstrap-rbd       
client.bootstrap-rbd-mirror                           
    key:                       
    caps:    [mon]    allow    profile    bootstrap-rbd-mirror       
client.bootstrap-rgw                           
    key:                       
    caps:    [mon]    allow    profile    bootstrap-rgw       
client.cephx                           
    key:                       
    caps:    [mds]    allow    rw           
    caps:    [mon]    allow    r           
    caps:    [osd]    allow    rw    tag    cephfs    data=cephfs-data
mgr.pve-srv2                           
    key:                       
    caps:    [mds]    allow    *           
    caps:    [mon]    allow    profile    mgr       
    caps:    [osd]    allow    *           
mgr.pve-srv3                           
    key:                       
    caps:    [mds]    allow    *           
    caps:    [mon]    allow    profile    mgr       
    caps:    [osd]    allow    *           
mgr.pve-srv4                           
    key:                       
    caps:    [mds]    allow    *           
    caps:    [mon]    allow    profile    mgr       
    caps:    [osd]    allow    *
 
issue still exist, still looking for help\guidance

i am creating the mount under root user (sudo mount -a)
and trying to access it as another user (local\DC user) not as cephx (this user not really exist, it is exist only to create the share)

i can see the file folder and files are mounted using

mount command inside fstab
Code:
ceph-mon1.storage:6789,ceph-mon2.storage:6789,ceph-mon3.storage:6789:/data /mnt/ftd ceph name=cephx,secretfile=/etc/ceph/ceph.client.cephx.keyring,ro,_netdev,noatime    0   0

ls
Code:
-rwxr-xr-x 1 user2_ldap domain^users 219873796 Jan 1 18:20 /mnt/ftd/data/file.bin
 
This has more to do with the permissions set on the directories and files, then the ceph auth.
 
This has more to do with the permissions set on the directories and files, then the ceph auth.
i can access from lcx inside proxmox with user user2_ldap,
but from desktop (outside of proxmox) with mount using fstab with the same user(user2_ldap) i can go in and out to internal directories but i cannot access its files contents
 
logs under strace:
Code:
openat(AT_FDCWD, "/mnt/ftd/data/file.bin", O_RDONLY) = 3
read(3, 0x7ffd679877b0, 8192)           = -1 EPERM (Operation not permitted)
 
i tried mounting under admin (with admin key) and it works,
but i dont think this is smart. still looking for better solution
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!