proxmxo managed cephfs mount on external computer (non porxmox) cant access file content

ilia987

Member
Sep 9, 2019
237
10
23
35
i have usefully integrated ceph(proxmox based) in all the lxc containers,
now i want to integrate it outside of proxmox for some user for read only access , to replace the current nfs share,
what do i need to do ? what params to put in /etc/fstab
 

ilia987

Member
Sep 9, 2019
237
10
23
35
i followed the gide above,

and when i mount i get
Code:
mount error 13 = Permission denied

i executed the command:
Code:
mount -t ceph ceph-mon1.storage:6789:/ /mnt/mycephfs -o name=cephx,secretfile=/etc/ceph/ceph.client.cephx.keyring
working only when setting only / in path when creating authorization


i could not mount sub dir, only root



works:
Code:
mount -t ceph ceph-mon1.storage:6789,ceph-mon2.storage:6789,ceph-mon3.storage:6789:/ /mnt/mycephfs -o name=cephx,secretfile=/etc/ceph/ceph.client.cephx.keyring

does not work fstab
Code:
ceph ceph-mon1.storage:6789,ceph-mon2.storage:6789,ceph-mon3.storage:6789:/ /mnt/ftd ceph name=cephx,secretfile=/etc/ceph/ceph.client.cephx.keyring,noatime 0 0
 
Last edited:

ilia987

Member
Sep 9, 2019
237
10
23
35
update :have some progress i can mount it and see the files but i dont have permission to read the files :(
Code:
ceph-mon1.storage:6789:/data /mnt/ftd ceph name=cephx,secretfile=/etc/ceph/ceph.client.cephx.keyring,ro,_netdev,noatime    0   0
 

wolfgang

Proxmox Retired Staff
Retired Staff
Oct 1, 2014
6,496
496
103
Did you create a new user for accessing the cephfs?
or do you use the default root what is created by Proxmox VE?
 

ilia987

Member
Sep 9, 2019
237
10
23
35
Did you create a new user for accessing the cephfs?
or do you use the default root what is created by Proxmox VE?
what i did:
Code:
#on client, outside of proxmox (desktop)
sudo su
apt-get install ceph-common

mkdir /etc/ceph

ssh  ceph-mon1.storage
ssh root@ceph-mon1.storage "ceph config generate-minimal-conf" | tee /etc/ceph/ceph.conf
chmod 644 /etc/ceph/ceph.conf

ssh root@ceph-mon1.storage " ceph fs authorize cephfs-data client.cephx / rw" | tee /etc/ceph/ceph.client.cephx.keyring

chmod 600 /etc/ceph/ceph.client.cephx.keyring
#edit ceph.client.cephx.keyring to contain only key



mkdir /mnt/mycephfs
#edit fstab
ceph-mon1.storage:6789,ceph-mon2.storage:6789,ceph-mon3.storage:6789:/data /mnt/ftd ceph name=cephx,secretfile=/etc/ceph/ceph.client.cephx.keyring,ro,_netdev,noatime    0   0

mounts created using proxmox (gui) works inside lxc conainers,

but when i try to create cephfs mount outside the proxmox ( local desktop) i can only see the files , i cannot access theirs contents.
 

wolfgang

Proxmox Retired Staff
Retired Staff
Oct 1, 2014
6,496
496
103
Do you use the same ceph version at the server and client?
 

Alwin

Proxmox Retired Staff
Retired Staff
Aug 1, 2017
4,617
453
88
Can you please post the ceph auth list (w/o key ;)) of the user?
 

ilia987

Member
Sep 9, 2019
237
10
23
35
Can you please post the ceph auth list (w/o key ;)) of the user?
all keys are removed
Code:
mds.pve-srv2                           
    key:                       
    caps:    [mds]    allow    *           
    caps:    [mon]    allow    profile    mds       
    caps:    [osd]    allow    rwx           
mds.pve-srv3                           
    key:                       
    caps:    [mds]    allow    *           
    caps:    [mon]    allow    profile    mds       
    caps:    [osd]    allow    rwx           
mds.pve-srv4                           
    key:                       
    caps:    [mds]    allow    *           
    caps:    [mon]    allow    profile    mds       
    caps:    [osd]    allow    rwx           
osd.0                           
    key:                       
    caps:    [mgr]    allow    profile    osd       
    caps:    [mon]    allow    profile    osd       
    caps:    [osd]    allow    *           
osd.1                           
    key:                       
    caps:    [mgr]    allow    profile    osd       
    caps:    [mon]    allow    profile    osd       
    caps:    [osd]    allow    *           
osd.10                           
    key:                       
    caps:    [mgr]    allow    profile    osd       
    caps:    [mon]    allow    profile    osd       
    caps:    [osd]    allow    *           
osd.11                           
    key:                       
    caps:    [mgr]    allow    profile    osd       
    caps:    [mon]    allow    profile    osd       
    caps:    [osd]    allow    *           
osd.2                           
    key:                       
    caps:    [mgr]    allow    profile    osd       
    caps:    [mon]    allow    profile    osd       
    caps:    [osd]    allow    *           
osd.3                           
    key:                       
    caps:    [mgr]    allow    profile    osd       
    caps:    [mon]    allow    profile    osd       
    caps:    [osd]    allow    *           
osd.4                           
    key:                       
    caps:    [mgr]    allow    profile    osd       
    caps:    [mon]    allow    profile    osd       
    caps:    [osd]    allow    *           
osd.5                           
    key:                       
    caps:    [mgr]    allow    profile    osd       
    caps:    [mon]    allow    profile    osd       
    caps:    [osd]    allow    *           
osd.6                           
    key:                       
    caps:    [mgr]    allow    profile    osd       
    caps:    [mon]    allow    profile    osd       
    caps:    [osd]    allow    *           
osd.7                           
    key:                       
    caps:    [mgr]    allow    profile    osd       
    caps:    [mon]    allow    profile    osd       
    caps:    [osd]    allow    *           
osd.8                           
    key:                       
    caps:    [mgr]    allow    profile    osd       
    caps:    [mon]    allow    profile    osd       
    caps:    [osd]    allow    *           
osd.9                           
    key:                       
    caps:    [mgr]    allow    profile    osd       
    caps:    [mon]    allow    profile    osd       
    caps:    [osd]    allow    *           
client.admin                           
    key:                       
    caps:    [mds]    allow    *           
    caps:    [mgr]    allow    *           
    caps:    [mon]    allow    *           
    caps:    [osd]    allow    *           
client.bootstrap-mds                           
    key:                       
    caps:    [mon]    allow    profile    bootstrap-mds       
client.bootstrap-mgr                           
    key:                       
    caps:    [mon]    allow    profile    bootstrap-mgr       
client.bootstrap-osd                           
    key:                       
    caps:    [mon]    allow    profile    bootstrap-osd       
client.bootstrap-rbd                           
    key:                       
    caps:    [mon]    allow    profile    bootstrap-rbd       
client.bootstrap-rbd-mirror                           
    key:                       
    caps:    [mon]    allow    profile    bootstrap-rbd-mirror       
client.bootstrap-rgw                           
    key:                       
    caps:    [mon]    allow    profile    bootstrap-rgw       
client.cephx                           
    key:                       
    caps:    [mds]    allow    rw           
    caps:    [mon]    allow    r           
    caps:    [osd]    allow    rw    tag    cephfs    data=cephfs-data
mgr.pve-srv2                           
    key:                       
    caps:    [mds]    allow    *           
    caps:    [mon]    allow    profile    mgr       
    caps:    [osd]    allow    *           
mgr.pve-srv3                           
    key:                       
    caps:    [mds]    allow    *           
    caps:    [mon]    allow    profile    mgr       
    caps:    [osd]    allow    *           
mgr.pve-srv4                           
    key:                       
    caps:    [mds]    allow    *           
    caps:    [mon]    allow    profile    mgr       
    caps:    [osd]    allow    *
 

Alwin

Proxmox Retired Staff
Retired Staff
Aug 1, 2017
4,617
453
88

ilia987

Member
Sep 9, 2019
237
10
23
35
issue still exist, still looking for help\guidance

i am creating the mount under root user (sudo mount -a)
and trying to access it as another user (local\DC user) not as cephx (this user not really exist, it is exist only to create the share)

i can see the file folder and files are mounted using

mount command inside fstab
Code:
ceph-mon1.storage:6789,ceph-mon2.storage:6789,ceph-mon3.storage:6789:/data /mnt/ftd ceph name=cephx,secretfile=/etc/ceph/ceph.client.cephx.keyring,ro,_netdev,noatime    0   0

ls
Code:
-rwxr-xr-x 1 user2_ldap domain^users 219873796 Jan 1 18:20 /mnt/ftd/data/file.bin
 

Alwin

Proxmox Retired Staff
Retired Staff
Aug 1, 2017
4,617
453
88
This has more to do with the permissions set on the directories and files, then the ceph auth.
 

ilia987

Member
Sep 9, 2019
237
10
23
35
This has more to do with the permissions set on the directories and files, then the ceph auth.
i can access from lcx inside proxmox with user user2_ldap,
but from desktop (outside of proxmox) with mount using fstab with the same user(user2_ldap) i can go in and out to internal directories but i cannot access its files contents
 

ilia987

Member
Sep 9, 2019
237
10
23
35
logs under strace:
Code:
openat(AT_FDCWD, "/mnt/ftd/data/file.bin", O_RDONLY) = 3
read(3, 0x7ffd679877b0, 8192)           = -1 EPERM (Operation not permitted)
 

ilia987

Member
Sep 9, 2019
237
10
23
35
i tried mounting under admin (with admin key) and it works,
but i dont think this is smart. still looking for better solution
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get your own in 60 seconds.

Buy now!