5.3 and cephfs on external ceph cluster

Discussion in 'Proxmox VE: Installation and configuration' started by ufm, Dec 6, 2018.

  1. ufm

    ufm New Member

    Joined:
    Oct 13, 2010
    Messages:
    13
    Likes Received:
    0
    Hi!

    Does anyone have a recipe for how to use cephfs on an external cluster?

    On the wiki (https://pve.proxmox.com/wiki/Storage) RBD and CephFS refer to the same page (RBD)

    WBR,
    Fyodor.
     
  2. Alwin

    Alwin Proxmox Staff Member
    Staff Member

    Joined:
    Aug 1, 2017
    Messages:
    1,752
    Likes Received:
    151
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  3. ufm

    ufm New Member

    Joined:
    Oct 13, 2010
    Messages:
    13
    Likes Received:
    0
    This documentation refers to parameters "path" and "subdir". But I not see this parameters in GUI. It's possible to configure cephfs via web or only via cli?
     
  4. Alwin

    Alwin Proxmox Staff Member
    Staff Member

    Joined:
    Aug 1, 2017
    Messages:
    1,752
    Likes Received:
    151
    Both ways. For a external ceph cluster, you only need to add a storage and to be able to edit the monitor information, you need to uncheck the 'Use Proxmox VE managed hyper-converged cephFS'.
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  5. ufm

    ufm New Member

    Joined:
    Oct 13, 2010
    Messages:
    13
    Likes Received:
    0
    but in GUI there are no these fields:
    cephfs.png
     
  6. Alwin

    Alwin Proxmox Staff Member
    Staff Member

    Joined:
    Aug 1, 2017
    Messages:
    1,752
    Likes Received:
    151
    Which fields are you referring too? The 'ID' identifies the storage (name), the 'Monitor(s)' of the external cluster need to be added and 'User name' of the user that has access on the foreign cluster. The MDS and which cephfs pool is communicated by the monitors of the cluster.
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  7. ufm

    ufm New Member

    Joined:
    Oct 13, 2010
    Messages:
    13
    Likes Received:
    0
    As I wrote above: Documentation refers to parameters "path" and "subdir". But I not see this parameters in GUI.
     
  8. Alwin

    Alwin Proxmox Staff Member
    Staff Member

    Joined:
    Aug 1, 2017
    Messages:
    1,752
    Likes Received:
    151
    You only need these if you want the mount to be on a different path or if you would like to mount a subdirectory of cephfs. As these are special options, they can only be set in the storage.cfg itself.
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  9. ufm

    ufm New Member

    Joined:
    Oct 13, 2010
    Messages:
    13
    Likes Received:
    0
    Yes, I need this. Thanks for the answer!
     
  10. ufm

    ufm New Member

    Joined:
    Oct 13, 2010
    Messages:
    13
    Likes Received:
    0
    When I attempt to connect to external ceph cluster I got error:
    libceph: bad option at 'secretfile=/etc/pve/priv/ceph/ceph-fs.secret'
    ceph-fs.secret contain only key.
     
  11. Alwin

    Alwin Proxmox Staff Member
    Staff Member

    Joined:
    Aug 1, 2017
    Messages:
    1,752
    Likes Received:
    151
    It needs to contain only the ceph key from the keyfile.
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  12. ufm

    ufm New Member

    Joined:
    Oct 13, 2010
    Messages:
    13
    Likes Received:
    0
    Hmm. I know - my English is very, very bad. But not so much?
    This file "contain only the ceph key from the keyfile".
     
  13. Alwin

    Alwin Proxmox Staff Member
    Staff Member

    Joined:
    Aug 1, 2017
    Messages:
    1,752
    Likes Received:
    151
    Please check that it is one line and that it has no new line.
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  14. ufm

    ufm New Member

    Joined:
    Oct 13, 2010
    Messages:
    13
    Likes Received:
    0
    Yes. One line and no new line.
     
  15. Alwin

    Alwin Proxmox Staff Member
    Staff Member

    Joined:
    Aug 1, 2017
    Messages:
    1,752
    Likes Received:
    151
    Hm... just checked, I think you need to update the cephfs client to luminous that it works. Easiest with 'pveceph install', then all ceph packages are installed with luminous.
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  16. ufm

    ufm New Member

    Joined:
    Oct 13, 2010
    Messages:
    13
    Likes Received:
    0
    Hmm.

    Very interesting.

    Previously I used version 5.2
    I upgrade it to 5.3 by apt update/apt upgrade/apt dist-upgrade
    After upgrade 'apt' says - all packages is fresh and there is nothing to update.
    I execute 'pveceph install' and stop it (i.e. I do not install anything).
    I execute 'apt update/apt upgrade' again. And see:
    Аfter updating everything started working.
    I think it's a bug in updating/upgrading process and should be fixed.

    But anyway - thanx again.
     
  17. Alwin

    Alwin Proxmox Staff Member
    Staff Member

    Joined:
    Aug 1, 2017
    Messages:
    1,752
    Likes Received:
    151
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  18. ufm

    ufm New Member

    Joined:
    Oct 13, 2010
    Messages:
    13
    Likes Received:
    0
    root@pve01:~# apt show ceph-common
    Package: ceph-common
    Version: 12.2.8-pve1
     
  19. sb-jw

    sb-jw New Member

    Joined:
    Jan 23, 2018
    Messages:
    2
    Likes Received:
    0
    Thanks for this Thread.
    I was able to solve our Problem with this Thread, i had to replace the original Repo from CEPH with the Proxmox Repo, after "apt upgrade" i was able to mount cephFS from our external Server.

    A little Background Info about our current Setup:
    Currently we using an external CEPH Storage on Debian which already running on Mimic (deployed with Croit), so i have changed the repo to original from CEPH to get the Mimic Updates. Our external Backup CEPH is running with Proxmox itself, currently they is exporting the RBD through an Gateway NFS VM, but i want to use cephFS to storing the Backups on them. With the original Update from ceph we wasnt able to mount it. The Error Message: "create storage failed: error with cfs lock 'file-storage_cfg': mount error: dmesg | tail or so. (500)".

    But now its solved.
     
  1. This site uses cookies to help personalise content, tailor your experience and to keep you logged in if you register.
    By continuing to use this site, you are consenting to our use of cookies.
    Dismiss Notice