[SOLVED] existing cephfs has unknown status, after package update, can't mount or create new cephfs storage

[gowger]

Hi there,

After the last package update, on a cluster with version 6.1-7 vms failed to come back up due to missing cephfs dependencies. Ceph is reports no problems and health status is green.

However I get:

mount error: See "systemctl status mnt-pve-cephfs.mount" and "journalctl -xe" for details. (500)

trying to re-enable the only cephfs storage.

And when I try to create a new one, I get:

create storage failed: error with cfs lock 'file-storage_cfg': mount error: See "systemctl status mnt-pve-test.mount" and "journalctl -xe" for details. (500)

I have not upgraded to Nautilus yet. I'm still on Luminous from the 5.x install.

I am not keen on trying to upgrade until I have resolved this issue.

Any ideas on how to get things back to normal?

The only other recent change apart from the upgrade was plugging in an external USB drive with ZFS to use for backups. I have now removed this, eliminating it from any config just in case.

 

[Alwin]

What does the status show? And is the mountpoint empty?

systemctl status mnt-pve-cephfs.mount
ceph -s

 

[gowger]

It appears to be caused by a newly introduced mount param.

root@pve4:~# systemctl status mnt-pve-cephfs.mount                                                                                                                                                                                                                                                                           
● mnt-pve-cephfs.mount - /mnt/pve/cephfs                                                                                                                                                                                                                                                                                     
   Loaded: loaded (/run/systemd/system/mnt-pve-cephfs.mount; static; vendor preset: enabled)                                                                                                                                                                                                                                 
   Active: failed (Result: exit-code) since Sat 2020-02-22 00:04:17 GMT; 5 days ago                                                                                                                                                                                                                                         
    Where: /mnt/pve/cephfs                                                                                                                                                                                                                                                                                                   
     What: 192.168.1.103:6789,192.168.1.125:6789,192.168.1.141:6789,192.168.1.149:6789:/                                                                                                                                                                                                                                     
                                                                                                                                                                                                                                                                                                                            
Feb 22 00:04:17 pve4 systemd[1]: Mounting /mnt/pve/cephfs...                                                                                                                                                                                                                                                                 
Feb 22 00:04:17 pve4 mount[4669]: mount error 22 = Invalid argument                                                                                                                                                                                                                                                         
Feb 22 00:04:17 pve4 systemd[1]: mnt-pve-cephfs.mount: Mount process exited, code=exited, status=22/n/a                                                                                                                                                                                                                     
Feb 22 00:04:17 pve4 systemd[1]: mnt-pve-cephfs.mount: Failed with result 'exit-code'.                                                                                                                                                                                                                                       
Feb 22 00:04:17 pve4 systemd[1]: Failed to mount /mnt/pve/cephfs.

root@pve4:~# ceph -s
  cluster:
    id:     77a03fa2-b35a-4304-98fb-c0f91d5e1fc5
    health: HEALTH_OK
 
  services:
    mon: 4 daemons, quorum pve3,pve4,pve,pve2
    mgr: pve(active), standbys: pve2, pve4, pve3
    mds: cephfs-1/1/1 up  {0=pve2=up:active}, 1 up:standby
    osd: 20 osds: 20 up, 20 in
 
  data:
    pools:   5 pools, 1192 pgs
    objects: 114.29k objects, 440GiB
    usage:   1.25TiB used, 1.75TiB / 3.00TiB avail
    pgs:     1192 active+clean
 
  io:
    client:   0B/s rd, 37.8KiB/s wr, 4op/s rd, 4op/s wr

 

[Alwin]

Check the journal, it may tell you what option it is. And is the mount directory empty?

 

[gowger]

The /mnt/pve/cephfs directory exists, and new subdirs of /mnt/pve/ are created if I try to create new cephfs filesystems in the gui.

journal shows these errors:

mount error 22 = Invalid argument

 kernel: libceph: bad option at 'conf=/etc/pve/ceph.conf'

I noticed that two version of libceph are installed. Could this be a problem?

dpkg -l|grep ceph
ii  ceph                                 12.2.13-pve1                                    amd64        distributed storage and file system
ii  ceph-base                            12.2.13-pve1                                    amd64        common ceph daemon libraries and management tools
ii  ceph-common                          12.2.13-pve1                                    amd64        common utilities to mount and interact with a ceph storage cluster
ii  ceph-fuse                            12.2.13-pve1                                    amd64        FUSE-based client for the Ceph distributed file system
ii  ceph-mds                             12.2.13-pve1                                    amd64        metadata server for the ceph distributed file system
ii  ceph-mgr                             12.2.13-pve1                                    amd64        manager for the ceph distributed storage system
ii  ceph-mon                             12.2.13-pve1                                    amd64        monitor server for the ceph storage system
ii  ceph-osd                             12.2.13-pve1                                    amd64        OSD server for the ceph storage system
ii  libcephfs1                           10.2.11-2                                       amd64        Ceph distributed file system client library
ii  libcephfs2                           12.2.13-pve1                                    amd64        Ceph distributed file system client library
ii  python-cephfs                        12.2.13-pve1                                    amd64        Python 2 libraries for the Ceph libcephfs library

the cephfs mount is currently disabled as it breaks the system, so my storage.cfg is as follows:

dir: local
        path /var/lib/vz
        content iso,backup,vztmpl

lvmthin: local-lvm
        thinpool data
        vgname pve
        content rootdir,images

cephfs: cephfs
        disable
        path /mnt/pve/cephfs
        content iso,backup
        maxfiles 1

rbd: ceph-rbd
        content rootdir,images
        krbd 0
        pool ceph-rbd

 

[tom]

I noticed that two version of libceph are installed. Could this be a problem?

The current version of Ceph for Proxmox VE 6.1 is 14.2.6 - seems you missed some updates.

 

[gowger]

I can also add that the mount file contains the following:

Options=name=admin,secretfile=/etc/pve/priv/ceph/cephfs.secret,conf=/etc/pve/ceph.conf

 

[gowger]

The current version of Ceph for Proxmox VE 6.1 is 14.2.6 - seems you missed some updates.

Well I have not upgraded the filesystem to nautilus yet, as far as I understand it, that would require manually changing the package repositories for ceph.

I didn't want to do that yet until absolutely necessary.

 

[tom]

package repositories for ceph

Which repo do you use now? I do not recommend to run CephFS with the 12.x release, use current version.

 

[gowger]

I bit the bullet and performed the upgrade. It did indeed solve all problems.

Thank you very much for your help.