mount error: exit code 16 (500) on cephfs mount

[ronweister]

mount error: exit code 16 (500)

getting this error on external cephfs mount with ec datapool

Can anyone help?

 

[wahmed]

If you are still having the issue, simply update your Proxmox node and it will get fixed.

 

[Whatever]

I'm facing the same issue with the latest 5.4

syslog

May  7 14:44:27 pve-node4 pvestatd[2269]: A filesystem is already mounted on /mnt/pve/cephfs-backup
May  7 14:44:27 pve-node4 pvestatd[2269]: mount error: exit code 16
May  7 14:44:37 pve-node4 pvestatd[2269]: A filesystem is already mounted on /mnt/pve/cephfs-backup
May  7 14:44:37 pve-node4 pvestatd[2269]: mount error: exit code 16

root@pve-node4:~# df -h
Filesystem                                                                 Size  Used Avail Use% Mounted on
udev                                                                        63G     0   63G   0% /dev
tmpfs                                                                       13G  266M   13G   3% /run
/dev/mapper/pve-root                                                        15G  3.0G   11G  22% /
tmpfs                                                                       63G   63M   63G   1% /dev/shm
tmpfs                                                                      5.0M     0  5.0M   0% /run/lock
tmpfs                                                                       63G     0   63G   0% /sys/fs/cgroup
/dev/fuse                                                                   30M   44K   30M   1% /etc/pve
/dev/sde1                                                                   94M  5.5M   89M   6% /var/lib/ceph/osd/ceph-28
/dev/sdf1                                                                   94M  5.5M   89M   6% /var/lib/ceph/osd/ceph-29
/dev/sdd1                                                                   94M  5.5M   89M   6% /var/lib/ceph/osd/ceph-27
10.10.10.101:6789,10.10.10.102:6789,10.10.10.103:6789,10.10.10.104:6789:/  7.2T  4.2T  3.0T  59% /mnt/pve/cephfs-backup
/dev/sda1                                                                   94M  5.5M   89M   6% /var/lib/ceph/osd/ceph-30
/dev/sdb1                                                                   94M  5.5M   89M   6% /var/lib/ceph/osd/ceph-31
/dev/sdc1                                                                   94M  5.5M   89M   6% /var/lib/ceph/osd/ceph-32
tmpfs                                                                       13G     0   13G   0% /run/user/0

If I "cd" to "/mnt/pve/cephfs-backup" I can see all the data (so this resource is already mounted but not recognized by pvestat

Any ideas?

 

[Alwin]

How does your storage.cfg look like? And did you try to unmount the storage to see if it gets mounted by itself?

 

[Whatever]

root@pve-node2:~# cat /etc/pve/storage.cfg 
dir: local
        disable
        path /var/lib/vz
        content images
        maxfiles 0
        shared 0

zfspool: local-zfs
        disable
        pool rpool/data
        blocksize 8k
        content images
        nodes pve-node2,pve-node3,pve-node1
        sparse 0

zfspool: data-zfs
        disable
        pool tank/data
        blocksize 8k
        content images,rootdir
        nodes pve-node2,pve-node3,pve-node1
        sparse 0

rbd: rbd
        content images
        krbd 0
        pool rbd

cephfs: cephfs-backup
        path /mnt/pve/cephfs-backup
        content backup,vztmpl,iso
        maxfiles 3

 

[Whatever]

How does your storage.cfg look like? And did you try to unmount the storage to see if it gets mounted by itself?

unmounting "/mnt/pve/cephfs-backup" works (temporary) for all nodes but one.

root@pve-node3:~# umount /mnt/pve/cephfs-backup
umount: /mnt/pve/cephfs-backup: target is busy
        (In some cases useful info about processes that
         use the device is found by lsof(8) or fuser(1).)

However, in several hours an error comes back on all other nodes

 

[Alwin]

On node3 something is still accessing the mount, could you find out what it is? And can you please post a 'pveversion -v'?