cephFS not mounting till all nodes are up (7.3.6)

[dlasher]

5 node deployment in lab, noticed something odd.

Cephfs fails to mount on any node nodes until *ALL* nodes are up. IE 4 of 5 machines up, cephfs still fails.

Given the pool config of cephfs_data and cephfs_metadata (both 3/2 replicated) I don't understand why this would be the case.

In theory it should mount as soon as 3 nodes are up, right?

pve-manager/7.3-6/723bb6ec (running kernel: 5.15.85-1-pve)
root@pmx1:~# pveversion  -v
proxmox-ve: 7.3-1 (running kernel: 5.15.85-1-pve)
pve-manager: 7.3-6 (running version: 7.3-6/723bb6ec)
pve-kernel-helper: 7.3-4
pve-kernel-5.15: 7.3-2
pve-kernel-5.15.85-1-pve: 5.15.85-1
pve-kernel-5.15.83-1-pve: 5.15.83-1
pve-kernel-5.15.53-1-pve: 5.15.53-1
pve-kernel-5.15.39-3-pve: 5.15.39-3
ceph: 16.2.11-pve1
ceph-fuse: 16.2.11-pve1
corosync: 3.1.7-pve1
criu: 3.15-1+pve-1
glusterfs-client: 9.2-1
ifupdown: residual config
ifupdown2: 3.1.0-1+pmx3
ksm-control-daemon: 1.4-1
libjs-extjs: 7.0.0-1
libknet1: 1.24-pve2
libproxmox-acme-perl: 1.4.3
libproxmox-backup-qemu0: 1.3.1-1
libpve-access-control: 7.3-1
libpve-apiclient-perl: 3.2-1
libpve-common-perl: 7.3-2
libpve-guest-common-perl: 4.2-3
libpve-http-server-perl: 4.1-5
libpve-storage-perl: 7.3-2
libspice-server1: 0.14.3-2.1
lvm2: 2.03.11-2.1
lxc-pve: 5.0.2-1
lxcfs: 5.0.3-pve1
novnc-pve: 1.3.0-3
openvswitch-switch: 2.15.0+ds1-2+deb11u2.1
proxmox-backup-client: 2.3.3-1
proxmox-backup-file-restore: 2.3.3-1
proxmox-mail-forward: 0.1.1-1
proxmox-mini-journalreader: 1.3-1
proxmox-offline-mirror-helper: 0.5.1-1
proxmox-widget-toolkit: 3.5.5
pve-cluster: 7.3-2
pve-container: 4.4-2
pve-docs: 7.3-1
pve-edk2-firmware: 3.20220526-1
pve-firewall: 4.2-7
pve-firmware: 3.6-3
pve-ha-manager: 3.5.1
pve-i18n: 2.8-2
pve-qemu-kvm: 7.1.0-4
pve-xtermjs: 4.16.0-1
qemu-server: 7.3-3
smartmontools: 7.2-pve3
spiceterm: 3.2-2
swtpm: 0.8.0~bpo11+2
vncterm: 1.7-1
zfsutils-linux: 2.1.9-pve1

 

[gurubert]

CephFS can be mounted as soon as the cluster works (MONs have quorum) and at least one MDS is ready to serve requests.
Where do the MONs run and where the MDSs?

 

[dlasher]

CephFS can be mounted as soon as the cluster works (MONs have quorum) and at least one MDS is ready to serve requests.
Where do the MONs run and where the MDSs?

Each node has both a MON and an MDS - so in the above example we are 4/5 MON and 4/5 MDS (with only 2 MDS needed)

... hence the puzzle ...

 

[gurubert]

Can you share the output of "ceph fs status"?

 

[dlasher]

I'll be rebooting the cluster again today (it is a lab after all) but here's the "current" status with all (5) nodes up and everything happy.

~# ceph fs status
cephfs - 7 clients
======
RANK  STATE   MDS      ACTIVITY     DNS    INOS   DIRS   CAPS
 0    active  mx4  Reqs:    0 /s  83.2k  48.0k  8789   2020
      POOL         TYPE     USED  AVAIL
cephfs_metadata  metadata  4002M   597G
  cephfs_data      data    12.7T  42.4T
STANDBY MDS
    mx3
    mx1
    mx2
    mx5
MDS version: ceph version 16.2.11 (578f8e68e41b0a98523d0045ef6db90ce6f2e5ab) pacific (stable)

 

[gurubert]

Your data and metadata pools use replication with size=3?

How do the cluster and fs status look when one node is not up and you are not able to mount the CephFS?

 

[dlasher]

Your data and metadata pools use replication with size=3? ** YES **

How do the cluster and fs status look when one node is not up and you are not able to mount the CephFS?

Will look later today.

Curious, if it's the actual mount statement that's the problem. For example, once it's mounted, it lists all (5) hosts. Could it be any single host missing prevents the mount?

198.18.53.101,198.18.53.102,198.18.53.103,198.18.53.104,198.18.53.105:/  50026520576  4572213248 45454307328  10% /mnt/pve/cephfs

 

[gurubert]

if it's the actual mount statement that's the problem.

Do you only list one monitor in the mount statement? Then that could be the issue here.

 

[dlasher]

Do you only list one monitor in the mount statement? Then that could be the issue here.

The reverse of that. I'm asking if having all (5) monitors listed in the mount statement is causing the problem when (1) is missing.

 

[gurubert]

I'm asking if having all (5) monitors listed in the mount statement is causing the problem when (1) is missing.

As long as any one of the monitors listed is available the mount should finish successfully.