Storage CephFS and multiple file systems (passing options)

[Nathan Stratton]

I am using Proxmox 5.4 with CephFS and multiple file systems. One filesystem is called cephfs and it's on NVMe and the other is cephfs_slow and it's on standard SATA. I can manually mount each file system with:

mount -t ceph -o mds_namespace=cephfs virt0,virt4,virt8,virt12:/ /foo
mount -t ceph -o mds_namespace=cephfs_slow virt0,virt4,virt8,virt12:/ /bar

However I want Proxmox to use cephfs_slow for backups and since I can't pass parameters to mount in the GUI I keep mounting cephfs because it was created first rather then cephfs_slow, where I want my backups.

 

[dcsapak]

you can put your mount options in /etc/pve/storage.cfg
e.g.

cephfs: foo
        options 'mds_namespace=foo'
..
..

 

[starkruzr]

you can put your mount options in /etc/pve/storage.cfg
e.g.

cephfs: foo
        options 'mds_namespace=foo'
..
..

Hi,

Should this still work in the latest version of Proxmox? I've tried this and it doesn't seem to have changed the mounts.

 

[Alwin]

Should this still work in the latest version of Proxmox? I've tried this and it doesn't seem to have changed the mounts.

This should still work. It depends also if you use fuse or the kernel client.

 

[starkruzr]

This should still work. It depends also if you use fuse or the kernel client.

Can you expand a little on that? Putting this in my /etc/pve/storage.cfg still results in my first cephfs only being loaded, twice.

root@ibnmajid:~# cat /etc/pve/storage.cfg
dir: local
    path /var/lib/vz
    content vztmpl,backup,iso

lvmthin: local-lvm
    thinpool data
    vgname pve
    content images,rootdir

nfs: aquarii-rtechvol
    export /mnt/RTECHVOL
    path /mnt/pve/aquarii-rtechvol
    server 192.168.9.71
    content rootdir,images,snippets,vztmpl,backup,iso
    maxfiles 1

rbd: fastwrx
    content images,rootdir
    krbd 0
    pool fastwrx

rbd: largewrx
    content images,rootdir
    krbd 0
    pool largewrx

cephfs: fastwrxFS
    path /mnt/pve/fastwrxFS
    content backup,vztmpl,iso,snippets
    maxfiles 1
    options 'mds_namespace=fastwrxFS'

cephfs: largewrxFS
    path /mnt/pve/largewrxFS
    content backup,vztempl,ioo,snippets
    maxfiles 1
    options 'mds_namespace=largewrxFS'

 

[starkruzr]

I've also tried double quotes instead of single quotes; same result.

 

[starkruzr]

If I mount it on the command line, it works:

root@ibnmajid:~# mount -t ceph -v -o name=admin,mds_namespace=largewrxFS 192.168.9.10,192.168.9.11,192.168.9.12:/ /mnt/largewrxFS/
parsing options: rw,name=admin,mds_namespace=largewrxFS
mount.ceph: unrecognized mount option "mds_namespace", passing to kernel.
root@ibnmajid:~# ls /mnt/largewrxFS
root@ibnmajid:~# ls /mnt/pve/fastwrxFS/
benchmark  dump  ncdata  nextclouddata    snippets  template
root@ibnmajid:~# ls /mnt/pve/largewrxFS/
benchmark  dump  ncdata  nextclouddata    snippets  template
root@ibnmajid:~# mount
...
192.168.9.10,192.168.9.11,192.168.9.12:/ on /mnt/pve/fastwrxFS type ceph (rw,relatime,name=admin,secret=<hidden>,acl)
192.168.9.10,192.168.9.11,192.168.9.12:/ on /mnt/pve/largewrxFS type ceph (rw,relatime,name=admin,secret=<hidden>,acl)
192.168.9.10,192.168.9.11,192.168.9.12:/ on /mnt/largewrxFS type ceph (rw,relatime,name=admin,secret=<hidden>,acl,mds_namespace=largewrxFS)

I suppose I could mount it in /etc/fstab and then make it a non-shared "directory" item on each node, but that feels extremely hacky and inviting doom when updates come or other changes hit.

 

[Alwin]

Can you expand a little on that? Putting this in my /etc/pve/storage.cfg still results in my first cephfs only being loaded, twice.

The namespace option seems to be officially introduced in Ceph Octopus. While the kernel seems to know this option. The mount command does not.
https://docs.ceph.com/docs/octopus/man/8/mount.ceph/

options 'mds_namespace=largewrxFS'

Try the options without any quotes. And multiple options are separated by comma only.

And last but not least, upgrade your installation, since also PVE 5.4 will be EoL end of July this year.
https://pve.proxmox.com/wiki/FAQ

 

[starkruzr]

The namespace option seems to be officially introduced in Ceph Octopus. While the kernel seems to know this option. The mount command does not.
https://docs.ceph.com/docs/octopus/man/8/mount.ceph/

The mount command passes it through to the kernel. You can see an "error message" for the unrecognized mds_namespace option there, but you can also see that the result of the mount command at /mnt/fastwrxFS is correct and the correct volume is mounted. It just hasn't mounted it correctly with storage.cfg options yet.

Try the options without any quotes. And multiple options are separated by comma only.

And last but not least, upgrade your installation, since also PVE 5.4 will be EoL end of July this year.
https://pve.proxmox.com/wiki/FAQ

I tried no quotes. I also tried double quotes. I only have one option (the mds_namespace one). This is on PVE 6.1-8, which as far as I know does not support Octopus yet.

 

[Alwin]

The mount command passes it through to the kernel. You can see an "error message" for the unrecognized mds_namespace option there, but you can also see that the result of the mount command at /mnt/fastwrxFS is correct and the correct volume is mounted. It just hasn't mounted it correctly with storage.cfg options yet.

I just mixed things. The mds_namespace seems to be deprecated in octopus. But anyhow, I ran the mount with the namespace on my test cluster. And with an Nautilus client and cluster, the mount worked without issues.

I tried no quotes. I also tried double quotes. I only have one option (the mds_namespace one). This is on PVE 6.1-8, which as far as I know does not support Octopus yet.

It doesn't support it yet. On Proxmox VE 6.1, the mount is done directly through systemd. You should find a mount unit. There the mds_namespace should be visible.

 

[starkruzr]

I just mixed things. The mds_namespace seems to be deprecated in octopus. But anyhow, I ran the mount with the namespace on my test cluster. And with an Nautilus client and cluster, the mount worked without issues.

Wait, if they deprecate the mds_namespace option in Octopus, how do you select which cephfs to mount?

It doesn't support it yet. On Proxmox VE 6.1, the mount is done directly through systemd. You should find a mount unit. There the mds_namespace should be visible.

Where am I looking for this? I'd think it would need to be in /etc/pve so that it's synced between nodes, but that doesn't seem to be the case. I went looking in /etc/systemd/system but that only seems to do general things like starting the mdses, mons, etc.

 

[Alwin]

Wait, if they deprecate the mds_namespace option in Octopus, how do you select which cephfs to mount?

It's then called fs=.
https://docs.ceph.com/docs/octopus/man/8/mount.ceph/

Where am I looking for this? I'd think it would need to be in /etc/pve so that it's synced between nodes, but that doesn't seem to be the case. I went looking in /etc/systemd/system but that only seems to do general things like starting the mdses, mons, etc.

It's a dynamically created unit, run systemctl and search for a unit with the storage name. Then you can use systemctl show <unit>.

 

[starkruzr]

Oh my God. I get it now. It generates the systemd unit files for mounting cephfs filesystems in /run/systemd/system based on what it gets from /etc/pve/storage.cfg, and will dynamically update them when /etc/pve/storage.cfg is changed, but it doesn't reload and restart the mounts when it does this, so you can change it all you want all day in /etc/pve/storage.cfg and nothing will happen for cephfs mounts until you do that manually on each machine. Which I did and now all the cephfs mounts are correct. Thank you!!

Another thing I learned while doing this is that you cannot put the option in single quotes like 'mds_namespace', it has to be "plain" in storage.cfg. (At least as of Ceph Nautilus $LATEST and PVE 6.1-8.) With the quotes, when I reloaded and restarted the mounts, the unit start failed and complained it didn't understand the option with the quotes in it. I got rid of them and that's when it started working -- host by host, so I had to reload and restart for all three hosts.

I want to contribute details about this to the Wiki, if we all have write access.

 

[Alwin]

I want to contribute details about this to the Wiki, if we all have write access.

The systemd unit will be hopefully removed in Proxmox VE 7, since it is just a workaround for the current systemd unit order problematic. It shouldn't be needed at all.

 

[starkruzr]

Ah, that's interesting. How close are we to release 7, and what will replace this current method of generating systemd mounts dynamically?

 

[Alwin]

Ah, that's interesting. How close are we to release 7, and what will replace this current method of generating systemd mounts dynamically?

Not close. Mounts are handled by systemd but the order of things doesn't allow a graceful shutdown. So this is why the separate unit. The handling stays still the same. A change of the storage.cfg will need unmount & mount.