Ceph mount a PG/pool for "Images & ISOs"

[liszca]

wanted to mount a ceph PG for Image and ISOs. Its just to have all images and ISO the same on every node.
I named the PG "proxmox"

To check and put the mount somewhere I edited file: "/etc/pve/storage.cfg"

cephfs: proxmox
        path /mnt/pve/proxmox
        content iso,images
        fs-name proxmox
        monhost 192.168.0.10  192.168.0.11  192.168.0.12  192.168.0.13

The result is it doesn't come to work on all the nodes

 

[aaron]

Did you create a MDS (metadata server), better at least two, and a Ceph FS?

Please provide the output of the following commands within [CODE][/CODE] for better readability.

ceph -s
pveceph pool ls --noborder

Also, to get the terminology right: The hierarchy is, from the top down: Pool -> Placement group (PG) -> Objects. The PGs exist to make the accounting for all the objects easier. So that it can be done on the PG level, and not for each individual object.

There should be two pools for a Ceph FS present, if created through the Proxmox VE tooling (GUI or pveceph fs create. One {fsname}_metadata and one {fsname}_data.

 

[liszca]

Did you create a MDS (metadata server), better at least two, and a Ceph FS?

Yes I did, but I am not sure if I did right:

Note "(ceph1)" happend while trying to do something different: " ceph fs volume create test1" Later I renamed it. I is that needed? before both where in standby.

ceph -s

  cluster:
    id:     ddfe12d5-782f-4028-b499-71f3e6763d8a
    health: HEALTH_OK

  services:
    mon: 4 daemons, quorum aegaeon,anthe,atlas,calypso (age 2h)
    mgr: calypso(active, since 3h), standbys: aegaeon
    mds: 1/1 daemons up, 1 standby
    osd: 4 osds: 4 up (since 2h), 4 in (since 2h)

  data:
    volumes: 1/1 healthy
    pools:   5 pools, 113 pgs
    objects: 39.99k objects, 152 GiB
    usage:   455 GiB used, 3.3 TiB / 3.7 TiB avail
    pgs:     113 active+clean

  io:
    client:   38 KiB/s wr, 0 op/s rd, 7 op/s wr

pveceph pool ls --noborder

Name             Size Min Size PG Num min. PG Num Optimal PG Num PG Autoscale Mode PG Autoscale Target Size PG Autoscale
.mgr                3        2      1           1              1 on
ceph                3        2     32                         32 on
cephfs.ceph.data    3        2     32                         32 on
cephfs.ceph.meta    3        2     16          16             16 on
proxmox             3        2     32                         32 on

There should be two pools for a Ceph FS present, if created through the Proxmox VE tooling (GUI or pveceph fs create. One {fsname}_metadata and one {fsname}_data.

I am confused by the volume "ceph1" created earlier. Whats the thing with a volume?

 # ceph fs volume ls
[
    {
        "name": "ceph1"
    }
]

 

[aaron]

Looks like some things got a bit messy playing around. I recommend that you remove the Ceph file system "ceph1" and its associtated pools. Then also the pool "proxmox" and the storage config for it and setup the Ceph FS through either the Proxmox VE GUI or with the pveceph fs create command.

This way, you will also get the correct storage config on Proxmox VE.

 

[liszca]

Does it make sense to have multiple MDS on each node?


how about the managers does it make sense to have multiple in standby

 

[liszca]

My Conclusion is it makes sense to have multiple MDS and managers on standby incase one dies because that node is down.

 

[aaron]

Having at least one more MGR or MDS on a another node is a good idea, as you said, so they can take over if the node where the current active one runs on, fails.

The MONs work similar as Proxmox VE nodes themselves, by forming a quorum (majority). While MGR and MDS work in active/stand-by modes.

If you have multiple CephFS' you might even want to run multiple MDS per node.

 

[liszca]

I couldn't figure out what to prepare to make this Guide work and have ceph mounted into a local folder on each node:
https://pve.proxmox.com/wiki/Storage:_CephFS

 

[aaron]

If you run CephFS on the local, hyperconverged, Ceph cluster, you don't need to do it manually like this. Remove all remains of former tries and then use the GUI to create a new CephFS and make sure that the checkbox "Add Storage" is enabled. It will handle it all for you and you will see the created CephFS mounted under /mnt/pve/{cephfs name}

 

[liszca]

If you run CephFS on the local, hyperconverged, Ceph cluster, you don't need to do it manually like this. Remove all remains of former tries and then use the GUI to create a new CephFS and make sure that the checkbox "Add Storage" is enabled

I managed to get it to work, but not with the name "proxmox" it was complaining its already used. But I was unable to figure out where cleanup is needed

By the way many thanks!