ceph df shows 0 for data pool in a ec pool setup

[rainer042]

Hello,

I have a pve cluster "A" (7.3) which has NO hyperconverged ceph. There is another pve cluster "D" (7.3) which has a lot of ceph storage. So I created one 5+3 ec pool using pveceph pool create pool_d --erasure-coding k=5,m=3 which results in a pool_d-data and a pool_d-metadata. Next I created another ec pool to be used on pve cluster "A" named "pool_a-data" and "pool_a-metadata" with the same 5+3 ec pool parameters also using pveceph.

On cluster "A" I added storage pool "pool_a" using the graphical interface which all worked fine. In between I have several VMs running on "A" with their storage on cluster "D". Works fine so far. However recently I took a look at ceph df on cluster "D" and saw that for pool "pool_a" only the metadada pool shows a usage, not the data pool which actually should most of the used storage which is strange:

#<D>: ceph df
--- POOLS ---
POOL                      ID  PGS   STORED  OBJECTS     USED  %USED  MAX AVAIL
device_health_metrics      1    1  444 MiB       91  1.7 GiB      0     54 TiB
rbd                        3   32  2.3 KiB        3   25 KiB      0     54 TiB
pool_d-data                6  256   95 GiB   24.47k  152 GiB   0.07    136 TiB
pool_d-metadata            7  128  143 KiB       18  530 KiB      0     72 TiB
pool_a-data               12  256      0 B        0      0 B      0    136 TiB
pool_a-metadata           13  128   13 TiB    3.34M   38 TiB  14.84     72 TiB

As you can see for pool_d-data and pool_d-metadata ceph df reports data usage in GB.
However if you look at the data for the pool_a used by cluster "A" only the metadata pool has STORED GB data.

Is this just a display problem or is this erasure pool not used as it should?

Thanks
Rainer

 

[gurubert]

pveceph pool create pool_d --erasure-coding k=5,m=3 which results in a pool_d-data and a pool_d-metadata

How does running this command result in two pools?

 

[alexskysilk]

I reread the OP three times, and I think I'm braindead because I don't understand what you're asking. are you using the ec pool(s) for rbd, cephfs, or rgw? and please provide the content of /etc/pve/storage.cfg?

 

[rainer042]

How does running this command result in two pools?

There are technical details for RBDs/cephfs in EC-pools that are the reason that two pools are needed. One for the data and another for the metadata. In ceph df you usually see that the metadata pool uses some storage but the data pool much more, which is not the case in my setup. The two pools are created automatically by pveceph since I specified to create an EC-Pool.

 

[rainer042]

I reread the OP three times, and I think I'm braindead because I don't understand what you're asking. are you using the ec pool(s) for rbd, cephfs, or rgw? and please provide the content of /etc/pve/storage.cfg?Its rbd

 

[rainer042]

You are right I forgot: Its rbd I use for my VMs.

 

[rainer042]

On the side of cluster "A" the storage.cfg entry for the pool in question looks like this:

rbd: pool_a
        content images
        krbd 0
        monhost a.b.c.d
        pool pool_a-metadata
        username admin

On Cluster "D" I also took a look at storage.cfg for pool_d:

rbd: pool_d
        content images,rootdir
        data-pool pool_d-data
        krbd 1
        pool pool_d-metadata

So it looks like I should simply add a line for the data-pool like

rbd: pool_a
        content images
        krbd 0
        monhost a.b.c.d
        pool pool_a-metadata
        data-pool pool_a-data
        username admin

So I think the best would be to remove pool_a again on both cluster "A" and "D", and then recreate it, this time with the data-pool option added to storage.cfg on cluster "A". Any objections?

 

[alexskysilk]

No objections from me but you might want to be aware that proxmox will not deploy vms to an ec pool, limiting its usefulness to cephfs only.

 

[rainer042]

I use proxmox pve 7.3 on one cluster with a lot of VMs and all of them have their storage on a hyperconverged ceph pool with EC. No problem at all, except for pve webinterface that does forget about the datapool option when a rbd storage on EC is used on a remote pve cluster as I just learned. So I do not understand what you wanted to express in your last comment saying: "proxmox will not deploy vms to an ec pool" ?

 

[alexskysilk]

hey thanks for making me look! looks like it was fixed in pve7.something (https://bugzilla.proxmox.com/show_bug.cgi?id=1816)