RBD: show available quota instead of total storage

[delko]

Trying RBD from external ceph cluster, attached the storage and seems all OK but one detail.

Why do I see the total storage amount instead of the pool quota? I created a dedicated pool to proxmox with 5 TB quota but in the GUI we see the total ceph storage (60TB) instead of 5TB.

That's quite dangerous, since an operator will never notice he's approaching to the limit (if he sees 60TB he can presume that's the limiting point). Why not showing the quota that ceph exposes?

 

[bbgeek17]

Hey @delko , welcome to the forum.

I am not an expert on Ceph, but perhaps the output of these commands may shed some light:

• ceph df detail shows a QUOTA BYTES column per pool.
• ceph osd pool get-quota <pool> shows the quota that’s set.
• ceph df

If ceph df itself shows 60 TB, PVE is just passing it along.

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[delko]

Hey @delko , welcome to the forum.

If ceph df itself shows 60 TB, PVE is just passing it along.

Hi, I politely disagree.
If pve will fail writing once reached 5TB, I would expect to see 5TB as total size. It doesn't matter to me if the underlying storage system (ceph) has more space.
If I use a centralized ceph cluster for all my SAN activities (that's what a SAN is made for), and I assign quotas to the clients, I would expect those clients to be reasonably smart to understand how the storage works and show me useful info (and the total "non usable" space is not very useful).

Since ceph exposes both df AND quota (and "ceph osd pool get-quota Proxmox-testing" show the exact 5TB amount) I would expect to see that value in the webgui, in the cli, etc.

I'm just wondering why that's not happening, if it is a deliberated choice by the team or just an oversight

 

[bbgeek17]

Hi @delko,
The information is available in the source: https://github.com/proxmox/pve-storage/blob/master/src/PVE/Storage/RBDPlugin.pm

my $df = $rados->mon_command({ prefix => 'df', format => 'json' });

my $pool = $scfg->{'data-pool'} // $scfg->{pool} // 'rbd';

my ($d) = grep { $_->{name} eq $pool } @{ $df->{pools} };

my $free = $d->{stats}->{max_avail};
my $used = $d->{stats}->{stored};
my $total = $used + $free;

return ($total, $free, $used, $active);

PVE's use of Ceph is generally intended for built-in Ceph implementation.

You can see in the code:
The PVE plugin calls df without requesting detail, then calculates its reported total from stored + max_avail. It doesn't apply quota_max_bytes itself. Consequently, if Ceph's max_avail is not constrained by the pool quota, PVE can report a capacity close to the cluster's available capacity instead of the configured 5 TB limit.

Could this be improved? Certainly. Build a use-case that is generally applicable, open a report. If you have support subscription - utilize that to open a support case.

There is nothing in the code to indicate whether this was an oversight, deliberate, or just didnt evolve with time.

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[waltar]

Maybe you define instead a "user" quota of 5TB which in fact does not reflect the volume size as expected ?