Ceph OSD Full

dr.aka

New Member
May 20, 2023
2
0
1
Hello,
I am a beginner with Proxmox.

My ceph storage is full, I have 3 nodes with one OSD each, and the smallest (80GB) is full (more than 90%), each OSD is created from lvm volume and there are hdds, I know that as long as I have a single OSD that is much fuller than the others, it will be the limiting factor for how much space is estimated as free. And I know that Ceph is more efficient with larger disks which are almost the same size, but I wanted to give it a try...

I have followed the thread Ceph storage fully although storage is still available,
I have increased the number of PGs but nothing changed for me, or I misunderstood some Ceph concepts.

Here is the output, in attachments, of ceph osd df tree, pveceph pool ls --noborder, ceph -s, ceph osd pool autoscale-status and ceph balancer status

What have I done wrong ?

Here is the output of pveversion -v

Code:
proxmox-ve: 7.4-1 (running kernel: 5.15.107-2-pve)
pve-manager: 7.4-3 (running version: 7.4-3/9002ab8a)
pve-kernel-5.15: 7.4-3
pve-kernel-5.15.107-2-pve: 5.15.107-2
pve-kernel-5.15.102-1-pve: 5.15.102-1
pve-kernel-5.15.30-2-pve: 5.15.30-3
ceph: 17.2.5-pve1
ceph-fuse: 17.2.5-pve1
corosync: 3.1.7-pve1
criu: 3.15-1+pve-1
glusterfs-client: 9.2-1
ifupdown2: 3.1.0-1+pmx3
ksm-control-daemon: 1.4-1
libjs-extjs: 7.0.0-1
libknet1: 1.24-pve2
libproxmox-acme-perl: 1.4.4
libproxmox-backup-qemu0: 1.3.1-1
libproxmox-rs-perl: 0.2.1
libpve-access-control: 7.4-2
libpve-apiclient-perl: 3.2-1
libpve-common-perl: 7.4-1
libpve-guest-common-perl: 4.2-4
libpve-http-server-perl: 4.2-3
libpve-rs-perl: 0.7.6
libpve-storage-perl: 7.4-2
libspice-server1: 0.14.3-2.1
lvm2: 2.03.11-2.1
lxc-pve: 5.0.2-2
lxcfs: 5.0.3-pve1
novnc-pve: 1.4.0-1
proxmox-backup-client: 2.4.1-1
proxmox-backup-file-restore: 2.4.1-1
proxmox-kernel-helper: 7.4-1
proxmox-mail-forward: 0.1.1-1
proxmox-mini-journalreader: 1.3-1
proxmox-offline-mirror-helper: 0.5.1-1
proxmox-widget-toolkit: 3.6.5
pve-cluster: 7.3-3
pve-container: 4.4-3
pve-docs: 7.4-2
pve-edk2-firmware: 3.20230228-2
pve-firewall: 4.3-2
pve-firmware: 3.6-5
pve-ha-manager: 3.6.1
pve-i18n: 2.12-1
pve-qemu-kvm: 7.2.0-8
pve-xtermjs: 4.16.0-1
qemu-server: 7.4-3
smartmontools: 7.2-pve3
spiceterm: 3.2-2
swtpm: 0.8.0~bpo11+3
vncterm: 1.7-1
zfsutils-linux: 2.1.11-pve1
 

Attachments

  • ceph_balancer_status.png
    ceph_balancer_status.png
    58.5 KB · Views: 6
  • ceph_osd_df_tree.png
    ceph_osd_df_tree.png
    152.2 KB · Views: 7
  • ceph_osd_pool_autoscale-status.png
    ceph_osd_pool_autoscale-status.png
    82.4 KB · Views: 5
  • ceph_-s.png
    ceph_-s.png
    367.7 KB · Views: 6
I have forgot to add pveceph pool ls --noborder
 

Attachments

  • pveceph_pool_ls_--noborder.png
    pveceph_pool_ls_--noborder.png
    94.4 KB · Views: 10

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!