Ceph Journal filling up / directory

musangberjanggut

New Member
Sep 7, 2014
16
0
1
Kajang, Malaysia, Malaysia
Hi Guys,

looking for advise. i'm having proxmox server running with ceph. now i can see that the OSD journals has filled up my / directory. below is the output of df -h

/dev/mapper/pve-root 95G 95G 0 100% /


15G osd/ceph-1/current/2.2e_head/DIR_E
15G osd/ceph-1/current/2.2e_head
7.7G osd/ceph-1/current/2.3e_head/DIR_E/DIR_3
7.5G osd/ceph-1/current/2.3e_head/DIR_E/DIR_B
16G osd/ceph-1/current/2.3e_head/DIR_E
16G osd/ceph-1/current/2.3e_head
7.3G osd/ceph-1/current/2.54_head/DIR_4/DIR_5
7.1G osd/ceph-1/current/2.54_head/DIR_4/DIR_D
15G osd/ceph-1/current/2.54_head/DIR_4
15G osd/ceph-1/current/2.54_head
7.5G osd/ceph-1/current/2.76_head/DIR_6/DIR_7
7.5G osd/ceph-1/current/2.76_head/DIR_6/DIR_F
15G osd/ceph-1/current/2.76_head/DIR_6
15G osd/ceph-1/current/2.76_head
7.4G osd/ceph-1/current/2.2f_head/DIR_F/DIR_2
7.3G osd/ceph-1/current/2.2f_head/DIR_F/DIR_A
15G osd/ceph-1/current/2.2f_head/DIR_F
15G osd/ceph-1/current/2.2f_head
7.5G osd/ceph-1/current/2.3c_head/DIR_C/DIR_3
7.4G osd/ceph-1/current/2.3c_head/DIR_C/DIR_B
15G osd/ceph-1/current/2.3c_head/DIR_C
15G osd/ceph-1/current/2.3c_head
7.2G osd/ceph-1/current/2.2d_head/DIR_D/DIR_2
7.4G osd/ceph-1/current/2.2d_head/DIR_D/DIR_A
15G osd/ceph-1/current/2.2d_head/DIR_D
15G osd/ceph-1/current/2.2d_head
7.7G osd/ceph-1/current/2.74_head/DIR_4/DIR_7
7.5G osd/ceph-1/current/2.74_head/DIR_4/DIR_F
16G osd/ceph-1/current/2.74_head/DIR_4
16G osd/ceph-1/current/2.74_head
7.6G osd/ceph-1/current/2.59_head/DIR_9/DIR_5
7.6G osd/ceph-1/current/2.59_head/DIR_9/DIR_D
16G osd/ceph-1/current/2.59_head/DIR_9
16G osd/ceph-1/current/2.59_head
7.4G osd/ceph-1/current/2.7b_head/DIR_B/DIR_7
7.5G osd/ceph-1/current/2.7b_head/DIR_B/DIR_F
15G osd/ceph-1/current/2.7b_head/DIR_B
15G osd/ceph-1/current/2.7b_head
7.7G osd/ceph-1/current/2.33_head/DIR_3/DIR_3
7.5G osd/ceph-1/current/2.33_head/DIR_3/DIR_B
16G osd/ceph-1/current/2.33_head/DIR_3
16G osd/ceph-1/current/2.33_head
7.3G osd/ceph-1/current/2.42_head/DIR_2/DIR_4
7.2G osd/ceph-1/current/2.42_head/DIR_2/DIR_C
15G osd/ceph-1/current/2.42_head/DIR_2
15G osd/ceph-1/current/2.42_head
7.5G osd/ceph-1/current/2.37_head/DIR_7/DIR_3
7.5G osd/ceph-1/current/2.37_head/DIR_7/DIR_B
15G osd/ceph-1/current/2.37_head/DIR_7
15G osd/ceph-1/current/2.37_head
7.6G osd/ceph-1/current/2.3_head/DIR_3/DIR_0
7.4G osd/ceph-1/current/2.3_head/DIR_3/DIR_8
15G osd/ceph-1/current/2.3_head/DIR_3
15G osd/ceph-1/current/2.3_head
7.4G osd/ceph-1/current/2.61_head/DIR_1/DIR_6
7.3G osd/ceph-1/current/2.61_head/DIR_1/DIR_E
15G osd/ceph-1/current/2.61_head/DIR_1
15G osd/ceph-1/current/2.61_head
7.3G osd/ceph-1/current/2.45_head/DIR_5/DIR_4
7.3G osd/ceph-1/current/2.45_head/DIR_5/DIR_C
15G osd/ceph-1/current/2.45_head/DIR_5
15G osd/ceph-1/current/2.45_head
7.5G osd/ceph-1/current/2.3a_head/DIR_A/DIR_3
7.3G osd/ceph-1/current/2.3a_head/DIR_A/DIR_B
15G osd/ceph-1/current/2.3a_head/DIR_A
15G osd/ceph-1/current/2.3a_head
7.3G osd/ceph-1/current/2.6_head/DIR_6/DIR_0
7.4G osd/ceph-1/current/2.6_head/DIR_6/DIR_8
15G osd/ceph-1/current/2.6_head/DIR_6
15G osd/ceph-1/current/2.6_head
7.3G osd/ceph-1/current/2.3f_head/DIR_F/DIR_3
7.5G osd/ceph-1/current/2.3f_head/DIR_F/DIR_B
15G osd/ceph-1/current/2.3f_head/DIR_F
15G osd/ceph-1/current/2.3f_head
7.3G osd/ceph-1/current/2.48_head/DIR_8/DIR_4
7.5G osd/ceph-1/current/2.48_head/DIR_8/DIR_C
15G osd/ceph-1/current/2.48_head/DIR_8
15G osd/ceph-1/current/2.48_head
7.5G osd/ceph-1/current/2.66_head/DIR_6/DIR_6
7.4G osd/ceph-1/current/2.66_head/DIR_6/DIR_E
15G osd/ceph-1/current/2.66_head/DIR_6
15G osd/ceph-1/current/2.66_head
7.4G osd/ceph-1/current/2.7_head/DIR_7/DIR_0
7.3G osd/ceph-1/current/2.7_head/DIR_7/DIR_8
15G osd/ceph-1/current/2.7_head/DIR_7
15G osd/ceph-1/current/2.7_head
7.3G osd/ceph-1/current/2.43_head/DIR_3/DIR_4
7.0G osd/ceph-1/current/2.43_head/DIR_3/DIR_C
15G osd/ceph-1/current/2.43_head/DIR_3
15G osd/ceph-1/current/2.43_head
7.4G osd/ceph-1/current/2.4e_head/DIR_E/DIR_4
7.1G osd/ceph-1/current/2.4e_head/DIR_E/DIR_C
15G osd/ceph-1/current/2.4e_head/DIR_E
15G osd/ceph-1/current/2.4e_head
7.7G osd/ceph-1/current/2.40_head/DIR_0/DIR_4
7.8G osd/ceph-1/current/2.40_head/DIR_0/DIR_C
16G osd/ceph-1/current/2.40_head/DIR_0
16G osd/ceph-1/current/2.40_head
7.5G osd/ceph-1/current/2.5_head/DIR_5/DIR_0
7.3G osd/ceph-1/current/2.5_head/DIR_5/DIR_8
15G osd/ceph-1/current/2.5_head/DIR_5
15G osd/ceph-1/current/2.5_head
7.5G osd/ceph-1/current/2.65_head/DIR_5/DIR_6
7.6G osd/ceph-1/current/2.65_head/DIR_5/DIR_E
15G osd/ceph-1/current/2.65_head/DIR_5
15G osd/ceph-1/current/2.65_head
7.1G osd/ceph-1/current/2.56_head/DIR_6/DIR_5
7.1G osd/ceph-1/current/2.56_head/DIR_6/DIR_D
15G osd/ceph-1/current/2.56_head/DIR_6
15G osd/ceph-1/current/2.56_head
7.4G osd/ceph-1/current/2.46_head/DIR_6/DIR_4
7.5G osd/ceph-1/current/2.46_head/DIR_6/DIR_C
15G osd/ceph-1/current/2.46_head/DIR_6
15G osd/ceph-1/current/2.46_head
7.1G osd/ceph-1/current/2.a_head/DIR_A/DIR_0
7.7G osd/ceph-1/current/2.a_head/DIR_A/DIR_8
15G osd/ceph-1/current/2.a_head/DIR_A
15G osd/ceph-1/current/2.a_head
7.7G osd/ceph-1/current/2.47_head/DIR_7/DIR_4
7.7G osd/ceph-1/current/2.47_head/DIR_7/DIR_C
16G osd/ceph-1/current/2.47_head/DIR_7
16G osd/ceph-1/current/2.47_head
7.4G osd/ceph-1/current/2.57_head/DIR_7/DIR_5
7.4G osd/ceph-1/current/2.57_head/DIR_7/DIR_D
15G osd/ceph-1/current/2.57_head/DIR_7
15G osd/ceph-1/current/2.57_head
7.0G osd/ceph-1/current/2.6a_head/DIR_A/DIR_6
7.4G osd/ceph-1/current/2.6a_head/DIR_A/DIR_E
15G osd/ceph-1/current/2.6a_head/DIR_A
15G osd/ceph-1/current/2.6a_head
7.5G osd/ceph-1/current/2.b_head/DIR_B/DIR_0
7.3G osd/ceph-1/current/2.b_head/DIR_B/DIR_8
15G osd/ceph-1/current/2.b_head/DIR_B
15G osd/ceph-1/current/2.b_head
7.3G osd/ceph-1/current/2.7c_head/DIR_C/DIR_7
7.6G osd/ceph-1/current/2.7c_head/DIR_C/DIR_F
15G osd/ceph-1/current/2.7c_head/DIR_C
15G osd/ceph-1/current/2.7c_head
7.2G osd/ceph-1/current/2.5a_head/DIR_A/DIR_5
7.7G osd/ceph-1/current/2.5a_head/DIR_A/DIR_D
15G osd/ceph-1/current/2.5a_head/DIR_A
15G osd/ceph-1/current/2.5a_head
7.2G osd/ceph-1/current/2.6b_head/DIR_B/DIR_6
7.6G osd/ceph-1/current/2.6b_head/DIR_B/DIR_E
15G osd/ceph-1/current/2.6b_head/DIR_B
15G osd/ceph-1/current/2.6b_head
7.5G osd/ceph-1/current/2.5e_head/DIR_E/DIR_5
7.2G osd/ceph-1/current/2.5e_head/DIR_E/DIR_D
15G osd/ceph-1/current/2.5e_head/DIR_E
15G osd/ceph-1/current/2.5e_head
7.5G osd/ceph-1/current/2.7d_head/DIR_D/DIR_7
7.4G osd/ceph-1/current/2.7d_head/DIR_D/DIR_F
15G osd/ceph-1/current/2.7d_head/DIR_D
15G osd/ceph-1/current/2.7d_head
7.6G osd/ceph-1/current/2.5f_head/DIR_F/DIR_5
7.3G osd/ceph-1/current/2.5f_head/DIR_F/DIR_D
15G osd/ceph-1/current/2.5f_head/DIR_F
15G osd/ceph-1/current/2.5f_head
7.6G osd/ceph-1/current/2.68_head/DIR_8/DIR_6
7.6G osd/ceph-1/current/2.68_head/DIR_8/DIR_E
16G osd/ceph-1/current/2.68_head/DIR_8
16G osd/ceph-1/current/2.68_head
7.7G osd/ceph-1/current/2.60_head/DIR_0/DIR_6
7.7G osd/ceph-1/current/2.60_head/DIR_0/DIR_E
16G osd/ceph-1/current/2.60_head/DIR_0
16G osd/ceph-1/current/2.60_head
7.4G osd/ceph-1/current/2.2_head/DIR_2/DIR_0
7.3G osd/ceph-1/current/2.2_head/DIR_2/DIR_8
15G osd/ceph-1/current/2.2_head/DIR_2
15G osd/ceph-1/current/2.2_head
7.0G osd/ceph-1/current/2.64_head/DIR_4/DIR_6
7.5G osd/ceph-1/current/2.64_head/DIR_4/DIR_E
15G osd/ceph-1/current/2.64_head/DIR_4
15G osd/ceph-1/current/2.64_head
7.5G osd/ceph-1/current/2.69_head/DIR_9/DIR_6
7.5G osd/ceph-1/current/2.69_head/DIR_9/DIR_E
15G osd/ceph-1/current/2.69_head/DIR_9
15G osd/ceph-1/current/2.69_head
7.3G osd/ceph-1/current/2.6e_head/DIR_E/DIR_6
7.2G osd/ceph-1/current/2.6e_head/DIR_E/DIR_E
15G osd/ceph-1/current/2.6e_head/DIR_E
15G osd/ceph-1/current/2.6e_head
7.4G osd/ceph-1/current/2.0_head/DIR_0/DIR_0
7.3G osd/ceph-1/current/2.0_head/DIR_0/DIR_8
15G osd/ceph-1/current/2.0_head/DIR_0
15G osd/ceph-1/current/2.0_head
7.2G osd/ceph-1/current/2.6f_head/DIR_F/DIR_6
7.3G osd/ceph-1/current/2.6f_head/DIR_F/DIR_E
15G osd/ceph-1/current/2.6f_head/DIR_F
15G osd/ceph-1/current/2.6f_head
7.3G osd/ceph-1/current/2.1_head/DIR_1/DIR_0
7.3G osd/ceph-1/current/2.1_head/DIR_1/DIR_8
15G osd/ceph-1/current/2.1_head/DIR_1
15G osd/ceph-1/current/2.1_head
7.3G osd/ceph-1/current/2.73_head/DIR_3/DIR_7
7.7G osd/ceph-1/current/2.73_head/DIR_3/DIR_F
15G osd/ceph-1/current/2.73_head/DIR_3
15G osd/ceph-1/current/2.73_head
7.3G osd/ceph-1/current/2.8_head/DIR_8/DIR_0
7.4G osd/ceph-1/current/2.8_head/DIR_8/DIR_8
15G osd/ceph-1/current/2.8_head/DIR_8
15G osd/ceph-1/current/2.8_head
7.4G osd/ceph-1/current/2.70_head/DIR_0/DIR_7
7.3G osd/ceph-1/current/2.70_head/DIR_0/DIR_F
15G osd/ceph-1/current/2.70_head/DIR_0
15G osd/ceph-1/current/2.70_head
7.3G osd/ceph-1/current/2.71_head/DIR_1/DIR_7
7.5G osd/ceph-1/current/2.71_head/DIR_1/DIR_F
15G osd/ceph-1/current/2.71_head/DIR_1
15G osd/ceph-1/current/2.71_head
7.4G osd/ceph-1/current/2.7a_head/DIR_A/DIR_7
7.3G osd/ceph-1/current/2.7a_head/DIR_A/DIR_F
15G osd/ceph-1/current/2.7a_head/DIR_A
15G osd/ceph-1/current/2.7a_head
7.3G osd/ceph-1/current/2.7f_head/DIR_F/DIR_7
7.3G osd/ceph-1/current/2.7f_head/DIR_F/DIR_F
15G osd/ceph-1/current/2.7f_head/DIR_F
15G osd/ceph-1/current/2.7f_head

is there a way for me to delete the OSD journal or move it to other partitions? or any advise?

thanks.
 
...
is there a way for me to delete the OSD journal or move it to other partitions? or any advise?
....

Is this happen on all nodes?
Put a disk in, create a new osd and start it, then wait until the rebalancing is done, stop the old osd and remove it from ceph. All can be done in the proxmox gui.
 
Only happened to 1 node. so it can be done from GUI?

How can it happen to only 1 node? Can u provide your fstab?

Yes u can do it in the GUI put u have to plug in a physical disk first! Or can u connect 1 more nodes? It should reblance after more osd's come in.
 
Hi Guys,

looking for advise. i'm having proxmox server running with ceph. now i can see that the OSD journals has filled up my / directory. below is the output of df -h

/dev/mapper/pve-root 95G 95G 0 100% /


15G osd/ceph-1/current/2.2e_head/DIR_E
15G osd/ceph-1/current/2.2e_head
7.7G osd/ceph-1/current/2.3e_head/DIR_E/DIR_3
7.5G osd/ceph-1/current/2.3e_head/DIR_E/DIR_B
...
15G osd/ceph-1/current/2.7f_head/DIR_F
15G osd/ceph-1/current/2.7f_head

is there a way for me to delete the OSD journal or move it to other partitions? or any advise?

thanks.

Hi,
you don't see the OSD journals here. This is the normal OSD content and I assume it's on seperate disks (as it should be).

"df -h" isn't the right tool, because it's also shows content from other filesystems (mounted). ncdu is much better for that (you need a little bit space to install this) if you use "ncdu -x /" you see content on the root-filesystem only.

Perhaps you mon-logs fill the root? look with "du -hs /var/log/ceph"

Udo
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!