Ceph OSD losing journal often

CadilLACi

Active Member
Sep 20, 2016
15
0
41
42
Hello!

I operate a home lab for experimental purposes.
It has 3 Proxmox nodes with each:
1ssd for system+lvm storage + free space
1hdd for ceph storage

One of my boxes goes funky from time to time.
The drive names (sda, sdb, sdc) get mixed up!!!!!

For example, 1 set a 20 gig partition on the sda ssd for ceph journal (sda4). The hdd is sdb.
I made the osd using

"pveceph createosd /dev/sdb -journal_dev /dev/sda4"

I reboot, and now the HDD is sda and the ssd is sdb!!!!!

This happens, and now ceph doesn't find the journal device, the processes hang, and the ceph storage becomes degraded.

Strange enough, tha proxmox server itself starts, and the ceph storage recognizes the osd for the disk it is, only the journal device is missing.....

Did anyone come across this problem earlier?
 
Hi!

After long experimentation, it turned out that on one of my vm-s I had a disk which had cache enabled.
Turns out with CEPH storage backend, that is not good.

Stopped the VM, changed to no cache, and my problems disappeared since then.




recovering the journal partition:

The journal device in this ceph implementation is a link to a device.

A command similar to this one solved it: (ofc i had to know which partition was the one the server didn't find)

ln -s /dev/sde1 /var/lib/ceph/osd/ceph-0/journal

Didn't even have to reboot after that, just restart ceph mon and up/in the osd.

Comparing the server/Syslog tab and the server/ceph/log tab was really helpful. Nice implemetation.

CadilLACi
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!