Ceph OSD losing journal often

Discussion in 'Proxmox VE: Installation and configuration' started by CadilLACi, Sep 20, 2016.

  1. CadilLACi

    CadilLACi New Member

    Joined:
    Sep 20, 2016
    Messages:
    11
    Likes Received:
    0
    Hello!

    I operate a home lab for experimental purposes.
    It has 3 Proxmox nodes with each:
    1ssd for system+lvm storage + free space
    1hdd for ceph storage

    One of my boxes goes funky from time to time.
    The drive names (sda, sdb, sdc) get mixed up!!!!!

    For example, 1 set a 20 gig partition on the sda ssd for ceph journal (sda4). The hdd is sdb.
    I made the osd using

    "pveceph createosd /dev/sdb -journal_dev /dev/sda4"

    I reboot, and now the HDD is sda and the ssd is sdb!!!!!

    This happens, and now ceph doesn't find the journal device, the processes hang, and the ceph storage becomes degraded.

    Strange enough, tha proxmox server itself starts, and the ceph storage recognizes the osd for the disk it is, only the journal device is missing.....

    Did anyone come across this problem earlier?
     
  2. CadilLACi

    CadilLACi New Member

    Joined:
    Sep 20, 2016
    Messages:
    11
    Likes Received:
    0
    Hi!

    After long experimentation, it turned out that on one of my vm-s I had a disk which had cache enabled.
    Turns out with CEPH storage backend, that is not good.

    Stopped the VM, changed to no cache, and my problems disappeared since then.




    recovering the journal partition:

    The journal device in this ceph implementation is a link to a device.

    A command similar to this one solved it: (ofc i had to know which partition was the one the server didn't find)

    ln -s /dev/sde1 /var/lib/ceph/osd/ceph-0/journal

    Didn't even have to reboot after that, just restart ceph mon and up/in the osd.

    Comparing the server/Syslog tab and the server/ceph/log tab was really helpful. Nice implemetation.

    CadilLACi
     
  1. This site uses cookies to help personalise content, tailor your experience and to keep you logged in if you register.
    By continuing to use this site, you are consenting to our use of cookies.
    Dismiss Notice