Issue removing Ceph OSD

Discussion in 'Proxmox VE: Installation and configuration' started by Paspao, May 8, 2019.

Tags:
  1. Paspao

    Paspao Member

    Joined:
    Aug 1, 2017
    Messages:
    37
    Likes Received:
    1
    Hello,

    I had to temporary remove a drive (OSD) from Ceph but I think I made it wrong.

    - I set noout
    - I Outed the OSD but I saw data redundancy started the same (I though setting noout avoided that).
    - I stopped OSD.8 service

    Then I extracted hot swap bay.

    When I inserted disk back device was renamed from /dev/sda to /dev/sdd.

    In syslog i see:
    May 8 18:15:47 proxmox kernel: [1055817.730194] sd 0:0:0:0: [sdd] Attached SCSI disk
    May 8 18:15:48 proxmox systemd[1]: Starting Ceph disk activation: /dev/sdd1...
    May 8 18:15:48 proxmox systemd[1]: Starting Ceph disk activation: /dev/sdd2...
    May 8 18:15:48 proxmox sh[862700]: main_trigger: main_trigger: Namespace(cluster='ceph', dev='/dev/sdd1', dmcrypt=None, dmcrypt_key_dir='/etc/ceph/
    dmcrypt-keys', func=<function main_trigger at 0x7f5f79a43230>, log_stdout=True, prepend_to_path='/usr/bin', prog='ceph-disk', setgroup=None, setuser=None, sta
    tedir='/var/lib/ceph', sync=True, sysconfdir='/etc/ceph', verbose=True)
    May 8 18:15:48 proxmox sh[862700]: command: Running command: /sbin/init --version
    May 8 18:15:48 proxmox sh[862700]: command_check_call: Running command: /bin/chown ceph:ceph /dev/sdd1
    May 8 18:15:48 proxmox sh[862700]: command: Running command: /sbin/blkid -o udev -p /dev/sdd1
    May 8 18:15:48 proxmox sh[862700]: command: Running command: /sbin/blkid -o udev -p /dev/sdd1
    May 8 18:15:48 proxmox sh[862700]: main_trigger: trigger /dev/sdd1 parttype 4fbd7e29-9d25-41b8-afd0-062c0ceff05d uuid 50905fa8-b4ef-481d-8a6f-44774
    cb97062
    May 8 18:15:48 proxmox sh[862700]: command: Running command: /usr/sbin/ceph-disk --verbose activate /dev/sdd1
    May 8 18:15:48 proxmox sh[862705]: main_trigger: main_trigger: Namespace(cluster='ceph', dev='/dev/sdd2', dmcrypt=None, dmcrypt_key_dir='/etc/ceph/
    dmcrypt-keys', func=<function main_trigger at 0x7f9dcd7d4230>, log_stdout=True, prepend_to_path='/usr/bin', prog='ceph-disk', setgroup=None, setuser=None, statedir='/var/lib/ceph', sync=True, sysconfdir='/etc/ceph', verbose=True)
    May 8 18:15:48 proxmox sh[862705]: command: Running command: /sbin/init --version
    May 8 18:15:48 proxmox sh[862705]: command_check_call: Running command: /bin/chown ceph:ceph /dev/sdd2
    May 8 18:15:48 proxmox sh[862705]: command: Running command: /sbin/blkid -o udev -p /dev/sdd2
    May 8 18:15:48 proxmox sh[862705]: command: Running command: /sbin/blkid -o udev -p /dev/sdd2
    May 8 18:15:48 proxmox sh[862705]: main_trigger: trigger /dev/sdd2 parttype cafecafe-9b03-4f30-b4c6-b4b80ceff106 uuid feb876c0-b68c-48d9-ae4a-15de97c17e6e
    May 8 18:15:48 proxmox sh[862705]: command: Running command: /usr/sbin/ceph-disk --verbose activate-block /dev/sdd2
    May 8 18:15:49 proxmox kernel: [1055819.294152] XFS (sdd1): Filesystem has duplicate UUID e128faee-669d-4062-91f7-de3f3f2d40f2 - can't mount
    May 8 18:15:49 proxmox sh[862700]: main_trigger:
    May 8 18:15:49 proxmox sh[862700]: main_trigger: main_activate: path = /dev/sdd1
    May 8 18:15:49 proxmox sh[862700]: get_dm_uuid: get_dm_uuid /dev/sdd1 uuid path is /sys/dev/block/8:49/dm/uuid
    May 8 18:15:49 proxmox sh[862700]: command: Running command: /sbin/blkid -o udev -p /dev/sdd1
    May 8 18:15:49 proxmox sh[862700]: command: Running command: /sbin/blkid -p -s TYPE -o value -- /dev/sdd1
    May 8 18:15:49 proxmox sh[862700]: command: Running command: /usr/bin/ceph-conf --cluster=ceph --name=osd. --lookup osd_mount_options_xfs
    May 8 18:15:49 proxmox sh[862700]: command: Running command: /usr/bin/ceph-conf --cluster=ceph --name=osd. --lookup osd_fs_mount_options_xfs
    May 8 18:15:49 proxmox sh[862700]: mount: Mounting /dev/sdd1 on /var/lib/ceph/tmp/mnt.JiZ54Q with options noatime,inode64
    May 8 18:15:49 proxmox sh[862700]: command_check_call: Running command: /bin/mount -t xfs -o noatime,inode64 -- /dev/sdd1 /var/lib/ceph/tmp/mnt.JiZ54Q
    May 8 18:15:49 proxmox sh[862700]: mount: wrong fs type, bad option, bad superblock on /dev/sdd1,
    May 8 18:15:49 proxmox sh[862700]: missing codepage or helper program, or other error
    May 8 18:15:49 proxmox sh[862700]: In some cases useful info is found in syslog - try
    May 8 18:15:49 proxmox sh[862700]: dmesg | tail or so.
    May 8 18:15:49 proxmox sh[862700]: Traceback (most recent call last):
    May 8 18:15:49 proxmox sh[862700]: File "/usr/sbin/ceph-disk", line 11, in <module>
    May 8 18:15:49 proxmox sh[862700]: load_entry_point('ceph-disk==1.0.0', 'console_scripts', 'ceph-disk')()
    May 8 18:15:49 proxmox sh[862700]: File "/usr/lib/python2.7/dist-packages/ceph_disk/main.py", line 5736, in run
    May 8 18:15:49 proxmox sh[862700]: main(sys.argv[1:])
    May 8 18:15:49 proxmox sh[862700]: File "/usr/lib/python2.7/dist-packages/ceph_disk/main.py", line 5687, in main
    May 8 18:15:49 proxmox sh[862700]: args.func(args)
    May 8 18:15:49 proxmox sh[862700]: File "/usr/lib/python2.7/dist-packages/ceph_disk/main.py", line 3777, in main_activate
    May 8 18:15:49 proxmox sh[862700]: reactivate=args.reactivate,
    May 8 18:15:49 proxmox sh[862700]: File "/usr/lib/python2.7/dist-packages/ceph_disk/main.py", line 3517, in mount_activate
    May 8 18:15:49 proxmox sh[862700]: path = mount(dev=dev, fstype=fstype, options=mount_options)
    May 8 18:15:49 proxmox sh[862700]: File "/usr/lib/python2.7/dist-packages/ceph_disk/main.py", line 1454, in mount
    May 8 18:15:49 proxmox sh[862700]: raise MountError(e)
    May 8 18:15:49 proxmox sh[862700]: ceph_disk.main.MountError: Mounting filesystem failed: Command '['/bin/mount', '-t', u'xfs', '-o', 'noatime,inode64', '--', '/dev/sdd1', '/var/lib/ceph/tmp/mnt.JiZ54Q']' returned non-zero exit status 32
    ...
    May 8 18:15:49 proxmox kernel: [1055819.490489] XFS (sdd1): Filesystem has duplicate UUID e128faee-669d-4062-91f7-de3f3f2d40f2 - can't mount

    The disk in GUI is listed as Usage: partitions (no more as Ceph osd.8).

    What did I made wrong?

    What are the steps to remount in correctly o recreate this OSD?

    Thank you.
    P
     
  2. Paspao

    Paspao Member

    Joined:
    Aug 1, 2017
    Messages:
    37
    Likes Received:
    1
    It seems some intelligent job passed and fixed it automatically, it was again set as "OSD.8 " and I could put back in .
     
  1. This site uses cookies to help personalise content, tailor your experience and to keep you logged in if you register.
    By continuing to use this site, you are consenting to our use of cookies.
    Dismiss Notice