Issue removing Ceph OSD

[Paspao]

Hello,

I had to temporary remove a drive (OSD) from Ceph but I think I made it wrong.

- I set noout
- I Outed the OSD but I saw data redundancy started the same (I though setting noout avoided that).
- I stopped OSD.8 service

Then I extracted hot swap bay.

When I inserted disk back device was renamed from /dev/sda to /dev/sdd.

In syslog i see:

Spoiler: syslog

May 8 18:15:47 proxmox kernel: [1055817.730194] sd 0:0:0:0: [sdd] Attached SCSI disk
May 8 18:15:48 proxmox systemd[1]: Starting Ceph disk activation: /dev/sdd1...
May 8 18:15:48 proxmox systemd[1]: Starting Ceph disk activation: /dev/sdd2...
May 8 18:15:48 proxmox sh[862700]: main_trigger: main_trigger: Namespace(cluster='ceph', dev='/dev/sdd1', dmcrypt=None, dmcrypt_key_dir='/etc/ceph/
dmcrypt-keys', func=<function main_trigger at 0x7f5f79a43230>, log_stdout=True, prepend_to_path='/usr/bin', prog='ceph-disk', setgroup=None, setuser=None, sta
tedir='/var/lib/ceph', sync=True, sysconfdir='/etc/ceph', verbose=True)
May 8 18:15:48 proxmox sh[862700]: command: Running command: /sbin/init --version
May 8 18:15:48 proxmox sh[862700]: command_check_call: Running command: /bin/chown ceph:ceph /dev/sdd1
May 8 18:15:48 proxmox sh[862700]: command: Running command: /sbin/blkid -o udev -p /dev/sdd1
May 8 18:15:48 proxmox sh[862700]: command: Running command: /sbin/blkid -o udev -p /dev/sdd1
May 8 18:15:48 proxmox sh[862700]: main_trigger: trigger /dev/sdd1 parttype 4fbd7e29-9d25-41b8-afd0-062c0ceff05d uuid 50905fa8-b4ef-481d-8a6f-44774
cb97062
May 8 18:15:48 proxmox sh[862700]: command: Running command: /usr/sbin/ceph-disk --verbose activate /dev/sdd1
May 8 18:15:48 proxmox sh[862705]: main_trigger: main_trigger: Namespace(cluster='ceph', dev='/dev/sdd2', dmcrypt=None, dmcrypt_key_dir='/etc/ceph/
dmcrypt-keys', func=<function main_trigger at 0x7f9dcd7d4230>, log_stdout=True, prepend_to_path='/usr/bin', prog='ceph-disk', setgroup=None, setuser=None, statedir='/var/lib/ceph', sync=True, sysconfdir='/etc/ceph', verbose=True)
May 8 18:15:48 proxmox sh[862705]: command: Running command: /sbin/init --version
May 8 18:15:48 proxmox sh[862705]: command_check_call: Running command: /bin/chown ceph:ceph /dev/sdd2
May 8 18:15:48 proxmox sh[862705]: command: Running command: /sbin/blkid -o udev -p /dev/sdd2
May 8 18:15:48 proxmox sh[862705]: command: Running command: /sbin/blkid -o udev -p /dev/sdd2
May 8 18:15:48 proxmox sh[862705]: main_trigger: trigger /dev/sdd2 parttype cafecafe-9b03-4f30-b4c6-b4b80ceff106 uuid feb876c0-b68c-48d9-ae4a-15de97c17e6e
May 8 18:15:48 proxmox sh[862705]: command: Running command: /usr/sbin/ceph-disk --verbose activate-block /dev/sdd2
May 8 18:15:49 proxmox kernel: [1055819.294152] XFS (sdd1): Filesystem has duplicate UUID e128faee-669d-4062-91f7-de3f3f2d40f2 - can't mount
May 8 18:15:49 proxmox sh[862700]: main_trigger:
May 8 18:15:49 proxmox sh[862700]: main_trigger: main_activate: path = /dev/sdd1
May 8 18:15:49 proxmox sh[862700]: get_dm_uuid: get_dm_uuid /dev/sdd1 uuid path is /sys/dev/block/8:49/dm/uuid
May 8 18:15:49 proxmox sh[862700]: command: Running command: /sbin/blkid -o udev -p /dev/sdd1
May 8 18:15:49 proxmox sh[862700]: command: Running command: /sbin/blkid -p -s TYPE -o value -- /dev/sdd1
May 8 18:15:49 proxmox sh[862700]: command: Running command: /usr/bin/ceph-conf --cluster=ceph --name=osd. --lookup osd_mount_options_xfs
May 8 18:15:49 proxmox sh[862700]: command: Running command: /usr/bin/ceph-conf --cluster=ceph --name=osd. --lookup osd_fs_mount_options_xfs
May 8 18:15:49 proxmox sh[862700]: mount: Mounting /dev/sdd1 on /var/lib/ceph/tmp/mnt.JiZ54Q with options noatime,inode64
May 8 18:15:49 proxmox sh[862700]: command_check_call: Running command: /bin/mount -t xfs -o noatime,inode64 -- /dev/sdd1 /var/lib/ceph/tmp/mnt.JiZ54Q
May 8 18:15:49 proxmox sh[862700]: mount: wrong fs type, bad option, bad superblock on /dev/sdd1,
May 8 18:15:49 proxmox sh[862700]: missing codepage or helper program, or other error
May 8 18:15:49 proxmox sh[862700]: In some cases useful info is found in syslog - try
May 8 18:15:49 proxmox sh[862700]: dmesg | tail or so.
May 8 18:15:49 proxmox sh[862700]: Traceback (most recent call last):
May 8 18:15:49 proxmox sh[862700]: File "/usr/sbin/ceph-disk", line 11, in <module>
May 8 18:15:49 proxmox sh[862700]: load_entry_point('ceph-disk==1.0.0', 'console_scripts', 'ceph-disk')()
May 8 18:15:49 proxmox sh[862700]: File "/usr/lib/python2.7/dist-packages/ceph_disk/main.py", line 5736, in run
May 8 18:15:49 proxmox sh[862700]: main(sys.argv[1:])
May 8 18:15:49 proxmox sh[862700]: File "/usr/lib/python2.7/dist-packages/ceph_disk/main.py", line 5687, in main
May 8 18:15:49 proxmox sh[862700]: args.func(args)
May 8 18:15:49 proxmox sh[862700]: File "/usr/lib/python2.7/dist-packages/ceph_disk/main.py", line 3777, in main_activate
May 8 18:15:49 proxmox sh[862700]: reactivate=args.reactivate,
May 8 18:15:49 proxmox sh[862700]: File "/usr/lib/python2.7/dist-packages/ceph_disk/main.py", line 3517, in mount_activate
May 8 18:15:49 proxmox sh[862700]: path = mount(dev=dev, fstype=fstype, options=mount_options)
May 8 18:15:49 proxmox sh[862700]: File "/usr/lib/python2.7/dist-packages/ceph_disk/main.py", line 1454, in mount
May 8 18:15:49 proxmox sh[862700]: raise MountError(e)
May 8 18:15:49 proxmox sh[862700]: ceph_disk.main.MountError: Mounting filesystem failed: Command '['/bin/mount', '-t', u'xfs', '-o', 'noatime,inode64', '--', '/dev/sdd1', '/var/lib/ceph/tmp/mnt.JiZ54Q']' returned non-zero exit status 32
...
May 8 18:15:49 proxmox kernel: [1055819.490489] XFS (sdd1): Filesystem has duplicate UUID e128faee-669d-4062-91f7-de3f3f2d40f2 - can't mount

The disk in GUI is listed as Usage: partitions (no more as Ceph osd.8).

What did I made wrong?

What are the steps to remount in correctly o recreate this OSD?

Thank you.
P

 

[Paspao]

It seems some intelligent job passed and fixed it automatically, it was again set as "OSD.8 " and I could put back in .