Issue removing Ceph OSD

Paspao

Active Member
Aug 1, 2017
69
2
28
55
Hello,

I had to temporary remove a drive (OSD) from Ceph but I think I made it wrong.

- I set noout
- I Outed the OSD but I saw data redundancy started the same (I though setting noout avoided that).
- I stopped OSD.8 service

Then I extracted hot swap bay.

When I inserted disk back device was renamed from /dev/sda to /dev/sdd.

In syslog i see:
May 8 18:15:47 proxmox kernel: [1055817.730194] sd 0:0:0:0: [sdd] Attached SCSI disk
May 8 18:15:48 proxmox systemd[1]: Starting Ceph disk activation: /dev/sdd1...
May 8 18:15:48 proxmox systemd[1]: Starting Ceph disk activation: /dev/sdd2...
May 8 18:15:48 proxmox sh[862700]: main_trigger: main_trigger: Namespace(cluster='ceph', dev='/dev/sdd1', dmcrypt=None, dmcrypt_key_dir='/etc/ceph/
dmcrypt-keys', func=<function main_trigger at 0x7f5f79a43230>, log_stdout=True, prepend_to_path='/usr/bin', prog='ceph-disk', setgroup=None, setuser=None, sta
tedir='/var/lib/ceph', sync=True, sysconfdir='/etc/ceph', verbose=True)
May 8 18:15:48 proxmox sh[862700]: command: Running command: /sbin/init --version
May 8 18:15:48 proxmox sh[862700]: command_check_call: Running command: /bin/chown ceph:ceph /dev/sdd1
May 8 18:15:48 proxmox sh[862700]: command: Running command: /sbin/blkid -o udev -p /dev/sdd1
May 8 18:15:48 proxmox sh[862700]: command: Running command: /sbin/blkid -o udev -p /dev/sdd1
May 8 18:15:48 proxmox sh[862700]: main_trigger: trigger /dev/sdd1 parttype 4fbd7e29-9d25-41b8-afd0-062c0ceff05d uuid 50905fa8-b4ef-481d-8a6f-44774
cb97062
May 8 18:15:48 proxmox sh[862700]: command: Running command: /usr/sbin/ceph-disk --verbose activate /dev/sdd1
May 8 18:15:48 proxmox sh[862705]: main_trigger: main_trigger: Namespace(cluster='ceph', dev='/dev/sdd2', dmcrypt=None, dmcrypt_key_dir='/etc/ceph/
dmcrypt-keys', func=<function main_trigger at 0x7f9dcd7d4230>, log_stdout=True, prepend_to_path='/usr/bin', prog='ceph-disk', setgroup=None, setuser=None, statedir='/var/lib/ceph', sync=True, sysconfdir='/etc/ceph', verbose=True)
May 8 18:15:48 proxmox sh[862705]: command: Running command: /sbin/init --version
May 8 18:15:48 proxmox sh[862705]: command_check_call: Running command: /bin/chown ceph:ceph /dev/sdd2
May 8 18:15:48 proxmox sh[862705]: command: Running command: /sbin/blkid -o udev -p /dev/sdd2
May 8 18:15:48 proxmox sh[862705]: command: Running command: /sbin/blkid -o udev -p /dev/sdd2
May 8 18:15:48 proxmox sh[862705]: main_trigger: trigger /dev/sdd2 parttype cafecafe-9b03-4f30-b4c6-b4b80ceff106 uuid feb876c0-b68c-48d9-ae4a-15de97c17e6e
May 8 18:15:48 proxmox sh[862705]: command: Running command: /usr/sbin/ceph-disk --verbose activate-block /dev/sdd2
May 8 18:15:49 proxmox kernel: [1055819.294152] XFS (sdd1): Filesystem has duplicate UUID e128faee-669d-4062-91f7-de3f3f2d40f2 - can't mount
May 8 18:15:49 proxmox sh[862700]: main_trigger:
May 8 18:15:49 proxmox sh[862700]: main_trigger: main_activate: path = /dev/sdd1
May 8 18:15:49 proxmox sh[862700]: get_dm_uuid: get_dm_uuid /dev/sdd1 uuid path is /sys/dev/block/8:49/dm/uuid
May 8 18:15:49 proxmox sh[862700]: command: Running command: /sbin/blkid -o udev -p /dev/sdd1
May 8 18:15:49 proxmox sh[862700]: command: Running command: /sbin/blkid -p -s TYPE -o value -- /dev/sdd1
May 8 18:15:49 proxmox sh[862700]: command: Running command: /usr/bin/ceph-conf --cluster=ceph --name=osd. --lookup osd_mount_options_xfs
May 8 18:15:49 proxmox sh[862700]: command: Running command: /usr/bin/ceph-conf --cluster=ceph --name=osd. --lookup osd_fs_mount_options_xfs
May 8 18:15:49 proxmox sh[862700]: mount: Mounting /dev/sdd1 on /var/lib/ceph/tmp/mnt.JiZ54Q with options noatime,inode64
May 8 18:15:49 proxmox sh[862700]: command_check_call: Running command: /bin/mount -t xfs -o noatime,inode64 -- /dev/sdd1 /var/lib/ceph/tmp/mnt.JiZ54Q
May 8 18:15:49 proxmox sh[862700]: mount: wrong fs type, bad option, bad superblock on /dev/sdd1,
May 8 18:15:49 proxmox sh[862700]: missing codepage or helper program, or other error
May 8 18:15:49 proxmox sh[862700]: In some cases useful info is found in syslog - try
May 8 18:15:49 proxmox sh[862700]: dmesg | tail or so.
May 8 18:15:49 proxmox sh[862700]: Traceback (most recent call last):
May 8 18:15:49 proxmox sh[862700]: File "/usr/sbin/ceph-disk", line 11, in <module>
May 8 18:15:49 proxmox sh[862700]: load_entry_point('ceph-disk==1.0.0', 'console_scripts', 'ceph-disk')()
May 8 18:15:49 proxmox sh[862700]: File "/usr/lib/python2.7/dist-packages/ceph_disk/main.py", line 5736, in run
May 8 18:15:49 proxmox sh[862700]: main(sys.argv[1:])
May 8 18:15:49 proxmox sh[862700]: File "/usr/lib/python2.7/dist-packages/ceph_disk/main.py", line 5687, in main
May 8 18:15:49 proxmox sh[862700]: args.func(args)
May 8 18:15:49 proxmox sh[862700]: File "/usr/lib/python2.7/dist-packages/ceph_disk/main.py", line 3777, in main_activate
May 8 18:15:49 proxmox sh[862700]: reactivate=args.reactivate,
May 8 18:15:49 proxmox sh[862700]: File "/usr/lib/python2.7/dist-packages/ceph_disk/main.py", line 3517, in mount_activate
May 8 18:15:49 proxmox sh[862700]: path = mount(dev=dev, fstype=fstype, options=mount_options)
May 8 18:15:49 proxmox sh[862700]: File "/usr/lib/python2.7/dist-packages/ceph_disk/main.py", line 1454, in mount
May 8 18:15:49 proxmox sh[862700]: raise MountError(e)
May 8 18:15:49 proxmox sh[862700]: ceph_disk.main.MountError: Mounting filesystem failed: Command '['/bin/mount', '-t', u'xfs', '-o', 'noatime,inode64', '--', '/dev/sdd1', '/var/lib/ceph/tmp/mnt.JiZ54Q']' returned non-zero exit status 32
...
May 8 18:15:49 proxmox kernel: [1055819.490489] XFS (sdd1): Filesystem has duplicate UUID e128faee-669d-4062-91f7-de3f3f2d40f2 - can't mount

The disk in GUI is listed as Usage: partitions (no more as Ceph osd.8).

What did I made wrong?

What are the steps to remount in correctly o recreate this OSD?

Thank you.
P
 
It seems some intelligent job passed and fixed it automatically, it was again set as "OSD.8 " and I could put back in .
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!