VM/CT Creation Issue When a OSD is "down"

[HE_Cole]

Hello Everyone!

I recently created and setup a Promox and Ceph cluster on my 5 servers and i am very happy with it so far.

I went to create my first CT and i was getting a error

error with cfs lock 'storage-VPS-MAIN-STOR1_ct': rbd rm 'vm-100-disk-2' error: rbd: error: image still has watchers
TASK ERROR: can't mount rbd volume vm-100-disk-2: received interrupt

AND

rbd: sysfs write failed
error with cfs lock 'storage-VPS-MAIN-STOR1_ct': can't unmap rbd volume vm-100-disk-0: rbd: sysfs write failed
TASK ERROR: command 'mkfs.ext4 -O mmp -E 'root_owner=100000:100000' /dev/rbd/VPS-MAIN-STOR1/vm-100-disk-0' failed: received interrupt

I checked my my OSD's and i noticed one was showing up as a partion not a osd.28 etc for some reason in the ceph logs i see

2018-11-24 17:33:07.057149 mon.he-s01-r01-pve01 mon.0 10.10.11.3:6789/0 12179 : cluster [INF] osd.28 failed (root=default,host=he-s04-r01-pve01) (connection refused reported by osd.13)

So i went to the server in the GUI under OSD and i set the bad OSd as out and destroyed it.

Now i can create CT's and VM's with no issues.

Is it normal when you lose a OSD for ceph/promox to give errors like above on ct/vm creation?

After destroying the osd i am unable to add it back and it is listed as a disk under disks but shows as PARTION, how do i re add it back after i destroyed it?

"I am going to replace that drive FYI but for after i replace it i need to be able to re add it without a reboot how do i do this?"

Thanks.