Ceph goes down after adding second boot disk for mirror

vRod

Active Member
Jan 11, 2012
34
1
28
Hi all,

I have a 3 node ceph cluster, consisting of HPE Gen9 servers. It's been running well since I set it up and I really enjoy the "no single point of failure" feature.

Now, during the installation I was using some S3700 100gb drives for boot zfs mirrors, however for one of the hosts, I only had one at the time. Now I want to add a second drive and convert the zfs pool to a mirror. Got it all working, but wanted to reboot the host in order to check that it can also boot with the second drive. Server boots fine but after 2-3 minutes at the proxmox login console, all my ceph services goes down.

Only way to get it back up is to either turn off the host or reboot it again without the newly added drive. Now, the new drive goes into port 2, whereas the ceph drives are plugged into 3-8. I suspect that the new drive changes the order of the drives (sda,b,c etc) and that ceph then falsely believes it's a ceph drive. However obviously it's not and ceph grinds to a halt. Now, the data and everything is secure, I had no issues there. It is however a little worrying. Tried to google around but couldn't really find anything.

Anyone tried this before? Where would I have to change the ceph config? I can imagine that each osd maps to a /dev/sdX and I suppose I need to find that file and change it.
 

Alwin

Proxmox Retired Staff
Retired Staff
Aug 1, 2017
4,617
457
88
Ceph knows about its disks, it check uuids and LVs for that. No order needed. Check in the logs what is happening during and after boot. This will give more clues to what is going on.
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get your own in 60 seconds.

Buy now!